DEPARTMENT OF HEALTH AND HUMAN SERVICES

> FOOD AND DRUG ADMINISTRATION
>
> CENTER FOR DRUG EVALUATION AND RESEARCH
>
> **CURRENT STATUS OF USEFUL WRITTEN PRESCRIPTION**
>
> **DRUG INFORMATION FOR PATIENTS**
>
> **VOLUME I**
>
> Tuesday, February 29, 2000
>
> 1:00 p.m.
>
> Double Tree Hotel
>
> 1750 Rockville Pike
>
> Rockville, Maryland
>
> [C]{.underline} [O]{.underline} [N]{.underline} [T]{.underline}
> [E]{.underline} [N]{.underline} [T]{.underline} [S]{.underline}
>
> [PAGE]{.underline}
>
> Moderator: Nancy M. Ostrove, Ph.D. 3
>
> Welcome: Nancy Smith, Ph.D. 7
>
> Why Are We Here: Thomas J. McGinnis, R.Ph 8
>
> History of Recent Private-Sector Information Efforts:
>
> Judith A. O\'Brien 17
>
> Background on the Patient Information Project:
>
> Karen Oster 24
>
> 1999 Patient Information Assessment:
>
> Bonnie Svarstad, Ph.D. 31
>
> Questions from the Audience 76
>
> Adjournment 120
>
> [P]{.underline} [R]{.underline} [O]{.underline} [C]{.underline}
> [E]{.underline} [E]{.underline} [D]{.underline} [I]{.underline}
> [N]{.underline} [G]{.underline} [S]{.underline}
>
> DR. OSTROVE: Good afternoon. Welcome to lovely downtown Rockville. We
> are really happy to see you today. I am Nancy Ostrove, Chief of the
> Research and Review Branch III within the Division of Drug Marketing,
> Advertising and Communications, in the Center for Drug Evaluation and
> Research \-- and if you got all that, you are better than most
> reporters that I have spoken with recently.
>
> We are so pleased that you have joined us here today to hear about the
> FDA sponsored interim study of the Private Sector\'s Prescription
> Medicine Information Program for Patients \-- I mean, that is what I
> have written down \--
>
> \[Laughter\]
>
> \-- and I wrote it! But we really are happy to see you here. You know,
> I was here about half an hour ago; it was kind of thin and I was
> thinking oh, no, don\'t tell me you aren\'t interested in this
> anymore. I can\'t believe it. And, in fact, that wasn\'t the case. So,
> I am really happy to see you here.
>
> I hope you will be able to stay for the full workshop, which is this
> afternoon and tomorrow. If you have only planned to be here for today,
> I would like to put a bug in your ear. What I would like you to do is
> to think about staying for tomorrow\'s small group breakout sessions.
> I am suggesting this because I think it would be helpful for both you
> and for the process. For you, I think it is helpful so that you will
> get a really good sense of other people\'s perspectives. In terms of
> the process, well, your staying and giving your perspective will
> ensure that others are aware of your perspective as well.
>
> Now, let me assure you we are going to read all the comments that are
> going to be submitted to the docket. As you know, we published a
> Federal Register notice with questions. We asked that comments be
> submitted to the docket concerning this, and we are going to read
> every single one of them in great detail and analyze them, the way we
> always do, but even though those comments are in the docket, they are
> not as accessible to other people. Then, there are perspectives that
> we expect that you will all be verbalizing tomorrow. So, that is kind
> of the reason.
>
> One thing \-- you know that you can never avoid is glitches. There is
> one glitch that I wanted to go over with you about just so that when
> you go through your packet you are not surprised. In the packet that
> you got when you registered today there is a lot of material. There
> are excerpts from the action plan. There is the interim report. There
> are a couple of separate pieces of paper on the left-hand side: one is
> the agenda; one is the list of questions that we have asked. You also
> have a copy of Public Law 104-180 in there.
>
> You also have something that lists ten criteria. Those are the
> criteria that were derived from the action plan that kind of served as
> the basis of the assessment. If you count the bullets, you will see
> there are only eight bullets. Oops! We kind of left out the last two.
> That was not on purpose, and that was not to imply in any way that
> those are not important because they are important. Specifically, the
> additional criteria \-- by the way, they are there in full in your
> packet with Dr. Svarstad\'s slides, on page six \-- the additional two
> are information that is legible and readily comprehensible to most
> consumers, and information is up to date and timely. Clearly, these
> are important criteria. That was just one of those little glitches,
> and if that is the only glitch that we have to deal with over the next
> couple of days I will be eternally grateful.
>
> So, speaking of non-glitches up to this point, I also would like to
> thank everyone who has been involved in putting together this workshop
> and, specifically, Ellen Tabak and Marcia Trenter and all of the
> people at OTCOM who have just worked so hard at getting it all
> together.
>
> So, with that in mind, what I would like to do is introduce Dr. Nancy
> Smith, Director of the Office of Training and Communications within
> the Center for Drug Evaluation and Research. Dr. Smith\'s training and
> early FDA experience has been as a statistician. However, as you can
> tell from her current position, in the past few years she has greatly
> expanded both her interests and her responsibilities. Dr. Smith served
> on the FDA task force that evaluated the system for managing the risks
> of FDA-approved medical products, and she is heading the Center\'s
> effort to expand communications, promoting a better understanding of
> the safe use of medicines.
>
> The Center, of course, believes that this private sector initiative to
> ensure that patients who receive new prescriptions and also get useful
> information about their prescriptions is an important piece of this
> overall effort to encourage the safe use of medicines.
>
> After Dr. Smith\'s remarks, what I would like is for Tom McGinnis to
> take the podium so we don\'t have to have any popping up and down
> constantly between people. Tom is Director of Pharmacy Affairs within
> FDA\'s Office of Policy, and provides agency liaison to health
> professional associations. Capt. McGinnis \-- oh, you are not in
> uniform today \-- works on issues of interest to both patients and
> pharmacy. Most recently, he has been extensively involved in the
> agency\'s examination of Internet pharmacies.
>
> Tom is no stranger to most of you in this room, having been involved
> for many years in FDA\'s efforts to ensure that patients receiving new
> prescriptions also get useful information about their medicines. Tom
> is going to revisit a bit of the history of FDA\'s patient information
> related efforts, kind of as a refresher in the context for the
> question of why are we here today.
>
> So, Nancy?
>
> **Welcome**
>
> DR. SMITH: Thank you, Nancy. I would also like to take this
> opportunity to welcome you to this meeting. The communication of
> accurate and unbiased information is extremely important to CDER and
> to the FDA. It is very important to us that patients can be more
> knowledgeable about their prescriptions.
>
> One of Dr. Jane Henney\'s first actions after being confirmed by the
> Senate as commissioner of the agency, about 16 months ago, was to form
> a task force to look into drug safety. The result of that effort was a
> report which was published in May of last year, and that report is
> available on the FDA website if anyone is interested.
>
> The report is titled, \"Managing the Risks from Medical Product Use.\"
> The report discusses risk management practices within the overall
> healthcare delivery system, focusing on the roles and responsibilities
> of each component.
>
> Risk assessment, which is the agency\'s primary role, was discussed
> both premarket and postmarket. The approval of medical products, as we
> all know, requires a demonstration of efficacy and safety. However,
> approval does not mean that a product is without risks. All drugs, all
> medicinal products have risks. A safe product is one that has
> reasonable risks given the magnitude of benefit that is expected and
> the alternatives that are available.
>
> The task force looked into various ways to manage these inherent
> risks. All participants in the medical product development and
> delivery system play an important role in maintaining this
> benefit/risk assessment.
>
> Our task at this meeting is to discuss an important part of that
> delivery system, the written information that a patient receives with
> their prescription. Consumers in the United States want to actively
> participate in their healthcare decisions. They need to know about the
> potential benefits and the potential risks of the products so that
> they can make informed decisions based on their personal values. The
> evaluation of the usefulness of the information they receive is of
> utmost importance to us, and we really appreciate your interest and
> involvement in this area.
>
> One other thing I wanted to point out while I am up here \-- when you
> registered you also received, or you should have received a small
> packet of information about another upcoming workshop that we would
> like to invite you to. This workshop will be held in late March, at
> the Bethesda Hyatt, and it is being co-sponsored by the FDA and the
> National Patient Safety Foundation. The title of this workshop is
> \"Safe Medical Treatments: Everyone has a Role.\" It is designed to be
> a forum for consumer organizations and patient organizations to talk
> with us, to have a role in discussing how we can help consumers become
> more involved in their own care. We encourage all of you who are
> interested in this to register for the workshop. I know there are a
> couple of people here who are involved in the workshop already, and we
> appreciate your participation there and here.
>
> So, again, thank you for coming and I look forward to what we will
> hear during the upcoming two days. Thank you.
>
> **Why We are Here?**
>
> MR. MCGINNIS: Thank you for the introduction, Nancy. I bring greetings
> from Dr. Henney who wanted to be here with us today but,
> unfortunately, the Senate scheduled our appropriations committee
> meeting today. She and other top agency officials are down there all
> day today answering questions. So, I have been asked to set the stage
> for our discussions over the next day and a half on patient
> information issues.
>
> First, let\'s quickly review 32 years of FDA involvement in patient
> information issues. It began back in 1968 when FDA first required
> warning and risk information on isoproterenol inhalation products.
> This was the first prescription medication that required
> consumer-oriented written information.
>
> Two years later, in 1970, FDA began requiring information for
> consumers, in response to concerns of women\'s health advocates and
> new data, on the potential for long-term effects related to oral
> contraceptives and other hormone-based products.
>
> In 1979, FDA proposed a rule that would have required prescription
> drug manufacturers to produce and distribute FDA-approved written
> information, known as patient package inserts or PPI\'s, for ten
> classes of drugs. This rule was withdrawn in 1982 with assurances from
> the private sector who would be able to give consumers needed
> information about their prescription drugs.
>
> When FDA took up this issue again in late 1991, as most of us in this
> room already know, inadequate access to useful patient information was
> a major cause of inappropriate use of prescription medications leading
> to unnecessary emergency room visits and hospital admissions. In 1995,
> FDA estimated that the cost of these hospitalizations was 20 billion
> dollars annually. Others estimated the costs as high as 77 billion
> dollars in 1995, which was the same amount of money that the U.S.
> spent on prescription medications in 1995. So we were spending one
> extra dollar for every dollar we spent on prescription medications for
> those misadventures that were caused.
>
> Today we are finding more and more Americans are using prescription
> drugs. In 1995, U.S. consumers were purchasing 2.1 billion
> prescriptions per year. Prescription drug usage has continued to
> increase yearly with almost three billion prescriptions being written
> in 1999 and that number is projected to reach four billion by the year
> 2004.
>
> As you have seen from FDA studies over the years and we will hear
> shortly from Dr. Svarstad, the rate of distribution of written
> prescription drug information materials has increased steadily over
> the last 15 years, however, the quality and understandability of such
> materials has been, and continues to be, variable.
>
> Why are we here? In the Federal Register of August 24, 1995, FDA
> published a proposed rule that aimed to increase the quality and
> quantity of written information about prescription medications given
> to patients. In the proposed rule, entitled \"Prescription Drug
> Product Labeling: Medication Guide Requirements,\" FDA encouraged the
> private sector to develop and distribute patient-oriented written
> information leaflets for all prescription drugs, and to set targets
> for the distribution of these leaflets. In addition to setting target
> distribution goals by specific dates, the proposed rule set criteria
> by which written information would be judged to determine whether it
> was useful and should, therefore, count toward accomplishment of the
> target goals.
>
> On February 14 and 15, 1996 FDA convened a workshop on the medication
> guide proposal, right here, in this same hotel.
>
> In August of 1996, Congress passed, and the President signed, Public
> Law 104-180 mandating that the private sector be given the opportunity
> to meet distribution and quality standards for voluntarily provided
> written information on prescription drugs. It also directed the
> Secretary of Health and Human Services to facilitate the development
> of a long-range action plan that meets the goals through private
> sector efforts.
>
> The Secretary asked the Keystone Center to convene a steering
> committee to collaboratively develop this action plan. Judy O\'Brien
> from the Keystone Center will describe that process shortly. The
> action plan, accepted by the Secretary in January, 1997, reiterated
> the target goals specified in the federal legislation.
>
> These goals were that by the year 2000 written information would be
> distributed to 75 percent of individuals receiving new prescription
> medications, and by the year 2006 95 percent of individuals receiving
> prescription medications would be given useful written information.
> The action plan generally endorsed the conceptual criteria specified
> in the public law for determining usefulness of medication
> information.
>
> Specifically, the law stated that such materials should be
> scientifically accurate; unbiased in content and tone; sufficiently
> specific and comprehensive; presented in an understandable and legible
> format that is readily comprehensible to consumers; and that it should
> be timely and up to date, and useful, that is, it enables the consumer
> to use the medications properly and appropriately, and they receive
> the maximum benefits of the medication and avoid harm. This action
> plan, including descriptions of the criteria, has been available on
> the Internet ever since the Secretary accepted the plan in January of
> 1997.
>
> Also, consistent with the public law, the action plan called for the
> development of a mechanism to periodically assess the quality of
> written information for patients.
>
> To test the methodology for collecting patient information materials
> and assessing their usefulness, FDA entered into a contract with the
> National Association of Boards of Pharmacy on September 25, 1998. As
> you will hear shortly in more detail from Karen Oster from NABP, the
> contract called for the selection of several state boards of pharmacy
> who would arrange for collecting, from a sample of pharmacies in their
> state, medication information materials given with new prescriptions
> for three commonly prescribed drugs chosen by FDA.
>
> The contract also called for the development of evaluation materials
> to assess the usefulness of the information through application of the
> Keystone action plan criteria. NABP arranged for this work to be done
> by Dr. Bonnie Svarstad, professor of pharmacy at the University of
> Wisconsin. The medication information materials were collected by
> participating state boards in 1999, and the final report from the
> evaluation of these materials was completed on December 21 of 1999.
>
> Later this afternoon Dr. Svarstad will discuss in detail the findings
> of the interim study of the status of useful written prescription drug
> information for patients and how the criteria specified in the action
> plan were employed.
>
> What FDA would like to get from this public meeting, following the
> presentation of the study methodology and results by Dr. Svarstad, is
> feedback from you on the methodology and the results. This feedback
> will then be used for development of the assessment mechanism that
> will be used by the agency to see if the year 2000 goal has been met.
>
> Specifically, FDA is seeking comments on several issues and you have a
> copy of these seven issues in your folder that was given to you. The
> first two we consider extremely important.
>
> The first one, what should the minimum standard or threshold be that
> must be met for written information to be considered useful? In other
> words, what is a passing grade?
>
> The second one, should certain criteria derived from the action plan
> recommendations be given more weight than others? If so, which
> criteria should be weighted more strongly than others, such as should
> the risk information be weighted more heavily than the storage
> information found in these leaflets? In the breakout groups, we will
> ask all groups to consider those first two.
>
> The remaining issues will be divided up amongst those groups. We will
> let those groups choose which ones they want to address. The first two
> we are asking every breakout group to address. Those two are the most
> important feedback information that we are seeking.
>
> The third one, are the evaluation forms an accurate translation of the
> action plan\'s criteria? Did the researchers take the criteria and the
> sub-criteria and come up with statements that they looked at in the
> materials and applied them properly?
>
> Should the assessment include additional criteria or types of
> information and, if so, what?
>
> Next, should there be a more detailed assessment of factors affecting
> readability and legibility for consumers, again, type size, style,
> spacing and contrast? As you will see later, most of that information
> is in 10-point type size.
>
> The next one, should the evaluation panel include consumers? As you
> will see in the discussions later, they were mostly educators and a
> practitioner. There were no consumers on the evaluation panel. And,
> what backgrounds should these consumers have if they were to be a
> reviewer of this information? How should they be involved in the
> evaluation process? Should they be an equal panel member?
>
> Next, this report collected information from U.S. retail pharmacies
> where one could walk in and present three prescriptions. Nothing was
> collected from the mail-order side of pharmacy business or from those
> non-retail pharmacies such as the pharmacy benefit managers. How do we
> collect information from those? We will ask for feedback from there.
>
> I want to thank you for joining us for this important meeting, and I
> look forward to working with you over the next day and a half to
> address these and other important issues. Thank you.
>
> DR. OSTROVE: I need a show of hands at this point, how many people are
> suffering from postprandial drowsiness? Nobody! This is very good. I
> saw one hand. That is very brave, Linda, thank you very much.
>
> Now that you have a sense of why we are here today, I thought it would
> be helpful for you to get kind of a better idea of how we structured
> the workshop. So, just very briefly. Basically, today is pretty
> didactic. We have already heard a little bit about the history of ind
> of the overall FDA involvement in this area. We are going to hear
> about the history of the interim assessment and, actually, we are
> going to hear a little bit about the action plan development itself,
> which will be followed by a report by Dr. Bonnie Svarstad of the study
> methodology and results.
>
> We will have a break in the middle. So, you know, if you do happen to
> suffer a little bit from that drowsiness or you need a bio-break,
> there will be a time for that.
>
> Tomorrow, the second day, armed with your understanding of the study
> and what led up to it, that is when we want you to talk to us and to
> give us your feedback, your thoughts. So, what we will be doing
> tomorrow is breaking into small groups, and tomorrow morning I will
> kind of explain what the little numbers on your badges mean, which
> basically has to do with breaking up into smaller groups, and each of
> these groups is going to have the opportunity, as Tom said, to address
> some of the questions that he has raised this morning that are in your
> packet and which you will also find in the Federal Register notice, if
> you happen to see it, announcing the workshop, and any of the older
> letters that went out to people who are part of the Keystone
> facilitated process and to those who were part of the public workshop
> that was held back in 1996.
>
> So, you may want to keep those questions in mind even today as you are
> listening to the history and to the study and the methodology because
> they may help you \-- those questions may help you to crystalize the
> thoughts that you want to express tomorrow in the small breakout
> sessions. Of course, at the end of the day tomorrow, after lunch
> basically, we will present the results of the breakout sessions.
>
> So, with that in mind, here is the plan for the rest of today: First,
> you are going to hear from Judy O\'Brien, who is an associate
> facilitator with the Keystone Center, the group that facilitated the
> process that resulted in the action plan \-- and, I am not going to
> give you the whole title because it never comes out right. You all
> know what I am talking about anyway.
>
> Currently, Judy is the Director of the Keystone\'s Energy Program but,
> more significant to today\'s presentation, Judy worked with Keystone
> during the development of the action plan. Today she is going to
> discuss the process and some of her observations.
>
> Judy will be followed by Karen Oster, who is assistant to the
> executive director of the National Association of Boards of Pharmacy,
> Carmen Catizone \-- that is not quite as long as Division of
> Marketing, blah, blah, blah, but it is getting there. Karen was
> NABP\'s project manager for the evaluation and she was the contact
> point for the expert panel evaluations that were led by Dr. Svarstad
> of the University of Wisconsin. Karen has been an invaluable resource
> and member of this effort.
>
> After Karen addresses NABP\'s role in the effort and why they were
> selected as the contractor, she will introduce Dr. Svarstad. Dr.
> Svarstad will speak for about an hour and then we will take this break
> that I referred to which, I am sure, at that point will be definitely
> well deserved and probably needed. Then, when we return, she will
> finish her presentation.
>
> When Dr. Svarstad has finished with her presentation, I will take
> questions from the audience and I will direct them to the appropriate
> individual. We thought it would make basically for a better flow of
> information if we didn\'t basically kind of take questions and chop up
> the presentation. The only limitation is that Judy O\'Brien can only
> be here for the early afternoon, but she has assured us that since Tom
> was also there at the time and \-- what would you say, about a quarter
> or about half the people here were there as well, probably they would
> be able to answer any questions so we should be okay.
>
> Now, for those of you who leave before we get into the question and
> answer period, please make sure that when you return tomorrow \--
> because I am going to remind you about this again \-- you bring the
> report and the questions that the FDA has presented that are in your
> folder. You know, we will have extras but not if everybody forgets. We
> will have wall charts up as well with the criteria and with the
> questions. So, don\'t worry about it. And, we will even have
> facilitators for your groups. So, Judy, you are up.
>
> **History of Recent Private-Sector Information Efforts**
>
> MS. O\'BRIEN: Thank you, Nancy, for that introduction.
>
> \[Slide\]
>
> My name is Judy O\'Brien, as has been mentioned, and I am a
> facilitator for the Keystone Center in our Washington, DC office. I am
> here today to help provide a little bit of context and background on
> the process that led up to the development of the Action Plan for the
> Provision of Useful Prescription and Medicine Information \-- one of
> our longest titles, I think, for one of our reports.
>
> I was a member of the four-facilitator project team that worked with
> this group of stakeholders in this short, very intensive process. When
> I was asked to do this I got immediately nervous because it was 1996
> and my memory doesn\'t necessarily go back that far so, hopefully, I
> will have recreated the process accurately and, if I have gotten
> something wrong, please feel free to let me know that but not right
> now.
>
> \[Slide\]
>
> I have structured my comments today into five general areas. First, I
> am just going to provide you with a brief background of the Keystone
> Center for those of you who aren\'t familiar with our organization.
> Then, a little bit on how we became involved in this effort; then some
> information on the convening process and how we pulled together this
> 34-person steering committee; and some information on the steering
> committee process itself and some of the outcomes that came of it;
> then just some general impressions that I had as part of the
> facilitation team.
>
> \[Slide\]
>
> So first, who is Keystone? Keystone is a non-profit consensus-building
> alternative dispute resolution organization. We are based out in
> Keystone, Colorado and have a small office here, in Washington, DC.
>
> We have two divisions in the organization, the science and public
> policy program, which is our facilitation branch or facilitation arm,
> and the Keystone science goal which is hands-on science-based
> education for students and teachers, and they are out in Colorado as
> well.
>
> Our role is to be the neutral third-party convener of multi-party
> stakeholder processes. In other words, we consider ourselves to be
> sort of the process experts to help bring folks together to look at
> public policy issues and try to resolve them through dialogue and
> consensus building \-- so, kind of in a different way than you would
> normally approach such issues. Our outcomes vary according to process
> but generally typically result in consensus-based policy
> recommendations for policy makers and stakeholders as well.
>
> \[Slide\]
>
> So, why was the Keystone Center involved in this process? How did we
> get roped into this very, very intense process from the beginning? HHS
> had had previous involvement with the Keystone Center through other
> dialogue processes and were happy with the results of that, so they
> asked us to facilitate this effort. Our proven experience of bringing
> together diverse groups to reach agreement was very appealing to FDA
> and to the Secretary, and it was one reason why they asked us to do
> this because they wanted to develop one action plan. So, in doing
> that, you know, they needed a facilitator to help move that process
> along.
>
> So, as a result of that, the August 26, 1996 FR notice included
> Keystone as the convener and the contact for this collaborative effort
> which, as you know, was mandated by Congress.
>
> \[Slide\]
>
> Our role was to convene the steering committee process, serve as the
> facilitator, provide organization and logistical services and
> expertise, and essentially help the group forge an agreement to
> develop this one, single action plan.
>
> \[Slide\]
>
> FDA\'s role was interesting. On the other hand, they actually
> consciously chose to take a step back from the process and allow the
> facilitators to work with the steering committee together towards
> achieving these goals. They were present at the meetings, provided
> technical support to the group, but did not serve as a member of the
> steering committee itself.
>
> \[Slide\]
>
> The convening \-- as the whole process was, was also in a very tight
> time frame; I think we probably pulled it together in less than two
> weeks. But, the FR notice itself called for a submission of letters of
> interested parties and included several points of information that
> they needed to provide to us.
>
> Those letters came to the Keystone Center and basically FDA gave
> Keystone the leeway to determine who the key stakeholders should be,
> who were the folks who needed to be at the table, and through a review
> of the letters and talking to other individuals we essentially came up
> with a group of 34 stakeholders. And, our definition of a key
> stakeholder are those that have the sort of ability to stop the
> agreement from moving forward and, in our minds, those were the people
> that needed to be at the table. So, that resulted in a group of 34
> individuals with varying perspectives but I think a common goal of
> working together to try to achieve this plan.
>
> \[Slide\]
>
> The steering committee process itself was very intense, as I
> mentioned. One hundred and twenty days was what Congress mandated for
> the group to have but, by the time we actually pulled together the
> group and had our first meeting, which was September 18, we had much
> less time than that. I think it was about a 75-day process from start
> to finish, from the first meeting to the final meeting which in any
> other type of process would have been very difficult to do. The
> steering committee met seven times in the DC area, which equated to
> about every two weeks. This group of folks was torn away from their
> regular daily jobs and coming to meet in various hotels like this in
> the Washington, DC area.
>
> \[Slide\]
>
> In between that, we also had work groups. We had four work groups set
> up. At the first meeting we set up these four groups that were
> revolving around the goals assessment, development of guidances and
> implementation. So, these groups got together in between the steering
> committee meetings for face-to-face meetings as well as conference
> calls. They developed a work plan \-- you know, a plan of action and
> how they were going to move forward to try to address the six elements
> that were required in the law.
>
> \[Slide\]
>
> The steering committee meetings then provided an opportunity for the
> work groups to report back to the larger group on the progress that
> they were making on their draft documents and receive feedback from
> the group, and that would enable them to continue to move forward with
> the process.
>
> \[Slide\]
>
> These were public meetings also, and that provided an opportunity for
> additional comment and guidance from those folks who were on the
> steering committee. I believe they were able to provide any comments
> to the steering committee members who would then relay them to the
> rest of the group, and that provided some additional insight for the
> group.
>
> At the end of the process the members were asked to submit letters to
> the Keystone Center that essentially served as a mechanism to
> determine their level of support for the plan and who, sort of, signed
> on at the end of the day.
>
> \[Slide\]
>
> Outcome \-- there were sort of three outcomes that I will just mention
> here. The first was consensus. The ultimate goal of this group was to
> develop consensus-based recommendations for the Secretary. And
> according to the report \-- this is the language right out of the
> report, but members of the committee agree to support the plan as a
> total package, although individually they may not have an equal amount
> of enthusiasm for each idea or recommendation. And, that is
> essentially the definition of consensus that the Keystone Center uses,
> that is, basically you can live with the whole package but you might
> not like each piece within it, and that is what we based that
> agreement on.
>
> \[Slide\]
>
> Implementation was another outcome, and on the steering committee
> there was definite agreement that there was going to be a need for
> support of the plan by the stakeholders in order for it to be
> successfully implemented. So, within the report there is language to
> the effect that each organization agrees to commit to supporting the
> plan\'s implementation in a manner consistent with its organizational
> scope and goals. So, everyone wasn\'t going to do it the same way but
> everyone needed to have some level of support for the plan.
>
> \[Slide\]
>
> Then, there was also a recommendation for a transition group to be
> formed. The purpose of this was to further develop the implementation
> strategy to address some of the issues that were unresolved or that
> there just wasn\'t enough time to address. The specific tasks for that
> group were outlined in the report.
>
> \[Slide\]
>
> Some of our overall impressions of the whole process: This really was
> a unique process from a facilitation perspective. The issues were
> definitely ripe. The work that was done previously, the years of work
> on these issues, as Tom talked about, really set the stage for this
> group to be able to move forward.
>
> There were definitely incentives in place, and that always helps in a
> process like this if you have such a short time frame. You know, this
> was a congressionally mandated effort. There was support from the
> Secretary of HHS. There was a hammer in place in case an agreeable
> solution wasn\'t reached. And, the stakeholders really wanted to do
> the right thing but they had different ways of going about doing it,
> and they needed something to sort of push it along and to kind of move
> the whole process forward from our perspective, and they needed this
> catalyst and I think this process served as that catalyst.
>
> \[Slide\]
>
> All of these factors really gave Keystone the latitude to help this
> group reach consensus and submit the plan according to schedule, and
> it really was an interesting, fun project, as crazy and sick as that
> might sound from the perspective of those who participated in it, but
> it was a great group of people who were really committed, and it was a
> lot of hard work, and I think if it hadn\'t been for the commitment of
> the steering committee members to really try to forge an agreement it
> wouldn\'t have happened.
>
> So, from our perspective as the facilitation team, it really was a
> successful effort and it was an interesting experience for us to be
> involved in. And, that concludes my comments.
>
> **Background on the Patient Information Project**
>
> MS. OSTER: Thank you for the nice introduction, Nancy. I would like to
> begin by thanking a few key individuals for their help with this
> project. A great big thank you goes out to Dr. Ellen Tabak of the FDA,
> Dr. Bonnie Svarstad and Dr. Dara Bultman of the University of
> Wisconsin. The project would not have been a success without your
> extraordinary effort.
>
> \[Slide\]
>
> I am here today to share some background information of the National
> Association of Boards of Pharmacy and how we helped to develop and
> carry out this research project. Our primary interest in this study
> was twofold: We wanted to ascertain the degree to which patient
> information was being distributed in regard to the \"Healthy People
> 2000\" objectives and, secondly, examine whether the information being
> distributed was useful to patients.
>
> The answer to these questions continues the interest of the NABP and
> the state boards of pharmacy in patient care activities such as
> patient counseling. Earlier studies conducted by NABP found that the
> offer to counsel and actual delivery of counseling to patients was
> less than expected and hoped for by regulators and patients. It was
> our hope that deficiencies in these areas were being met in some small
> way by the distribution of useful patient information.
>
> We also wanted to help the pharmacists to use the information and
> increase their interaction with patients in positive ways, directly or
> indirectly stimulating the interaction and dialogue between the
> patient and pharmacist even in situations where workload or lack of
> initiative on the part of the pharmacist arose as barriers.
>
> \[Slide\]
>
> As Tom mentioned earlier, back in August of 1996 Public Law 104-180
> mandating the private sector be given the opportunity to meet
> distribution and quality goals for written patient prescription
> medicine information was enacted. This law also directed that the
> Secretary of Health and Human Services facilitate the development of a
> long-range comprehensive action plan to meet these goals through
> private sector efforts.
>
> \[Slide\]
>
> As a result of this law being put into effect, the FDA asked NABP to
> appoint someone to a steering committee, made up of interested
> parties, that would develop the action plan. Carmen Catizone,
> executive director to the secretary of NABP, was appointed to the
> committee.
>
> The action plan called for the development of a mechanism to
> periodically assess the quality of written prescription information
> for patients. The steering committee thought the appropriate mechanism
> to assess the quality of the information being provided to consumers
> by pharmacists should be developed in conjunction with the individual
> state boards of pharmacy. The rationale reasoned that the actions of
> the pharmacist of this regard and subsequent monitoring would be based
> on state laws and regulations. It also followed that the study results
> indicated that if changes in practice needed to occur, then these
> changes must be made in concert with state pharmacy practice acts and
> regulations.
>
> The state boards of pharmacies, colleges of pharmacy and pharmacy
> trade associations were called on to continue to educate pharmacists
> about the importance of oral counseling on prescription medication and
> use. \"The Healthy People 2000\" goals which were affirmed by the
> steering committee were to provide useful written information to 75
> percent of individuals receiving new prescriptions by the year 2000
> and 95 percent of individuals by the year 2006.
>
> \[Slide\]
>
> The action plan developed by the steering committee called for
> prescription medicine information to be scientifically accurate,
> unbiased in content and tone, sufficiently specific and comprehensive,
> presented in understandable and legible format that is readily
> comprehensive to consumers, timely and up to date and useful.
>
> \[Slide\]
>
> After the action plan was developed and because of the contacts the
> state boards of pharmacy had with the states and NAPB had with the
> states, the FDA invited NABP to submit a formal request for proposal
> for the evaluation of prescription drug information materials study.
> Dr. Bonnie Svarstad, of the University of Wisconsin, was selected as
> the subcontractor on this project because of her expertise in this
> area. The FDA and NABP felt confident that she could effectively carry
> out this study.
>
> On September 4, 1998 the request for proposal was sent in to the FDA.
> The purpose of the work was to collect a sample of written materials
> currently being distributed to patients with receipt of a new
> prescription for selected prescription drugs; then, to evaluate the
> materials according to a protocol from the criteria outlined in the
> action plan and to report the results to the FDA and to the public.
>
> \[Slide\]
>
> The request for proposal outlined time frames and duties to be carried
> out by the subcontractor, Bonnie Svarstad.
>
> \[Slide\]
>
> The duties outlined in the RFP will be described in detail by Dr.
> Svarstad during her presentation.
>
> \[Slide\]
>
> The request for proposal was accepted by the FDA and became a fully
> executed contract on September 25, 1998. Ten states were selected for
> the study. The selection was based on their size and location
> throughout the United States. The ten states selected had also worked
> very closely with NABP on other projects in the past.
>
> \[Slide\]
>
> NABP held its first conference call with the states on November 3,
> 1998 to discuss the study. Participating in the conference call were
> New York, Arizona, Florida, Texas, Minnesota, Wisconsin, North
> Carolina and Ohio. Outlined in the conference call were the three
> drugs that would be used in the study; how the collection of the data
> would happen, and the use of state inspectors for same; the cost of
> the drugs and state reimbursement; and chain and independent pharmacy
> participation.
>
> After the call, all states were sent a timetable for the project and
> background information. Svarstad then sent NABP instructions for
> patient observers, which included \"do\'s\" and \"don\'t\'s\" when
> going into a pharmacy, and answers to questions normally asked by the
> pharmacist. An observer reporting form was included with this
> information.
>
> At the end of November Svarstad sent NABP a list of possible expert
> panelists. At this time, NABP was working with each of the states in
> an attempt to keep them in the study. Some of the states had to drop
> out of the study due to internal conflicts and lack of time, resources
> and staff.
>
> On March 2, 1999 Svarstad sent in an interim report on the study. The
> report stated that the patient observer visits were complete in the
> pilot study State of Wisconsin. The report also stated that final
> revisions to the protocol to be used in the final study were being
> made. The final list of participating states with contact names was
> then sent to Svarstad. NABP was working on collecting lists of
> pharmacies in each of the states at this time as well.
>
> \[Slide\]
>
> On April 8, 1999 a conference call was held with the participating
> states, Bonnie Svarstad and Dara Bultman to discuss the instructions
> for state coordinators and instructions for patient observers. The
> state contacts had many questions about what was needed from them. The
> discussion was about list of pharmacies, training of patient observers
> and the random sampling plan.
>
> After the call, Bonnie Svarstad and Dara Bultman helped NABP and the
> states with getting the sampling of pharmacies from each state
> compiled for use in the study.
>
> \[Slide\]
>
> Another conference call was held on April 19, 1999. The team of
> Svarstad and Bultman continued to help gather additional observer
> help, help with medical doctors and identities and random sampling
> information. Bultman assisted NABP with a lot of the phone work and
> follow-up questions that the states had regarding the project.
>
> In May, packets of information were sent out by Svarstad to the expert
> panelists clarifying their role in the study. The states gathered
> information from June of 1999 to November of 1999. NABP coordinated
> the reimbursement for prescription and observers\' time.
>
> \[Slide\]
>
> On December 21, 1999 Dr. Svarstad, principle investigator, and Dr.
> Bultman, project manager, completed their report on the eight-state
> study. That brings us to today and discussion on that study.
>
> \[Slide\]
>
> Now I would like to introduce Dr. Bonnie Svarstad. She is currently a
> professor at the University of Wisconsin, in Madison, Wisconsin, and
> is a nationally known expert in the fields of social and behavioral
> pharmacy and patient medication information. She has focused her
> career on conducting research in social and behavioral pharmacy, with
> emphasis on professional patient communication, patient compliance and
> factors affecting drug prescribing and drug utilization. She has
> published extensively in these areas, has served on numerous national
> scientific boards and panels, and has won many national awards for her
> work. Bonnie?
>
> **1999 Patient Information Assessment**
>
> DR. SVARSTAD: Thank you, Karen.
>
> \[Slide\]
>
> First I would like to thank Karen Oster for that fine introduction and
> to thank the workshop organizers for how this was organized. I very
> much appreciated how efficient it has been, and I am very much looking
> forward to these two days. I think this project would not have been
> possible but for a number of people that I would like to thank before
> getting into the study methodology and results.
>
> I consider it a privilege and honor to have been involved in this
> project. I have had an interest in this area and a concern, like many
> of you here, for twenty-some years. So, for me it is a personal honor
> to be involved at this stage because I think it is so important for
> the public. I think it is important for the pharmacy profession and
> the other health professions, and for other components of our society
> that I think share the concern about getting the most useful
> information to the public so they can use medications safely and
> effectively.
>
> First, let me thank my colleague, Dara Bultman, who cannot be here
> today because she is back home in the communications lab teaching our
> students. The project certainly would not have been possible without
> her help as a project manager.
>
> \[Slide\]
>
> The study, as you know, was done in cooperation with a number of
> parties, and I would like to begin by thanking them. First, the FDA
> \-- there is no doubt that Dr. Ellen Tabak enabled us to finish this
> project. Her professional expertise in the area of research, her quiet
> but firm insistence on deadlines and support, and trying to find the
> resources to do all of this in twelve months were unparalleled. I
> personally thank her for a wonderful job in helping us and, of course,
> Nancy Ostrove, Tom and the others at your office. I think anyone who
> is an academe always has a little bit of concern about getting
> involved in a project that has political or social issues involved
> because there are so many parties involved, but I found that their
> support and involvement in this study was just superb.
>
> NABP \-- I would like to thank Karen Oster and Carmen Catizone. We
> very much appreciated their involvement and support throughout the
> project, who made this project one that was very interesting and
> exciting for us, and helped us to make connection to all the
> interested parties, the state boards of pharmacy, etc.
>
> They are not here, I am sure, but the state coordinators and
> inspectors who collected this data \-- I wish we had a way of thanking
> them for all the work they did, but they were a critical key here
> because they were the ones who went out to the pharmacies and, as much
> as possible, collected data in an objective and in as fair way as
> possible. We very much appreciate their input.
>
> Then the expert panelists \-- and I would like identify them \-- also
> played a key role in this project in terms of helping to develop the
> forms that we used, reviewing and commenting on those forms, providing
> guidance to us in terms of study methodology, and giving us feedback
> and support all along the way. Their role also was critical and made
> it possible for us to do this project.
>
> \[Slide\]
>
> Let me say a few words and identify those panelists because of their
> critical role here, and I would like you to know a little bit about
> them, without going too much into their background but I would like
> you to know this because they played a very important role in
> evaluating all this information that came flowing into Wisconsin at
> one point.
>
> Heidi Anderson Harper is a pharmacist and a member of the faculty at
> Auburn University College of Pharmacy. She also happens now, I think,
> to be the head of the Social Administrative Sciences Division. She has
> had a career-long interest in the design and evaluation of drug
> information materials for the public, has worked with many pharmacies
> as well as public agencies to try to do a better job in this area.
>
> Robert Beardsley is at the University of Maryland. Bob is very
> well-known nationally for his work in this area. He has done research
> in the area. He is a pharmacist, has his Ph.D. and is also an
> associate dean. Bob and his colleagues, Carol Kimberlin and William
> Tindall, are authors of one of the most well-known and recognized
> textbooks on communication for pharmacists. So, he has taken a
> career-long interest not only in improving information but in figuring
> out ways to disseminate it to our students so that they are prepared
> to do this once they get out into practice. So, we are happy to have
> him involved.
>
> Chester A. Bond is a Pharm.D. He is now at Texas Tech. University. He
> used to be at the University of Wisconsin. He is a specialist in
> psychiatric pharmacy, continues to practice pharmacy, and is an
> associate dean. He is an author of some of the very classic
> evaluations that have been done to improve or enhance the
> pharmacist\'s role. So, he played a key role.
>
> Marie Gardner \-- I am sure of you know Marie. Marie is a Pharm.D. and
> she is at the University of Arizona. She has taken leadership
> nationwide in figuring out ways to better prepare pharmacists for
> their role as educators, and has made many innovations in this area.
> She continues to practice pharmacy.
>
> Carol Kimberlin is not a pharmacist. She is an educational
> psychologist, and I think would bring a consumer perspective to this
> panel. Carol is at the University of Florida, at Gainsville, and in
> the School of Pharmacy. She is a full professor, and has been involved
> for a number of years in communications research and has done landmark
> studies in teaching pharmacists and evaluating pharmacists\' role as
> educators.
>
> Duane Kirking has a Pharm.D. and a Ph.D. He is also in the audience.
> Would you stand, Duane? Thank you. Duane was a member of the panel.
> Duane is at the University of Michigan, and has a long-time interest
> in drug information. We selected him also because he is a careful
> reviewer of materials. I have had experience with that as an associate
> editor myself, and I thought he would provide a critical perspective.
> He also has a good understanding and expertise in the area of drug
> utilization, review and statistics.
>
> Sharlea Leatherwood is a pharmacist, and she was a very good addition
> to the panel, I believe, and was suggested by NABP. She is from
> Missouri, community pharmacist and past president of the Missouri
> Board of Pharmacy. So, she brings another perspective.
>
> Helene Lipton \-- Helen brings a consumer perspective because she is
> not a pharmacist, however, she has been at the University of
> California in the School of Pharmacy for a number of years, and is now
> a full professor and has a strong interest and history in health
> policy studies and, in particular, has an interest in drug use in the
> elderly. So, she added another perspective.
>
> Betsy Sleath is a pharmacist, has her Ph.D. in social and behavioral
> pharmacy, is a registered pharmacist, and she is now at the University
> of North Carolina. She has also done extensive research in
> communications.
>
> So, we tried to get a panel then that had some linkage to pharmacy
> practice and some experience and commitment to careful, objective
> evaluation of communication and drug information so that we would get
> as many \-- I think a strong, diverse panel, if you will, to evaluate
> this as carefully as we could.
>
> I should also say a thing about our project manager, Dara Bultman.
> Dara came to us from Purdue. She has a master\'s in clinical pharmacy,
> a Ph.D. in social science, but she has continued to practice pharmacy,
> and continues to practice to this day. I think she has almost over
> fifteen years of pharmacy practice experience, so brings with her,
> again, this integration of pharmacy practice as well as the science of
> trying to evaluate drug information.
>
> I suppose I should say how I got interested in this before going too
> far, My own background is in medical sociology. I came to pharmacy
> through the back door, so to speak. I guess my closest connection is
> that my uncle is a pharmacist, but I did my Ph.D. studies on
> doctor-patient communication, back in the early \'70s.
>
> I think I always remember the day that I discovered that as a
> non-pharmacist, non-health professional what I was trying to do in my
> very first study was to figure out how we could measure patient
> compliance with the prescription regimen. As many of you know, that is
> now a hot topic and a major public health problem \-- patients not
> taking their medications as prescribed. Well, I was trying to do one
> of the earlier studies. I was trying to develop a measure for patient
> compliance, and I figured out the obvious, I guess, that you cannot
> measure patient compliance unless you know how the drug is supposed to
> be taken. Right? Right!
>
> Well, part of my design had involved sitting in on the doctor-patient
> encounter. In fact, I watched over 300 encounters between doctors and
> patients and took very careful notes. The plan had been for me to then
> train interviewers to go out and interview patients about what they
> understood from that encounter. I went back to my office usually, sat
> down, lo and behold, I could not figure out how those drugs were
> supposed to be taken.
>
> At the time, I said is this a drug that should be taken every day or
> just as needed? You couldn\'t always tell because the label didn\'t
> always make it clear. Should the drug be taken for so many days?
> Should it be taken continuously? I could not always tell based on what
> I had heard the doctor say, based on what I had reviewed from the
> medical record, and in those days there was no written information
> being handed out to patients.
>
> I think the light bulb went on that there was a gap here in
> communication. Physicians simply were not providing what seemed to me
> sufficient information to the patient so that he or she could figure
> out how the medication should be used. Sure enough, when we went to
> interview the patients, they were quite confused. They made many
> mistakes. I think that is where I developed a career-long interest in
> this issue of information \-- how can we do a better job of it so that
> the consumer is less confused, more knowledgeable, more comfortable
> with their medications?
>
> Over the years I have done a number of studies. This was probably one
> of the fastest ones, in a year, but I think it is kind of a
> culmination, if you will, of a number of studies that we have done in
> VA pharmacies, outpatient pharmacies, community pharmacies and a
> variety of circumstances. So, we are pulling together, I hope, methods
> that we have used over the years, and I look forward very much to your
> comments on the methods that we used, in the next day or so.
>
> \[Slide\]
>
> Briefly an overview of past studies, without going into too much
> detail here. I think, as you know, some of you or many of you, there
> have been a number of studies in this area but they were largely based
> on patient report. Certainly, I think some of the more important
> studies \-- the most important studies have been done by Lou Morris
> and his colleagues here, at FDA.
>
> I think these studies have been interesting because if you look at
> them in terms of pharmacy at least, the reported rates of written
> information transmission have increased four-fold from 1982 until
> 1994. Back in 1982, one of the first studies that they reported showed
> that only 16 percent of patients reported getting some kind of written
> information. In 1994 that had jumped up to about 59 percent. So, we
> have seen something that has gradually been changing over time, and I
> think we need to keep that in mind as we look at these study results
> because we are looking at a change over probably several decades here,
> and changing views about what is appropriate, changing views, changing
> technology \-- change.
>
> While there have been a number of studies, few of these studies have
> evaluated the quality of written information per se, aside from just
> reporting on the number of people who get it. So, we felt that this
> was a very interesting study in this respect, in the sense that it
> gave us an opportunity to look carefully at the usefulness or quality
> of information.
>
> I should point out incidentally to you, if you are not aware of it,
> that when you look at the rates of written information out of the
> physician\'s office, the rates are quite low. In 1982 only 5 percent
> of the patients were getting written information from their physician,
> according to these studies. In 1994, about 15 percent of patients were
> getting written information from their physician. So, the change that
> we have seen in pharmacy is pretty remarkable.
>
> \[Slide\]
>
> Now, how does this study differ from other studies? Data collected by
> state inspectors \-- while there have been studies using the shopper
> technique, or having someone enter the pharmacy to collect
> information, I think this is one of the first studies or certainly one
> of the larger studies using state inspectors or, if not state
> inspectors, other persons who have been trained to collect written
> and/or oral information in the pharmacy.
>
> This was important because I think using state inspectors enabled us
> to minimize the recall problem that we would see if we asked consumers
> to report. We wanted to minimize that as much as possible. Also, I
> think we wanted to standardize the stimulus to the pharmacist because
> consumers have very different interests in information and they are
> very much likely to have different approaches in terms of coming into
> the pharmacy. Some people will ask for information, others will not.
> So, we wanted to standardize that and make it as uniform as possible.
>
> Pharmacies, as you heard from Karen, were sampled in eight states, and
> those eight states represent four different regions of the United
> states. These are not randomly selected but they do represent
> different areas, I think, in the country so that we at least have some
> diversity.
>
> The criteria \-- I think this is the first time that I am aware of
> that explicit criteria were developed, in this case from the action
> plan, so that they are visible to everyone. Every one can comment and
> see what we are doing. I think it is a much better technique because
> it can be subject to consensus building and, certainly, an attempt can
> then be made to build or develop standardized forms that can be tested
> in terms of their reliability. I will comment a little bit about that
> in a moment. It was also the first time that we had a national panel
> evaluating the information. So, this was clearly something that was
> unique.
>
> \[Slide\]
>
> The questions that we are going to deal with today are in the report,
> and I apologize to those of you who have done your homework and read
> the report but we will highlight them. First, what percentage of
> patients actually get information, that is, the frequency of any
> information getting to persons regardless of its quality?
>
> Secondly, how do experts rate the quality of that? Then we will take a
> look at some of the criteria, the specific criteria to see whether
> certain criteria are being met better than others. That may provide
> some further guidance or ideas for improvement.
>
> \[Slide\]
>
> In terms of the methodology, I would like to highlight a few points
> about sampling, the protocol itself, the pilot study, the forms, and
> our attempt to evaluate inter-rater reliability, and a little bit
> about how we accomplished the data processing.
>
> \[Slide\]
>
> First, let\'s talk about sampling. NABP, as Karen explained and I
> don\'t need to reiterate, there were selected ten states to start out
> with. We ended up with eight states that agreed to participate in
> 1999, and they are listed there. I think thanks to her introduction,
> which was very clear, I don\'t need to spend more time on this unless
> there are questions later this afternoon about what the implications
> of this are.
>
> \[Slide\]
>
> The next issue, and probably one of the hardest issues for us, the
> most difficult was how to select these pharmacies because each state
> has a different system. Some states have their pharmacies all
> available on a disc; others had hard copy. Some states have an
> extremely large number of pharmacies; others were smaller. So, this
> was quite a task. Illinois and Washington, we were able to sample
> statewide, that is, to randomly sample all pharmacies in that state to
> get forty pharmacies. That was not possible in the other states due to
> limited resources, inspector time, etc.
>
> So, other states sampled within certain geographic regions that we
> helped them select, and usually by zip code. The states varied
> somewhat by the number of regions that they were able to sample
> within, with some sampling one area, some sampling four to five areas,
> and others sampling seven areas. Within these regions though, we had
> what would be called in the business systematic random sample. That
> is, you start out in a random way, either table of random numbers or
> through computer software which helps you select the pharmacies
> randomly. This is very important, obviously, because we would not want
> inspectors or state representatives to either be selecting pharmacies
> that they are worried about or we don\'t want them to be selecting
> pharmacies that they think will put on a good face for the state. We
> really wanted to see a random sample so that we would get as close to
> a true picture as possible. And, I think we were able to accomplish
> that in a short period of time, which is very important.
>
> Because Wisconsin started with a pilot study, our sample was thirty
> pharmacies and in the other states we had forty pharmacies. So, there
> was a total of over 300 pharmacies in this final study.
>
> \[Slide\]
>
> Now, the observer protocol \-- we referred to the inspectors as
> patient observers for purposes of this discussion. These observers had
> minimal training, although they did have some training. We sent out
> manuals to them and we sent out commonly asked questions. Some of
> those materials are included in the appendix of the report.
>
> We encouraged the state coordinator to do some role-playing with these
> inspectors so that questions could come out, and if they did have
> questions they could get those questions back to us so that Dara or
> myself could respond, and the effort here was to, as much as possible,
> get uniformity without having a massively expensive study where we
> would go out to all of these states and conduct in person or
> face-to-face training. So, they did have some training and certainly
> they had a protocol that they were to follow.
>
> The majority of these individuals were, in fact, inspectors from
> outside the area where they were responsible in their state. So, if
> someone was responsible for inspecting in Upstate New York, we did not
> have them collect data in their region. We asked that they collect in
> another region so they would not be recognized, for obvious reasons,
> and, for the most part, I think they were able to do this without
> being recognized. On the observation protocol we had an item, \"did
> you recognize anybody in the pharmacy? Or, do you think anyone
> recognized you?\" If they did, that is if they recognized anyone or if
> someone recognized them, they were instructed to leave the pharmacy as
> diplomatically as possible and to replace that pharmacy.
>
> In some cases the state did not have enough inspectors to do this
> observation within the short time frame so we assisted them, if they
> needed it, in finding individuals to help them with the process, and
> they then authorized these individuals to do the inspection.
>
> Seventy-one percent of the visits were made by men, with an age range
> from 21 to 85 years. The male/female distribution here probably
> reflects an inspector employment situation. So, it was not possible
> for us to control or to dictate the demographic characteristics of the
> inspectors although we might like to see more diversity. The age range
> there of the older individuals probably reflects the non-inspectors
> who were hired to assist. Ultimately, we will be able to see whether
> or not any of the results vary by the characteristics of the observer
> because the observation form included some very brief items that asked
> the observer to give his or her age, gender and we, of course, know
> who was an inspector and who wasn\'t an inspector. So, we will be able
> to look at that. My impression though is that this will not be an
> issue.
>
> \[Slide\]
>
> The standard scenario \-- the observers had a defined health and
> medication history, and I think that information is provided in the
> report. Certainly, if you have other questions about that, you can ask
> later on. But, the idea here was to give the pharmacy or the
> pharmacist the easiest possible situation, that is, an uncomplicated
> health and medication history. The inspectors or patient observers
> were given very specific instructions: if you are asked this, this is
> what you say. If you are asked this, this is what you say. The idea
> here, of course, is to try to standardize the background of the
> patient observer so that they did not have other medications; they did
> not have other health problems; and they did not have any other
> extenuating circumstances so that the pharmacists would all be getting
> the same situation.
>
> All of the inspectors were given cash to cover these three
> prescriptions so that when the issue came up of how you are going to
> pay for this, they could say that they were paying for it in cash and
> they will collect, if they have insurance, somewhere else. We did not
> want to interrupt the dispensing process, nor did we want to place any
> undue demand on the pharmacy, so we tried to keep it as simple as
> possible.
>
> These medications were selected by NABP in collaboration with FDA. So,
> we didn\'t really have a role in this, but I think they did a good job
> of selecting them because you have a short-term medication \--
> amoxicillin. You have a medication that could be used in a various of
> ways and for a variety of conditions, which kind of calls for
> counseling. Then, of course, you have a chronic medication for
> depression that has other issues involved. So, I think it is
> interesting then that we were able to get data on three somewhat
> different types of medication.
>
> \[Slide\]
>
> Now on the other side, the patient observers were instructed again to
> give the pharmacist or the pharmacy an opportunity to give information
> spontaneously, voluntarily, without a request from the patient. So,
> they were told to be polite and to accept any information that they
> got. So, if the technician or clerk asked them, or the pharmacist,
> \"would you like written information, they would say, \"sure.\" Or,
> \"would you like to talk to the pharmacist?\" \"Sure.\" But they were
> not to seek information. The aim there was to try to be able to get
> findings here which would reflect information given spontaneously, and
> I think this was an important decision because, as you know, certain
> segments of the population may have low expectations, do not want to
> bother pharmacists by asking for things, or perhaps they simply do not
> feel that they deserve more information, or there may be many reasons
> why people do not seek information, and what we want here is whether
> they are getting it spontaneously or voluntarily.
>
> They were also told not to ask questions or to initiate talk, unless
> they were asked a question, and of course they were to respond
> appropriate. Again, we did not want them taking the time of the
> pharmacist or in some way influencing the encounter to keep this
> standardized.
>
> They were encouraged to role-play, as I pointed out earlier, and the
> common questions are listed in the appendix.
>
> Immediately after they were in the pharmacy, they went out to their
> car, transportation, or wherever, and immediately filled out a
> one-page observer form. That form asked them about their
> characteristics, the characteristics of the person with whom they
> spoke, and a few other items that basically asked them to describe the
> situation, including whether or not the pharmacist or person giving
> them the information mentioned it. I didn\'t put that in the report
> but I will try to remember to do that at the end because we have some
> information about how these sheets were actually disseminated by the
> pharmacy.
>
> \[Slide\]
>
> Now, we did a pilot study in Wisconsin basically to test this
> protocol, to see whether it was working, to see whether there were any
> problems in it that we would want to know about before asking seven
> other states to do it. We wanted to find out whether or not one visit
> to a pharmacy yielded reliable information, and basically to do what
> most researchers do in any pilot study, to test the protocol, its
> feasibility, etc.
>
> We also wanted to talk to the pharmacy manager because we wanted to
> know if we were recognized, or if they recognized the observer, or if
> they defined this as something that interrupted their pharmacy. So, we
> got consent from the pharmacy managers. They were told that there
> would be an unannounced visit at some time in the next time period,
> but they were not told when nor were they given any details about the
> scenario. We were surprised to the extent \-- they were very
> cooperative and we got a very wide representation of all types of
> pharmacies \-- very much appreciated.
>
> Two visits were made to each of those pharmacies by two different
> observers on different days. Managers were later interviewed by
> telephone. Dara Bultman did those interviews.
>
> \[Slide\]
>
> We found that one visit was sufficient. That is, if you got
> information on the first visit you were almost certain to get it on
> the second visit. There was high agreement in the frequency of written
> information between visit one and visit two. That told us that was an
> okay decision, in terms of the national study, to only make one visit
> to each pharmacy because you would get pretty much the same results.
>
> We discovered, of course, the obvious. We didn\'t think about it
> beforehand but, in retrospect, we should have anticipated it, that if
> you sampled and happened to get two or more pharmacies within a
> corporate pharmacy or chain pharmacy, you would have to change the
> identities because their computer profile system is such that you
> would be detected is such that you would be detected at the second
> pharmacy as having picked up the same medications in the past month.
> As soon as we discovered that, we were able to make that change and
> make the change in other states so that there was no problem there. We
> also found no other flaws in the design, so we decided that this pilot
> study could serve as data collection in the state.
>
> \[Slide\]
>
> We then went on to develop these patient information evaluation forms,
> and let me say just a few words about how we did that. We had one form
> for each drug, as you know by looking at the report. The final
> versions of those forms are provided in the appendices so you can see
> how that was done.
>
> Each form listed the ten general criteria that we came up with based
> on the Keystone panel report, and for each of these three drugs there
> were 28-32 sub-criteria for evaluating the information. That is, for
> each criterion we had some sub-criteria under it. The reason we did
> that was, of course, to enhance inter-rater reliability or agreement
> from one rater to the next because we want very much to have a tool
> that has good reliability.
>
> \[Slide\]
>
> The criteria, as you know, were developed from the action plan. We did
> the best we could. We focused on ten. You might come up with some
> others that were missed there but I think generally we were pretty
> comfortable with the way it turned out. We used the ten. We refer to
> these ten criteria to make the discussion of this easier. So, we will
> refer to criteria one, two, three, four, five, six, seven, eight,
> nine, ten type of thing.
>
> The forms were revised until all panelists approved. We sent out
> version one, version two, and got comments back from panelists, made
> changes and then sent out the revision. The idea here was probably
> very similar to the Keystone panel. The aim was to build a consensus
> and to also build on the somewhat different perspectives of the
> panelists, although it was remarkable in how much agreement they were.
> But these forms were agreed upon by the panelists. We found that it
> was not necessary to meet. Instead, we did this largely through the
> mail and e-mail and faxing, and so forth in an attempt to keep the
> study\'s costs down, and I think we were able to achieve that although
> we do have some panelist suggestions for the future which I will
> mention later.
>
> \[Slide\]
>
> The ten criteria, as you know \-- scientifically accurate, unbiased.
> You are now beginning to recognize those I think so I won\'t go
> through them, except that I personally think that the Keystone panel
> did a superb job in pulling these together. The reason I say that is,
> in part, because I am familiar with similar initiatives in other
> countries. I know that, for example, in Australia they are now in the
> process of trying to do an evaluation like this, or planning one. So,
> I have seen their criteria which are very similar to these criteria. I
> am also aware that the equivalent of the FDA is interested in doing
> this kind of evaluation in Sweden, and it is remarkable to see the
> similarity of criteria. I think that eventually we could see some
> international collaboration here and expect that to come out of the
> World Health Organization possibly down the line. So, I think the
> leadership that is being shown in the United States is quite
> remarkable here and will be helpful to other countries.
>
> \[Slide\]
>
> We had two basic methods for summarizing or rating these things. We
> called them, for the sake of simplicity, the one-point method and the
> nine-point method. I will just say a few words about each method.
>
> The one-point method basically gives one point for each sub-criterion
> with full or partial adherence. It is very important for you to know
> that this is full or partial adherence. In my view, we are taking a
> somewhat liberal or conservative, depending on your point of view \--
> maybe I shouldn\'t use that language \-- the raters, when rating this,
> gave either a partial check or a full check, partial check if the
> information sheet covered at least partially the items in that
> sub-criterion; a full check if it covered all of it. In reporting the
> results back to you, we are including in here whether it is partial or
> full. In other words, that means that if you have side effects listed
> or five side effects listed, they did not have to include all five in
> the information sheet to get a point. Okay? That is a pretty important
> point because you could find people who may say you should have
> nothing but full adherence, and I think that is a policy issue not a
> scientific issue. That is a policy issue.
>
> We defined, for the sake of summary, five levels of information
> quality using the one-point system: level zero if they got no
> information; level one if they got 1-19 percent of the sub-criteria;
> level two if they got 20-39 percent of the criteria; level three if
> they got 40, etc. So, you kind of see what we are doing here. We are
> trying to say what percentage of the criteria were met in that
> information sheet. So, you can see that when we get to the results.
>
> \[Slide\]
>
> The other method is a global rating. It is not truly global but it is
> more global than the one-point method. That is the nine-point method.
> One to nine points for each general criterion, each of the ten
> criteria. There you have three levels of adherence. If they gave it a
> 1-3, we considered that to be low adherence. If they gave it a 4-6, we
> considered that moderate adherence, and 7-9 high adherence. A 7-9
> might be if the information sheet really adhered to the sub-criteria
> very well. Okay?
>
> You can see how this was done. When you go to the appendices and you
> look at the rating form, you will see 1-9 and those are ratings that
> the rater put on there.
>
> \[Slide\]
>
> Inter-rater reliability \-- panelists were assigned to three subgroups
> by drug. So, we had an amoxicillin subgroup, an ibuprofen subgroup and
> a Paxil subgroup. The reason we did that I think was to enhance
> reliability, to make it easier for the panelists because once they
> became familiar with the criteria it would enable them to do an
> effective job on this. To make them try to review all drugs I think
> would be quite difficult and would probably require more time than we
> had. So, we were trying to streamline this as much as possible.
>
> Each subgroup included one practitioner and two drug information
> specialists. We are eventually going to evaluate reliability in this
> way to see if the practitioners had different views than the drug
> information people. But this is a method that is used in other health
> professions when rating practices. So, we were borrowing and building,
> if you will, a little bit on what other health professions do when
> rating the quality of a practice. In this case we are evaluating
> information practices. Panelists for this reliability study
> independently reviewed four items.
>
> \[Slide\]
>
> Then we calculated statistics, and if we looked at the total summary
> we got a Pearson correlation of the total score of 0.95, which those
> of you who are into statistics know is a very good score and suggests
> that the raters were in agreement, high agreement with each other even
> though they did this independently.
>
> But when you looked at the ten criteria you saw some differences in
> reliability that I think I would like to come back to later this
> afternoon. I would like you to remember this. If you look here, there
> is a Pearson correlation for 0.9. That means that the agreement is not
> quite as good because it is less than 1. That is for item criterion
> number eight, about legibility and comprehensibility. But the Pearson
> correlation is very close to 1, meaning very strong correlation, for
> criterion number two, and the others were all over 50. That is, the
> most disagreement among the panelists was on this criterion, whether
> it was legible and comprehensible. I would like to suggest later that
> we need to address that in the future work.
>
> \[Slide\]
>
> How did we process data? The packets were sent to UW Madison where we
> removed identifying information. We made copies and then we sent these
> copies and a packet out to the three panelist subgroups for their
> rating. Items were divided among the three panelists. That is, they
> were not multiple gradings by the panelist but each panelist handled
> about one-third of their groups. Okay? Items were independently
> reviewed by the panelists. Thus, each panelist rated one-third of
> their group or one-ninth of the total, if that is clear. Okay?
>
> Another way to think about that is in a given pharmacy the patient
> might have gotten three information sheet, one for ibuprofen, one for
> amoxicillin and one for paroxetine, and the way our system worked you
> would have three different raters rating this, which I think is
> actually an advantage.
>
> We are now at the point of starting to look at the results. The
> question is whether we should have a break here or whether I should
> \-- should we have a break here? I feel right to have that, and then
> we can start with the graphs. Thank you for your attention.
>
> DR. OSTROVE: It is about 2:50 now. Why don\'t we try and return by
> about 3:10?
>
> \[Brief recess\]
>
> DR. SVARSTAD: The intent here is to quickly go through some of the
> findings, and I should add that I asked the FDA staff at CDER to make
> some photographs of some of the actual information sheets so that you
> can see some of the examples, and I have added those to the file at
> the end. So, I would like for us to have some time for that, and I
> hope that we don\'t really need as much time as we originally said for
> the findings so we have more time for questions and discussion.
>
> \[Slide\]
>
> Percent of patient observers given any written information by drug \--
> I thought it would be pretty deadly to put those tables up here so my
> secretary has converted these into some graphics. But for those of you
> who want more details in terms of number of cases and exact
> percentages, you will find that in the report, tables 1 through 3. The
> results will be presented by ibuprofen, amoxicillin paroxetine.
>
> Wow \-- that was my first thought when I saw these data. This suggests
> or shows that over 80 percent of the patient observers in eight
> different states received at least some written information. Now, if
> you remember back to the summary that I gave you of review of past
> studies, I think the last report was for 1994 and at that point 59
> percent of the patients were getting written information. Our study
> would suggest that at least for ibuprofen over 80 percent are getting
> some written information. The same with amoxicillin. Same with
> paroxetine. In other words, there is consistency across drugs.
>
> Now, what do we make of that? I think that at the very least,
> regardless of what you say about content and about usefulness or
> format, we really should note the good news here. The good news is
> that three out of every four or more, by these data, are getting some
> written form of information, and I think this reflects remarkable
> progress and it is important to get that word out because I think that
> is not the case in other countries and it certainly wasn\'t the case
> in this country sometime ago. So, we need to always put things in
> historical context and to report the good news along with any concerns
> that we have.
>
> \[Slide\]
>
> Distribution of ratings using the one-point method, and this is for an
> N of 306. That is 306 pharmacies where we had complete data available.
> This is the first distribution of ratings that shows the percentage of
> sub-criteria that had full or partial adherence. You remember I said
> we had levels zero, 1 through 5, with 5 being the highest and 5
> meaning 80 percent of the criteria being met; zero meaning none.
>
> For ibuprofen, you see that the bulk of them are in the level 4 and 5,
> but that the majority did not receive information at level 5.
> Amoxicillin was somewhat better. That is, we see some variability here
> by drug in terms of the content. Now, for amoxicillin, that basically
> means that over 60 percent of the patient observers received
> information at this level, the highest level. The rest of them were
> variable. This percentage, here, reflects the 13 or so percent that
> got no information. The same with paroxetine.
>
> \[Slide\]
>
> Let\'s look at the individual criteria so that you can see some
> differences by criteria out of the one out of ten. What we have done
> is put some charts together that show ratings by criterion. The first
> one is does the information include information about the drug and its
> benefit? This is excluding the people that did not receive any
> information. Okay? I am pretty sure about that, although this slide is
> a little ambiguous. Your tables will show it more precisely.
>
> Seventy-eight percent of those information sheets had high adherence,
> that is, rated in the 7-9 category by the raters. That is pretty good,
> meaning that they are pretty good about giving the information about
> the drug and its benefits; 79 percent of the amoxicillin was rated
> highly, and for the paroxetine 77 percent. So, that is pretty similar
> across drug type.
>
> But you do see about 22 percent that are rated either moderate or low
> on this, meaning the raters felt there was not sufficient information.
> As you know, sometimes this could mean that the pharmacist is giving
> out a short form of the drug information, or it could mean that the
> information sheet, even the long form of the information sheet does
> not include enough information about the drug and its benefits.
>
> \[Slide\]
>
> Two: specific directions. Thirty-three percent of the ibuprofen sheets
> were rated as high adherence on specificity of directions \-- room for
> improvement there. Amoxicillin was better. So, you see some
> variability here. Paroxetine, 59 percent. So, you don\'t see the high
> levels that you saw on the first criterion.
>
> \[Slide\]
>
> Contraindications and what to do, criterion 3, pretty low on ibuprofen
> according to the raters, only 12 percent of the information sheets met
> high adherence on this particular criterion. A little bit better on
> amoxicillin, right here you see 31 percent high adherence; 49 percent
> and 20 percent moderate and low adherence; and about 21 percent for
> paroxetine.
>
> Now, if you have specific questions \-- I am sure you have discovered
> this about the report \-- you can go to tables 4-6. We are not going
> to do that right now and get sidetracked, but you can figure out by
> going those tables to see which of the sub-criteria were not met
> according to the raters to get some feeling for why this was so low,
> or another way to look at that is why these weren\'t higher. In other
> words, which of the sub-criteria were not being met. This is kind of a
> summary.
>
> \[Slide\]
>
> Precautions and how to avoid them \-- ibuprofen, only 5 percent \-- I
> think that is a mistake. No, let\'s see. I would have to check that.
> Is that a mistake?
>
> AUDIENCE PARTICIPANT: It is 5.3 percent in the report.
>
> DR. SVARSTAD: Yes, I would have to look at my table to see that, but
> that is pretty close to what the report is. But I know there was one
> table that we fixed because we found a typographical error, but the
> report as it is now I think is okay.
>
> You see consistently, fairly consistently here that ibuprofen received
> lower ratings, but paroxetine isn\'t getting very high ratings here
> either. So, in the area of precautions and contraindications the
> ratings were not perhaps what you would like to see.
>
> \[Slide\]
>
> Adverse drug reactions, considerably better. Over 80 percent met high
> adherence according to the raters. About 80 percent met it for
> amoxicillin and about 77 percent for paroxetine. So, you can begin to
> see that there are some real differences by the criteria that you are
> talking about here, with contraindications and precautions receiving
> lower ratings according to the raters.
>
> \[Slide\]
>
> Storage and general information \-- again, there were some lower
> ratings here and you can look at the more detailed tables to see which
> items were excluded to get an understanding for why these are low. I
> think when we discuss tomorrow, the question is, well, of all of these
> which ones are criteria that are really important, and should storage
> be given the same information as precautions and contraindications? In
> this report we have gone through each one systematically and not made
> a judgment as to which one should receive higher or lower weightings
> because the Keystone report did not make a distinction. So, we felt it
> was inappropriate for us at that stage, but I think it is very
> important for this discussion to take place tomorrow.
>
> \[Slide\]
>
> Unbiased in content and tone \-- no question that the raters saw these
> as unbiased in content and tone. If there was any criticism at all \--
> these cases, here, it was because the balance was on more risk
> information than benefit information, and maybe there should be a
> little bit more of the benefits so that consumers can make a judgment
> here between benefits and risks.
>
> I find it personally interesting, for example, that many of the
> information sheets might only have one or two lines about the benefits
> but many, many lines about the contraindications, and everything we
> know about patient adherence is that patients need to have an
> understanding of the benefits of something if they are to take it for
> life. So, I think this discussion should take place at some point. The
> Keystone report was not as clear on this as I might like, and I might
> suggest that we have some discussion about what kinds of benefit
> information do people really need. For example, in Wisconsin we were
> helping a local clinic redesign their information sheets and it
> happened to be the Dane County Mental Health Clinic for persons with
> severe mental illness, and we held focus groups with persons who have
> diagnosed schizophrenia. We were somewhat surprised to learn from them
> that they wanted \-- I don\'t know why we were surprised but they
> wanted more information about the benefits and they were quite serious
> about this, and I think they had a very good point. They are putting
> up and tolerating with a lot of side effects and it would be nice to
> know, when they start therapy, what the odds are of them being able to
> improve, and in what areas they are going to improve, for them to be
> making this decision. I think they have a valid point.
>
> \[Slide\]
>
> Legible and comprehensible \-- 85 percent rating across but you see
> some problem here, and I would remind you that this is one of the
> items where the raters were less in agreement and I am going to show
> you some data a little bit that may suggest that experts are not the
> best people to ask about legibility and comprehensibility. First off,
> they are all pretty young; they don\'t have macular degeneration and
> other issues that older people do, and the information is second
> nature to them so I have a feeling they are not so good about judging
> whether or not something is comprehensible. So, I think we really need
> to discuss that point.
>
> \[Slide\]
>
> Scientifically accurate \-- no question in the first two drugs. This
> does not suggest that it was scientifically not accurate. As I recall,
> it was less likely to be a disclaimer or other information missing for
> this particular one but you could look at the detailed reports on that
> but I think they came out very good on this criterion.
>
> \[Slide\]
>
> Up to date in publication \-- that means was the information up to
> date scientifically and did in include information about who the
> publisher or the vendor was. So, in the business of social science
> that is an item that is kind of difficult because you are mixing
> apples and oranges here, in my view, but at least I am honest about
> it. You can see on the more detailed report which of these were not
> met. In any case, you see that about 50 percent were both up to date
> and had the publication information; 55 percent and 45 percent \--
> pretty similar across the way. We could not always tell who the vendor
> was, and I am not sure whether this was because some information \--
> well, we really can\'t be sure because we couldn\'t go back to these
> pharmacies to ask. But on many of the sheers, of course, we could tell
> who the vendor was but on others we could not always tell who the
> vendor was.
>
> I think one of the interesting things about this project is that in
> the original contract we were not asked to analyze by vendor so we
> have not done that. You need some staff to do that. I think we could
> do that to facilitate the process here. It probably wouldn\'t take too
> much time, probably a graduate student could do it in a while, in the
> summer. But there is a lot of variability in how it is that the
> pharmacy prints out information from the same vendor.
>
> So, even though you have 50 information sheets from one vendor, you
> could have 30 different styles of it being printed out. Does that make
> sense? That is, some people might print it out in small print; others
> print it out in large print. Some people print it out in a printer
> that has sharp contrast; others print it out in a dot matrix printer
> that you can barely read. Some print it out on a sheet of paper that
> is very easy to see in terms of glare; others might print it out on a
> small sheet of paper with watermarks underneath it and you can see
> more of who owns the pharmacy than what you can reading the
> information. So, even though you might think that we have only a few
> sheets here and they are duplicated, actually we got an enormous
> variety because the pharmacy still has great latitude in how it uses
> that information or how it prints it out, not to mention maybe how it
> selects. But, we weren\'t able to find out whether they selected out.
> So, I would be a little careful about even doing a vendor analysis
> unless you had what the vendor originally gave them, which would
> require \-- it is kind of like evaluating term papers and whether or
> not there has been plagiarism, only just the opposite I guess. We have
> done that.
>
> \[Slide\]
>
> Summary of the main findings \-- now, before we get to some of these
> specimens, 87 percent of the patients received some form of written
> information. I think that is the good news of this report, suggesting
> considerable progress.
>
> Over 75 percent of the items received a high rating or high adherence
> for criterion number one, drug and its benefits; for criterion number
> five, adverse reactions; seven, unbiased in content and tone; eight,
> legibility, even though I have some concerns abut that one; and, nine,
> on accuracy and disclaimer. That is, in these areas the raters viewed
> over 75 percent of the items very positively.
>
> \[Slide\]
>
> But it seems clear that improvement is needed in other areas such as
> criterion two, on specificity of directions, which is good for some
> drugs but not all drugs; criterion three, on contraindications and
> what to do. It is not because contraindications are totally left out;
> it is because maybe they are not dealt with as fully as they should
> be. Okay? It is always important to remember that. It is not that
> contraindications are omitted; it is that they are not as complete as
> the raters felt they should be. Criterion four, on precautions; six,
> storage and general information; and ten, on publication information.
> Information quality varied somewhat by drug, as I have pointed out
> several times. So, I think in general the results carry some very
> positive results, as well as some concerns.
>
> \[Slide\]
>
> Now, what are some other findings that I think are important to
> highlight for this first attempt? I think that given the fact that
> this has never been done before in this way, we at least were able to
> develop a method to get excellent inter-rater reliability using
> standard forms. This is an accomplishment that will bode well for the
> next time around if further evaluations are done.
>
> Another finding I think is that we had lower inter-rater reliability
> for criterion eight, which is on legibility and comprehensibility. I
> think this suggests further discussion and work, and I have some ideas
> about that even though I haven\'t put them in the slides.
>
> \[Slide\]
>
> I think there are study limitations that we certainly, and many of you
> certainly, are aware of, and let\'s try to highlight a few of those so
> that we keep those in mind as we meet for the next day or so.
>
> At the top of the list I think is that we gave equal weight to these
> criteria and sub-criteria even though you may identify a variety of
> reasons why you would want to weight one more than another, which is a
> policy issue and not a scientific issue. From whose perspective? From
> FDA\'s perspective? From drug manufacturer\'s perspective? From health
> professional perspective or from consumer perspective? I mean, there
> are so many perspectives here and ways that you could weight these
> that I don\'t think it is a simple question.
>
> We have states that volunteered for the study. My one concern is that
> maybe those states have a different level of practice than the other
> states. We don\'t have any basis for making that conclusion because we
> don\'t know really what the situation is in the other states. I am
> less concerned about that one though because I feel that much of what
> we saw was in chain pharmacies which span state boundaries. So, I am
> not really terribly concerned about differing quality on written,
> although I should tell you that, in fact, what I have done is to
> analyze whether or not there were differences in the prevalence of
> different information by state to see if there sere any differences.
> There were not.
>
> But I can tell you that there are wide differences in oral counseling.
> The purpose of this workshop is focusing on written so I am not able
> to take the time to go through some of the findings that we got on
> those one-page forms with the inspectors, except to tell you that in
> the future I would suggest, and believe it is quite important, that we
> always remember that written goes with oral, and it is hard to
> disentangle those. I understand the congressional mandate but I do
> feel that it is important for us to always keep returning to the fact
> that written information should not be substituting oral information,
> and we always need to keep that in mind.
>
> I did go through the forms, the observer forms, to see how it was, for
> example, that the pharmacist disseminated that written information
> sheets. What we found was that only 35 percent of the written
> information sheets were given to the client or the patient, patient
> observer, with some kind of mention or with some kind of oral review,
> or with some kind of encouragement to read it. In other words, in the
> majority of the cases, according to the state inspectors, these
> written information sheets are being stuffed in the bag. They are not
> being discussed, reviewed, or mentioned in a positive way by the
> pharmacist. And, I would warn us all to remember that what evidence we
> do have on the effects of written information would suggest that their
> efficacy depends on oral review.
>
> The studies, for example, on the effects of written information on
> patient compliance would suggest that you really need the review by
> the pharmacist or professional if those things are going to make a
> difference. Why? Because if you are not encouraged to read it, many
> people will not read it. But if you are encouraged to read it, people
> will read it. And, if the pharmacist or health professional points out
> sections of it that are important I think the patient is much more
> likely to read it. Now, in this case, this is not an issue because we
> are talking about state inspectors. But I do think that we always need
> to keep remembering that oral goes with written. So, I will stop there
> and not beat that one too much into the ground.
>
> Thirdly, sampling procedures varied somewhat from state to state. I
> guess we always need to remember that we can have differences in rural
> and urban areas. I think we focus more on urban areas, although we had
> some rural states so we maybe got some rural here. But it would be
> desirable to have uniform sampling so that we can make sure that we
> have the kind of sampling that can enable you to draw conclusions more
> broadly.
>
> We had limited training of the patient observers, and we did not have
> a very good way of controlling or reviewing their work. It is not that
> I am questioning it but I think in any study you need to bear in mind
> that if you have this many people in eight different states, it is
> possible that some people interpreted their role or their job a little
> bit differently than others, and I cannot really comment on that,
> except to say that we were not able to give them the kinds of
> intensive training that you would in a study where you had maximum
> control. I am not terribly concerned about that, but I put it out
> there because I think we always need to remember it and bear it in
> mind when interpreting results.
>
> My most serious concern I saved for last, and that is with a contract
> focused on an expert panel which did not include a consumer panel or a
> consumer input. And, I feel that based on the results of the
> inter-reliability, the experts had some difficulty deciding whether
> things were legible or comprehensible. That is kind of a flag number
> one.
>
> I do think we should consider including some mechanism for consumer
> input the next time around. I think this could be done in a number of
> different ways that I would encourage you to discuss or think about,
> including ways that you might suggest. One might be to add consumers
> to the panel. That is okay. It is not quite as interesting but it is
> possible that you could do that. A second way might be to have two
> types of panels, an expert or professional panel and a consumer panel
> with them having distinctive roles or distinctive responsibilities,
> with the professionals being more responsible for evaluating the
> accuracy and the completeness of the information and the consumers
> being more responsible for evaluating the potential usefulness, the
> readability, the comprehensibility, the ease of following the thing,
> and so forth. So, you could think of ways that these two panels could
> complement each other. It is possible that you could take that
> criterion, number eight, and take it out of the expert forum and make
> that a consumer evaluation and put the two together.
>
> My colleagues and I and Dara Bultman actually spent a little time
> working on this in November and December, where we did develop a
> consumer rating form, and we developed a form for evaluating in more
> detail the design and format of the information sheets, and we came up
> with some very interesting results. That convince me that the consumer
> has a unique perspective here and that we should be eliciting it.
>
> \[Slide\]
>
> Now, some samples \-- we selected a sub-sample of information sheets
> from the sheets that were collected in November, and I put together a
> consumer panel of 24 consumers, 12 individuals with a college
> education and 12 with a high school education. This is not described
> in your report anywhere so there is no need to look for it. It is a
> small pilot study, a small method study that we did with other support
> and it was not part of this contract, but because I know we are
> discussing all of this I thought you might be interested in seeing
> these samples and maybe hearing what some of the consumers said about
> them, even though this is not a large-scale study.
>
> Here is one of the specimens. I have removed the identifying
> information up there. This is for ibuprofen. It came from pharmacy
> 644. You can ask yourself whether that meets the criteria that you
> think it should be meeting, but when we look at the consumer \-- you
> can\'t read it? Well, neither can I.
>
> \[Laughter\]
>
> You know, there is no way I could put this up so you could see it
> without changing the very nature of it. What I want you to pay
> attention to is that this is an exact photograph of it and I wanted to
> show a couple of things. I don\'t want you to get into your pharmacy
> role or your technical role of evaluating the content. I would like
> you to kind of see the style of these things in general. That is, how
> close are the lines; how are they organized, etc.
>
> Now, we may have tried to figure out some way that we could get a
> sample of these available on the net \-- I don\'t know, we haven\'t
> really talked about this but, trust me, we want to facilitate
> improvement here so, aside from releasing these which I think we would
> find difficult to do because they have so much identifying information
> on them, I want to just give you some idea here about the range of
> what we got and how consumers evaluated it.
>
> In this particular one we asked the consumer to rate, for example,
> organization; attractiveness; print size; tone, whether it was
> alarming or encouraging; how helpful it was to them; and spacing
> between the lines. In this particular one the consumer gave it a 1 on
> spacing, poor. If you really could read this, if you saw the original,
> you would probably agree with that consumer that it was pretty hard to
> read this one. The patient said the print is too small; the lines are
> too close together.
>
> The patient gave it a 4 on organization. You see the headings there.
> So, maybe that consumer thought it was pretty well organized.
>
> How attractive is it? They gave it a 2, unattractive. Print size, they
> gave a 1. They gave a 3 on helpful \-- 5 would have been helpful; 1 is
> unhelpful. And, 1 on poor spacing.
>
> Interestingly enough, when you looked at the consumer evaluations on
> this, often when it was poor lines, poor print, poor spacing, etc.,
> they often were critical about the whole thing, probably because it
> was just difficult for them to read.
>
> \[Slide\]
>
> Let\'s go to another one. Can you read that one? This is pharmacy 601.
> You see here that the name of the drug is kind of underneath here. It
> is ibuprofen. I have put yellow over here to cover the pharmacy name.
> This is the entire sheet. That is all the consumer got.
>
> I thought this was kind of interesting. \"This specific information
> may or may not apply \--\"
>
> \[Laughter\]
>
> \"\-- to your condition. Please consult your doctor.\" I teach
> pharmacy students and this one would not pass!
>
> Why am I taking this drug? For arthritic conditions, pain,
> inflammation, fever. That is drug benefits. Well how should I take it?
> Take with food, antacid as directed. Tell M.D. of other drugs you
> use/diseases you have, allergies or if pregnant. Limit alcohol intake.
>
> Are there any side effects? Dizziness, drowsiness, report. Eye/ear
> problems; urine color change; black stools; difficulty breathing;
> mental changes; sun sensitivity; stomach pain.
>
> How do I store this? Store at room temperature, away from moisture and
> sunlight. Do not store in the bathroom.
>
> If I should miss a dose? Take missed dose as soon as remembered but
> not if it is almost time for the next does. Do not double up.
>
> See, here is an example where we couldn\'t tell who really created
> this information. There is no publication information, no vendor, etc.
>
> This is what one of the members of our consumer panel said about this,
> \"very easy to understand, very easy to remember, very easy to locate
> what information you wanted.\" This is, of course, the tradeoff of
> giving information. You know, if something is easy it often is a
> little bit limited in terms of amount but at least they gave them
> pretty good ratings on that. How likely are you to read this? Somewhat
> likely.
>
> Then, when we get over to the other side of the form they said, about
> right on the medication, the amount of medication. Below is a list of
> topics, please indicate your opinion about how much information was
> provided on each topic and how useful you think this information would
> be if you were taking this medicine for the first time. Medication and
> benefits, they thought that is about right. Who should not use this
> medication? Too little information. Specific directions about how to
> take the medication? They said too little. Precautions that need to be
> taken while using the medication, they said too little. Possible side
> effects and what to do, about right. How to store it, about right. I
> think that is kind of interesting, you know, that they are making
> distinctions here.
>
> When they were asked to rate the organization, they gave it a 2 out of
> 5, with 1 being the lowest. Attractiveness, they gave it a 1, which is
> the lowest rate of attractiveness. They gave a 1 on print size; a 3 on
> helpfulness; a 1 on spacing even though the spacing looks pretty in
> comparison to others.
>
> \[Slide\]
>
> Now let\'s go to another sample. I could try to read through all that
> but I am not sure that would be very useful to you, but this one is a
> little bit more detailed. You can see here that they have some use of
> headings but they are all caps, which is not recommended because
> people find it hard to read when it is all capitalized. You see
> headings \-- uses; how to take; side effects; precautions; drug
> interactions; what to do with missed dose; and storage.
>
> When we asked the consumers what they thought of that one, they said,
> for example, when we go over to amount of information they thought
> there was about the right amount of information on directions,
> precautions and side effects. Then they had some differences of
> opinion about the medication and its benefits.
>
> When they were asked how well organized it is, they gave it a 4. When
> they were asked how attractive it is, they gave it a 3. When they were
> asked about print size, they gave it a 2. When they were asked whether
> it was helpful or not, they gave it a middle of the range, a 3. When
> they were asked about spacing between lines, they gave it a 2. In
> other words, they thought it was not spaced very well, too close
> together.
>
> \[Slide\]
>
> Here is another one. Now, you see actually this is put out by a vendor
> \-- maybe you can\'t read that; let\'s not worry about that. You see
> here common uses; how to use the medication; cautions; possible side
> effects. You see here again is this tendency to use all caps, and to
> use them quite liberally. You see here all these caps; all caps; all
> caps; all caps.
>
> Let\'s see what the consumer said about that one. I hope I am not
> boring you with detail here. This one said it was pretty easy to read.
> It was pretty easy to understand. It was pretty easy to remember. Very
> likely that they would read it; very likely that they would use it.
> About the right amount of information on each one of the criteria, the
> medication and its benefits; who should not use it; specific
> directions; precautions; side effects, etc. They have it 2\'s on
> everything there, about the middle.
>
> When asked about organization, they gave it a 4. When asked about
> attractiveness, they gave it a 5. I think there is a tendency for them
> to give higher stuff with some color and maybe the bolding facilitated
> that score. Print size, a 5. Helpfulness, a 5; and spacing, they gave
> it only a 3. They probably were pretty liberal on that one because
> there is not much spacing there but these are consumer evaluations.
>
> \[Slide\]
>
> Let\'s try another one, 721. Now, actually, it looks pretty similar to
> the other one, doesn\'t it? You again see the same organization. Now
> let\'s see what the consumer said about this one.
>
> About the right amount of information, a 2, a 2, a 2, a 2, a 2 on each
> one of the topics, about the right amount. They did not rate the
> organization, so missing data. They put a 2 as how attractive it is.
> This person didn\'t consider it very attractive. Print size, they gave
> only a 2. Spacing between the lines, they gave it a 1. In other words,
> they were kind of critical there of that one for spacing.
>
> \[Slide\]
>
> Oh, boy! Actually, this is an example of something I think that comes
> from a dot matrix printer. This is an entire sheet and then this
> patient got, in addition to this sheet, this little additional sheet
> with it. Now, you can\'t read this maybe from back there but this says
> Paxil 10 mg tablet. There is a 1 there with no explanation of what
> that is. May cause drowsiness. Then there is an \"A\" and \"LCOHOL.\"
> There is a poor wrap here I think. This is the way the patient got it.
> \"A LCOHOL intensifies effect; use care using machines.\" Then there
> is an 8 which says \"do not drink alcoholic B EVERAGES \--\" the wrap
> is not very good \-- \"when taking, space, space, this medicine.\"
> Then there is a number 13. Take or use this exactly \"A as\" directed.
> Do not skip doses or discontinue. Call doctor before taking OTC drugs.
> Some may affect \"A CTION\" of this med.
>
> Now let\'s see what the patient said. Very likely that they would read
> it; very likely that they would try to use it. They thought the sheet
> in general was well organized. They gave it a 3 on attractiveness; a 3
> on print; and a 2 on spacing. They added these comments at the end:
> The print type was somewhat hard to read, however, I liked the
> organization layout. The small sheet attached is not helpful, and
> unorganized. So, that is what they said on that one.
>
> \[Slide\]
>
> Now, what we decided to do, and this is about where I am going to end,
> in the process we decided, and unfortunately this doesn\'t come out
> very clearly either because it is an exact photo, but what we did is
> to take the Keystone criteria out of the appendix on design and
> format. I think it is Appendix G, as I recall. We took that and as
> carefully as we could used those criteria to redesign three sample
> sheets, three actual sheets that we collected.
>
> The idea here was to take the design format guidelines and redesign it
> so that it met the Keystone criteria, and to keep the content exactly
> as it was in the original. Is that clear? So, for example, the
> Keystone criteria may say something about type, and it may say
> something about the use of headings without all capital letters. It
> may say something about sharp contrast. It is easier to read something
> when it has sharp contrast. It says something about font size. It says
> something about using bullets. It makes it easier to read for people.
> And, it says a number of other things.
>
> Basically, what we did then was to create this sheet. I have numbered
> it here. This sheet is kind of similar to the prototypes that are in
> the appendix of the Keystone report. We took these sheets and we
> blended them in with the actual sheets and, actually, we randomly
> ordered them so that the patients on our patient panel would sometimes
> get this one first and sometimes they would get it last; sometimes
> they would get it second; sometimes they would get it third. Our
> intent here was to ask them to rate real sheets and a design sheet
> that we tucked in there, what we might call a model sheet. This was
> one of the model sheets, and we did not tell them that we put a model
> there. We said these were all sheets that we had collected. That is a
> little bit of a mislead.
>
> We had 24 consumers, as you remember, 12 that have a college education
> and 12 that don\'t. So, we had 24 total. We asked them to compare on
> organization this sheet with the sheets that I have been showing you,
> the real sheets. The real sheets got a mean of 3.5 and the model got a
> 4.8. We asked them to rate on attractiveness. The real sheets got a
> 2.8, the model got a 4.3. Print size, the real sheets got a 2.6, the
> model got a 4.9. Spacing, the real sheets got a 2.5, the model got a
> 4.7. Whether or not they were helpful, the real sheets got a 3.7, the
> model sheet got a 4.7. All of these were statistically significant, as
> you might guess but it made it interesting nevertheless.
>
> When we asked them how easy it was to read, they gave the real sheets
> a 3.6 and the model a 4.9. When we asked them how easy is it to
> understand, the gave the real sheets a 4 and the model a 4.9. How easy
> to remember \-- this is kind of interesting \-- they gave the real
> sheets a 3.5 and the model a 4.5.
>
> We did some other things here but I think that is enough to perhaps
> stimulate your thinking. These results were also statistically
> significant. Now, what this tells me I think is that if we did put
> together a consumer panel, they might have a very different view and
> they certainly would have a somewhat critical view of print size and
> how this information is laid out on the sheet, which may or may not
> have anything to do with the content of the information as it is put
> together by the vendors.
>
> So, we have a situation here where we need to be concerned, of course,
> about content but we also need to be concerned about how these sheets
> are being printed out and how they are being presented to consumers
> and what consumers think of this because it could get away from us, so
> to speak. In the interest of having something scientifically accurate,
> we have something that really is kind of a mess potentially, not
> because anyone is intending it to be that way but because we don\'t
> have clear standards, or we don\'t have a clear way of evaluating it.
> But I think there are some ways to evaluate it and it is something
> that we can discuss in the next day or so, something that we weren\'t
> able to do as carefully but I think we have a few ideas.
>
> I think I will end my presentation there which, fortunately, is sooner
> than five o\'clock, and I very much appreciate your attention. I know
> that many of you have read the report and I appreciate your indulgence
> with me going through the details. I now will turn it over to Nancy
> who will, I guess, monitor the questions or however you plan to do it.
>
> **Questions from the Audience**
>
> DR. OSTROVE: We are finishing a little earlier than we expected, which
> I think will probably be fine for everyone. No one really objects to
> finishing earlier. This also gives us a little bit of time for
> questions, more time for questions before I give you all my little
> notes about tomorrow.
>
> There are two mikes, one in the front and one toward the back, and we
> would appreciate, if you have questions come up to one of the mikes,
> and if you could give your name and affiliation so we have a sense for
> who is involved, and then start. Okay?
>
> MS. LEUNG-VEGA: Jane Leung-Vega, from Merck-Medco. I actually have two
> questions. One of the questions is what was determined as up to date
> information, that is, once the FDA or whatever criteria you used to
> set the information is scientific information, how soon would they
> need to have that published or available to the patient?
>
> DR. SVARSTAD: As I recall that, we gave them the most recent approved
> labeling for the medication, and the panelists had agreed that they
> would use that as the way to determine whether or not the information
> was up to date. So that was the most recent at the time that we
> developed and approved the forms, which would have been in spring of
> \'99. Does that answer your question?
>
> MS. LEUNG-VEGA: Well, what I wanted to know is, like, if the FDA
> approved a new individual how soon should that information be
> available to the patient? Three months? Six months?
>
> DR. SVARSTAD: I think that is a good question that we, in a sense, did
> not have to face, or maybe we chose not to face it because in the
> interest of doing the evaluation and having a standard form I think we
> were making the assumption that the approved labeling for the date at
> which that form was developed would be the one that we would use. I
> think our intent, original intent had been that these data would be
> collected even more quickly than we were able to collect, but I think
> your question is a good one for future. And, as I recall, there is no
> guidance on that.
>
> MS. LEUNG-VEGA: Okay. My second question is, of those vendors that you
> could identify in the written information, was there any that actually
> met a lot of the criteria?
>
> DR. SVARSTAD: That is another good question. The contract did not
> include staffing for analyzing vendor differences. This is not to say
> that we can\'t analyze that but we have not analyzed vendor
> differences.
>
> MS. LEUNG-VEGA: Okay, thank you.
>
> DR. SVARSTAD: You know, I am sure it would be useful to the vendors.
>
> \[Laughter\]
>
> Several of us were talking during the break and if anyone has ideas
> about, you know, what we could do to do additional analysis to assist
> the vendors in this, I would be more than willing, but also remember
> that we need some staff to do that. So, if you have some ideas \--
> resources are always an issue but I think it is an important question.
> I did not realize really that there could be as much selection here,
> so that probably makes me even more cautious about reporting vendor
> differences because you are not sure \-- are you reporting vendor
> differences or are you reporting what the pharmacist chooses to
> publish. The two are very different questions.
>
> Yes, sir?
>
> MR. SASICH: My name is Larry Sasich. I am from Public Citizens Health
> Research Group, in Washington, DC. I am somewhat maybe confused or
> concerned about the expert panel\'s definition of scientific accuracy.
> It seems like scientific accuracy may have been applied primarily to
> FDA-approved indications that you found in the patient information
> leaflets while, in fact, the intent was that scientific accuracy
> covers all of the labeling and that essentially means it is derived
> from, and consistent with, the FDA-approved labeling.
>
> In your patient information evaluation form for paroxetine, for
> example, your criteria five \-- in the criteria that you used, one of
> the criteria \--
>
> DR. SVARSTAD: Criteria that the panel approved.
>
> MR. SASICH: Yes, other side effects may occur. Check with your
> provider. If you go back and look at the Keystone criteria,
> particularly component (e) which deals with paroxetine, the PIEF you
> used failed to identify 14 precautions for the drug. Under component
> (f) for adverse drug reactions, the PIEF missed 25 adverse drug
> reactions. And, I know that we didn\'t have a clear definition of
> frequently occurring adverse drug reactions when the Keystone report
> was written but there is a regulatory definition of frequent adverse
> drug reactions, and those are the ones that occur with at least a
> frequency of one percent. And, do you really think that it is adequate
> communication of risk information to patients to say other side
> effects may occur and check with your doctor? That is the issue that
> we are trying to solve, and that we have been trying to solve for so
> long.
>
> DR. SVARSTAD: Do you want to try and answer it?
>
> DR. OSTROVE: I think one of the things that we tried to do with the
> interim report was to get an expert panel and have the experts to do
> this because these are experts in pharmacy and in communication
> because there are issues, and I think there are concerns always, with
> regard to how much information is useful; how much information ends up
> being too much information so that, in fact, what the patient ends up
> getting is so much that they are totally overwhelmed by all of it and
> it dilutes the important information.
>
> So the attempt in this particular instance was to have the experts
> make these determinations as opposed to, for instance, to having FDA
> making the determinations. That is one of the reasons we are here
> today, actually, to hear from the public as to, you know, where they
> feel the limitations are. So, I think it is very important that you
> give us these perspectives. Larry, we really do appreciate it but I am
> not sure that it is something that Dr. Svarstad should be answering
> because it is really in some sense a policy question.
>
> MR. SASICH: Nancy, just one comment, one of the consumer groups\'
> major points in the Keystone meeting was having to deal with the
> paternalism of health professionals that wanted to decide for us what
> information they think we should know. The Keystone criteria defined
> useful information and one of those criteria was informing patients
> about frequently occurring adverse effects.
>
> DR. OSTROVE: That is true, but there is not actually complete
> agreement about what frequently adverse events is.
>
> DR. SASICH: I know, and I said that.
>
> DR. OSTROVE: For instance, one out of a hundred, for certain products
> that is appropriate. For certain products it might be five out of a
> hundred. For certain products it may be even more. So, that is a
> difficult concern. It is one that is important to raise \--
>
> MR. SASICH: Sure.
>
> DR. OSTROVE: \-- and it is difficult to know where to draw the line on
> that.
>
> MR. SASICH: Well, we do have a regulatory line.
>
> DR. OSTROVE: In terms of what frequently occurring means for the
> package insert \--
>
> MR. SASICH: Right.
>
> \-- but that is not necessarily what frequently occurring means for a
> patient package insert.
>
> MR. SASICH: But to be useful it has to be consistent with or derived
> from.
>
> DR. OSTROVE: But to be useful does not necessarily mean that it has to
> have every single piece of information \--
>
> MR. SASICH: No, only the ones that are frequently occurring \--
>
> \[Laughter\]
>
> \-- so please let me know how you would make the decision about what
> risk information should be withheld from consumers because a group of
> pharmacists think they don\'t need to have it. That is not the point
> of this whole exercise.
>
> DR. OSTROVE: I think you are right, this is an important concern but I
> am not sure it is one that we can determine right now, and I am hoping
> that you will discuss it in your group tomorrow.
>
> MR. SASICH: Probably.
>
> \[Laughter\]
>
> DR. OSTROVE: All right!
>
> MS. POWELL: Marjorie Powell, from PhRMA. I have two questions that
> were unclear to me from your presentation. The first is that in the
> summary results that you put up there you included as one of the
> sub-criteria for criterion nine a disclaimer. But when I look at your
> tables, I find the disclaimer included within criterion six. It
> wasn\'t clear to me where the disclaimer got counted.
>
> DR. SVARSTAD: There might be a couple of disclaimers. I would have to
> look at the forms.
>
> MS. POWELL: Okay. Maybe you and I can talk about that later.
>
> DR. SVARSTAD: Yes.
>
> MS. POWELL: The other thing that was \--
>
> DR. SVARSTAD: As I recall the guidelines, there was a number of
> disclaimers. I mean, if you get back to the side effects question, I
> would certainly \--
>
> MS. POWELL: Because, as I remember, the disclaimer explicitly was this
> document does not contain all information; talk to your physician.
>
> DR. SVARSTAD: It could have been a typographical error. Which slide
> are you referring to? Do you recall?
>
> MS. POWELL: It was the one that described \--
>
> DR. SVARSTAD: Summary of findings?
>
> DR. POWELL: \-- summary of findings, criterion nine.
>
> DR. SVARSTAD: Let me just find it here. You have to go to the original
> form to see this. If you look, for example, at table 4, which is for
> ibuprofen \--
>
> MS. POWELL: It may not be worth taking everybody\'s time to work
> through it.
>
> DR. SVARSTAD: No, that is okay; that is why we are here. No, item nine
> is written information that is scientifically accurate. One of the
> sub-criteria under ibuprofen, for example, is that the indications for
> use are consistent with FDA labeling, and that is listed out. Then it
> notes that the medication may be used for other purposes.
>
> MS. POWELL: Okay, so you are using \--
>
> DR. SVARSTAD: And, that the general guide does not include information
> about non-approved uses.
>
> MS. POWELL: That answers my second question. You are using the term
> disclaimer to be a general reference to a statement that there may be
> unapproved uses \--
>
> DR. SVARSTAD: Yes, for the purposes of reporting here. The word
> disclaimer does not appear on the rater\'s evaluation form. If you
> look at item nine in either table 4 or table 5, you see, for example,
> under amoxicillin, item number nine is that indication for use is
> consistent with FDA labeling. The second sub-criterion is encourages
> communication with the provider. Then, the third point is that the
> general guide does not include information about non-approved uses.
> So, maybe I am not using it as you would.
>
> MS. POWELL: I think that is what the issue is.
>
> DR. SVARSTAD: Okay.
>
> MS. POWELL: But then that leads to my second question, if the patient
> information included information about an unapproved or an off-label
> use \--
>
> DR. SVARSTAD: Which if does not \-- oh, if it did, yes?
>
> MS. POWELL: If it did, how did you deal with that?
>
> MR. SASICH: For the patient information evaluation form for ibuprofen
> \--
>
> DR. SVARSTAD: Excuse me, let me clarify something here. I realize the
> controversy about the approved and non-approved uses. I do not want to
> get in the middle of it.
>
> MR. SASICH: Okay, I am just saying that you evaluated an off-label
> use, and that was Marjorie\'s question.
>
> DR. SVARSTAD: Excuse me, may I answer, please?
>
> MR. SASICH: Yes.
>
> DR. SVARSTAD: The panel discussed this. We decided to go with the
> Keystone report \--
>
> MR. SASICH: Yes?
>
> DR. SVARSTAD: The general guide does not include information about
> non-approved uses. If it listed a non-approved use I, frankly, cannot
> remember how we handled that. Duane, do you remember how we handled
> it? I do not remember seeing an instance of that. This is Dr. Duane
> Kirking. Do you want to comment on this?
>
> DR. KIRKING: Yes, I was trying to think back to exactly what we did as
> I was coming up. Quite honestly, I can only speak to the drug I did; I
> did not do them all. We didn\'t see any. I don\'t think we saw
> anything \--
>
> DR. SVARSTAD: I don\'t think we saw anything.
>
> DR. KIRKING: \-- in any of the labeling. I didn\'t see any in the one
> I did ad I don\'t think any of the others did. That doesn\'t mean it
> isn\'t an issue, but I don\'t think we saw any.
>
> MR. SASICH: In ibuprofen?
>
> DR. KIRKING: I know we had some controversy about it and I was one of
> those, in fact, that wanted to see more discussion of non-approved
> uses. This was when we set the whole process up before we actually
> looked at the materials. I think the decision was made \-- I think we
> were told, as I recall, that we weren\'t going to be considering
> non-approved uses at all in our evaluations.
>
> MS. POWELL: Thank you. I didn\'t want to get into the debate. I simply
> wanted to understand what it was that happened.
>
> DR. SVARSTAD: I appreciate that and, as I recall, one of the
> difficulties for us in implementing this, because the way I saw our
> role it was not to take a position on this controversy; it was to try
> to implement a consensus panel. So, I think when there are issues of
> controversy or lack of clarity, these issues need to be determined
> ahead of time so that we could have some guidance if it does occur.
> Unfortunately, I don\'t remember it ever occurring. So, I guess we
> were fortunate that it didn\'t come up this time, which doesn\'t
> answer your concern.
>
> MS. POWELL: Yes, that may have been a function of the product
> selection.
>
> DR. SVARSTAD: Right, exactly.
>
> MS. POWELL: Thank you.
>
> DR. SVARSTAD: You are welcome.
>
> MS. DAY: I am Ruth Day, from Duke University, and I have two much
> easier questions. The first one is you did alert all the pharmacy
> managers that somebody would be coming in a particular period of time
> \--
>
> DR. SVARSTAD: In Wisconsin.
>
> MS. DAY: \-- in Wisconsin. In the other states, did they not know
> anything?
>
> DR. SVARSTAD: No.
>
> MS. DAY: That is great; that is great. My second question has to do
> with the leaflets or the written information examples that you did
> get. You showed us a nice array of examples, but I am trying to get an
> idea out of the hundreds of pieces of paper that came in, how many of
> them appear to have come from comparable sources. I know that
> sometimes you can identify particular vendors and sometimes not, but
> if a content analysis were done you would find there are a certain
> number of types.
>
> DR. SVARSTAD: Of course.
>
> MS. DAY: So, I am interested in the type-token ratio about how many
> different types \-- were there ten types, and then some had hundreds
> and hundreds and some only had one or two? Can you comment on that,
> please?
>
> DR. SVARSTAD: Unfortunately, I cannot, Ruth, because we did not
> analyze that. If you look at the tables, we had over 700 pieces of
> information come in, and there is no doubt in my mind that we could do
> that in a month or less, but you do need to have some criteria for
> doing that and you have to have some people simply to sit down and
> look at them. When these came in we were getting in \-- you know, in a
> couple of months we had three states coming in. So, we were processing
> them and sending them off. So, this was never really part of the
> study. But I fully appreciate that it can be done, and would like to
> do it.
>
> MS. DAY: And then look at the rating results as a function \--
>
> DR. SVARSTAD: Yes, but the only concern I have is that you are not
> always sure if the vendor was responsible for what you got.
>
> MS. DAY: Well, aside from trying to decide who the vendors were \--
>
> DR. SVARSTAD: Yes. Let\'s be honest here. I think, you know, the
> number of vendors here I can put on both of my hands. You know, I
> think there are very few vendors that were responsible. You know,
> there was a number where it wasn\'t clear where it came from. But I
> think we all know who the major vendors were, but I could not comment
> for you \-- I would not want to give you any impressions that vendor A
> is getting this kind of rating versus vendor B and C.
>
> MS. DAY: No, I wasn\'t interested in that explicitly \--
>
> DR. SVARSTAD: Yes.
>
> MS. DAY: \-- but what percentage of all of the are piling up on a few,
> and if you could just look at those, and all the others are adding
> noise to your data \-- just a preliminary look I thought.
>
> DR. SVARSTAD: No. I hope you appreciate what is involved in just
> having them rated on the content, but it is something we can do. We
> certainly can.
>
> The question, I suppose, for this week really that I would turn back
> is, is this an important question and, if so, what kind of information
> would be useful on the next evaluation because that is, of course,
> where we are headed. I would assume that is where. So, if vendor
> differences are important, how would you do that and how would you
> justify that, or what would you get out of it? I would think some
> discussion of that would be useful.
>
> MS. DAY: Well, just on the surface, you could have those who have all
> of the side effects categorized by severity and frequency of
> occurrence, or just a sprinkling of some, and then just categorizing
> on that one variable that is interesting and look at the ratings.
>
> DR. SVARSTAD: There is no question that you could simply use the
> scales that we did have to do this, but I think what I am asking is
> what do you want to know about the vendors, and how would we do this
> in a way that is fair to the vendors, and how would you release that
> information? Coming from a university background, I am very sensitive
> to releasing information that reveals the identity of anyone,
> including a patient, a pharmacist, a pharmacy, a chain or a vendor. I
> am, frankly, a little stumped on how we would this but if you have
> some ideas I think that would be \--
>
> MS. DAY: I will just conclude by saying setting that vendor issue
> aside and all the political aspects of that, just looking at features,
> taking a feature approach and then looking \--
>
> DR. SVARSTAD: Sure.
>
> MS. DAY: Thank you.
>
> DR. SVARSTAD: Yes, you are welcome.
>
> DR. OSTROVE: I would just like to give you a little bit of background,
> for those especially who are asking about the kind of analyses that
> were done. Basically, it really does come down to a matter of
> logistics in terms of this particular study. We wanted to make sure
> that we could get the study done and get the results out to the public
> for your comment in enough time so that the vendors, whoever they may
> be, would have time to make whatever changes they saw fit on the basis
> of the study, even if the specifics of the study did not address their
> specific pieces and, as Bonnie points out, in many cases there are
> options that the pharmacist has to print out pieces of it and not the
> whole thing.
>
> We wanted to make sure basically that this information was available
> in a timely fashion, and we were limited in terms of the funding. We
> were graciously provided funding by the ASPE, by the Assistant
> Secretary for Planning and Evaluation, and it was only a certain
> amount and we wanted to make this sure that we had this done. So, it
> is not that Dr. Svarstad didn\'t want to do it. Believe me, she
> definitely wants to do these analyses but we could only fund certain
> kinds of analyses and we had to make sure that the study was done in a
> timely fashion. So, just keep that in the back of your mind.
>
> As a result, I think some of the questions kind of derive from that,
> some of the questions that we are asking you. Given now that we have
> all this out, let\'s hear all of these concerns. Certainly, I think
> Bonnie would love to collaborate people with regard to some of these
> additional analyses.
>
> MS. ALLINA: I am Amy Allina, from the National Women\'s Health
> Network. I have another question about scientific accuracy. Looking
> internally at the panel\'s own evaluations because it felt to me,
> reading it, like there was some internal inconsistency \--
>
> DR. SVARSTAD: I don\'t think on accuracy; there wasn\'t really.
>
> MS. ALLINA: Well, let me explain why I say that.
>
> DR. SVARSTAD: Sure.
>
> MS. ALLINA: To have the panel say that, for example with ibuprofen,
> there was 83 percent scientific accuracy and, at the same time, having
> them say 59 percent had low adherence on contraindications, or 40
> percent low adherence on precautions, to me \--
>
> DR. SVARSTAD: It seems inconsistent \--
>
> MS. ALLINA: Exactly, right.
>
> DR. SVARSTAD: Yes, I think in answer to your question, there really
> are two questions here. One is to what extent is a contraindication
> presented or included at all, regardless. Then, I think the other side
> of it is whether or not that information is up to date. So, I suppose
> you could say that the information sheet includes a contraindication
> but it is something that is not consistent with contemporary
> knowledge, whether you are talking about clinical knowledge or FDA
> knowledge.
>
> When we discussed this, my personal view on this is that this is kind
> of redundant. I mean, it seems to me that if you put down what are the
> approved indications, you put down what are the approved or accepted
> side effects, however you define that or however far you want to go
> with that, and then you also say scientifically accurate, I personally
> \-- and that is just my personal opinion \-- it seems a little
> redundant.
>
> MS. ALLINA: It seems to me it could be redundant but you have
> different numbers.
>
> DR. SVARSTAD: Well, but you see, there are two different issues here.
> There really are two different issues because as the form represents
> it, the question is whether or not the information was presented at
> all. Then, of the information that is presented, to what extent is it
> accurate?
>
> MS. ALLINA: Well, from a consumer perspective, the information isn\'t
> accurate if it is not complete.
>
> DR. SVARSTAD: Well, but you see, that is why we are trying to develop
> a form where you make clear what you mean by completeness \--
>
> MS. ALLINA: Sure, I understand that.
>
> DR. SVARSTAD: \-- specificity and accuracy. I appreciate where you are
> coming from, from a consumer\'s perspective, because, not being a
> pharmacist, I have probably been there. But I think there are a number
> of dimensions here \-- specificity, completeness, accuracy,
> legibility. All four of those dimensions could be used to evaluate the
> same sentence, and I think that is what we were trying to do.
>
> MS. ALLINA: It is very interesting data. Thank you.
>
> DR. SVARSTAD: Yes, thank you.
>
> MS. COHEN: A friend just told me I should give my real name but I
> will.
>
> \[Laughter\]
>
> MS. COHEN: You know it anyway.
>
> DR. OSTROVE: We know you!
>
> MS. COHEN: That is my problem! My name is Susan Cohen and I am a
> consumer advocate, and I have served on an advisory panel for the FDA.
> I have to tell you I have a lot of emotional reactions to what just
> happened. It is a panel of experts but consumers are experts in their
> own way too.
>
> DR. SVARSTAD: I agree with you.
>
> MS. COHEN: And, I feel very hurt that consumers were not really
> included. If I may make some suggestions, I think the food label has
> been one of the most successful things for consumers. Let\'s have a
> label on medication. Just tell consumers exactly right away what they
> can learn and what they cannot learn.
>
> I also think that every consumer should get a copy of their
> prescription so they can compare it against the medication they
> receive. I think consumers should get a list of generic questions that
> they should know how to ask the pharmacist, if they can ever see a
> pharmacist. Now, I can tell you that the pharmacy I go to is so busy
> that all I see is a clerk, and many consumers do that. Many pharmacies
> are under-staffed, and there really should be a ratio of the number of
> prescriptions filled to the number of pharmacists.
>
> So, I think there are some real problems for consumers. How are
> consumers reassured that they get the right medication? There is a lot
> of trust involved, but there should be something more that could tell
> them. Are there instructions to discard old medication?
>
> DR. SVARSTAD: I am sorry?
>
> MS. COHEN: Discard \-- that is my Boston accent; I don\'t know how
> else to tell you \-- to throw away \--
>
> \[Laughter\]
>
> \-- to throw away medication. Did you go into a very busy pharmacy to
> see how busy it is and what kind of attention people get? I was
> concerned that it wasn\'t multi-cultural, from what I can gather, in
> terms of your people going out. I mean, one of the things that the FDA
> encourages is that when they do any clinical trials they be
> multi-cultural.
>
> I just think that not to have really included consumers because they
> are the ones that really have to face the problem \-- because I am
> capable of doing something, it doesn\'t mean someone else is. So, with
> due respect to the Keystone and the study, I am profoundly troubled, I
> have to tell you, and I hope that you will be able to deal with
> consumers and talk to all of these wonderful consumer organizations,
> and I see some of the people here, and let people speak for
> themselves.
>
> I have to tell you my husband is a scientist so I am not in any way
> downgrading the people on the panel, and I am truly troubled, as you
> can tell, and I did give my true name.
>
> \[Laughter\]
>
> DR. SVARSTAD: Well, let me try to just respond to a couple of things,
> and I appreciate your comments. First off, I think I agree with you,
> as I stated, that in my personal view the lack of consumer input is
> the major study limitation. It was not our choice. So, I agree with
> you and I would like some support for that. I have spent my career
> being a consumer advocate, and that is why I am interested in this so
> I am in total agreement with you. I am not sure that I want the
> consumer evaluating the accuracy of information. When I said before my
> personal opinion, we all have opinions about this, but we all have
> roles in this but I would not want to be the one \-- I have glaucoma.
> So, I am a consumer of anti-glaucoma agents. Frankly, I do not want to
> evaluate the accuracy of my glaucoma information; I don\'t know about
> you.
>
> But, like you, I do want an input, and I don\'t want a paternalistic
> or maternalistic approach to it. So, I think we have to be very
> sensitive to that. But I would like to see some kind of multiple role,
> multiple perspectives in this. So, I am in agreement with you on that.
> I think we just have to think of ways to do that the next time around.
>
> Did we go to busy pharmacies? Yes. Did we keep track of whether they
> were busy? Yes. I am not presenting that data yet because that, again,
> wasn\'t part of the contract but we are going to check to see whether
> or not where there were some effects of business. That is, of course,
> somewhat hard to do because we were not allowed to collect any
> information about all these pharmacies.
>
> The third thing I think that you said is that you go to your pharmacy
> and you don\'t get information. I have two suggestions for you, if I
> may offer them. One is, you may have told your pharmacist of your
> unhappiness. I think they need to hear that. Then if they don\'t, go
> to a different pharmacy. It is not easy but I do think consumer
> expectations for a pharmacy have to include good information, both
> oral and written, and we have to communicate that to them. So, I am
> with you on that point. It is not easy though.
>
> Whether or not there should be standards about staffing, I think that
> is an issue for another workshop, but I am with you on that one too.
> Go ahead, Pat.
>
> MS. BUSH: Pat Bush, from United States Pharmacopeia, and I have been
> trying to think of where we should go from here. You have done a
> fantastic job in contributing to this evaluation of leaflets in terms
> of these ten criteria. I know how hard that kind of a study must have
> been.
>
> Looking at Keystone, after you get up to date and timely, it says
> useful. And, you don\'t have useful on this list because you really
> couldn\'t deal with it. As I know you know, USP has tried and has
> funded a study to try to evaluate the USP leaflets in terms of
> usefulness to patients.
>
> So, where can we go next? How can we get these kinds of things
> together? If we say that useful is the ultimate good \-- what are
> these leaflets for? They are for usefulness to patients. Right? If we
> can define that, then it seems to me what we need to do is take each
> of these criteria and format and see how much does each one of these
> contribute to usefulness. How can each one be made optimal? How can
> format be made optimal to contribute to usefulness? That should be the
> driving force. That is all. That is not a question.
>
> DR. SVARSTAD: Well, no, I think it calls for lots of input and lots of
> discussion here as to what, in fact, is useful, and I know that is
> what the Keystone panel was trying to do, but this is not simple. I
> agree with you and I think we just need to discuss it and keep
> discussing it.
>
> My understanding of this or my view on this is that I think this is
> going to evolve. You know, our definitions of usefulness, our
> definitions of accuracy, our definitions of completeness are likely to
> evolve, and that is the way it should be. I don\'t see how we can do
> this all in one event. It is too complicated. Yes, ma\'am?
>
> MS. HARE: My name is Doris Hare, and I am with the American Foundation
> for Maternal and Child Health. My concern is that all of these inserts
> should have a very clear warning regarding use during pregnancy, and I
> would like to see us say, use in pregnancy: none of the drugs approved
> by the FDA has been subjected to a follow-up to determine whether or
> not a drug may result in delayed long-term adverse effects on the
> child exposed [in utero]{.underline} to the drug. That is an absolute,
> undeniable fact, and women have a right to know that.
>
> I mean, this past season we have had tremendous experimentation, I am
> sure, with the flu season and everyone taking different kinds of drugs
> and, yet, there is not even an attempt to evaluate the effect of these
> drugs on the baby. Thank you.
>
> DR. SVARSTAD: Thank you.
>
> MR. NESBITT: I have a methodology question. Scott Nesbitt, with Health
> Resource. Did you weight any of the results based on the size of the
> pharmacy at all?
>
> DR. SVARSTAD: No.
>
> MR. NESBITT: That might be something that we want to consider given
> that with pharmacy A and pharmacy B one does 100 scripts a day and one
> does 600 scripts a day. As your pilot study showed, if you go in a
> second time you get the information \-- you didn\'t need to do more
> than one visit. So, in reality, if you get information from the larger
> pharmacy a lot more people are getting that information. So, the 87
> percent of people getting information is really 87 percent of
> pharmacies are distributing information and that might understate the
> actual number of people that are getting information.
>
> DR. SVARSTAD: I think there are a lot of questions that you could ask
> about the characteristics of the pharmacy in addition to size. One is
> availability and capacity of their technology, their computers, their
> printers, their staffing, etc. So, that is a whole study in itself,
> and a very important one but one that we couldn\'t do without that
> information. We had no way of determining what the volume was.
>
> MR. ROSS: John Ross, Ely Lilly and Co. I was just curious how many
> were more than one page long? What percentage?
>
> DR. SVARSTAD: We did not evaluate number of pages.
>
> MR. ROSS: Okay.
>
> DR. SVARSTAD: I would say the overwhelming percentage of all
> information sheets \-- I only remember one that exceeded one page.
>
> MR. ROSS: Thank you.
>
> DR. SVARSTAD: Maybe there were more than that; maybe ten, twelve. Most
> of them were one sheet.
>
> MS. SMITH: Dorothy Smith, with Consumer Health Information. I just
> want to congratulate you on this first step, this study. I think it
> really gives us a lot of good information.
>
> I am listening to the comment from the consumers and health
> professionals, and what I am hearing is that they want useful
> information. I agree with you totally that written information should
> never replace the verbal or the oral information, and I am wondering
> if one of the next steps \-- maybe you looked at this, the behavior
> modification techniques that might have been used in some of the writ
> ten instructions. You mentioned briefly color. I am wondering did any
> of these instructions address specific patient compliance problems
> unique to that drug or disease, or patient population? And, did any of
> them take medical illustrations to describe the condition, to help the
> person look at a picture?
>
> DR. SVARSTAD: We did not count the number that had any illustration
> but I, frankly, don\'t recall any.
>
> MS. SMITH: Okay, I think that may be an area in the future.
>
> DR. SVARSTAD: Yes, that is one of your questions. I think if we had
> been looking at a glaucoma agent or an asthma inhaler we would see a
> very different result on that, Dorothy.
>
> Then, your other question related to adherence \--
>
> MS. SMITH: Behavior modification.
>
> DR. SVARSTAD: Maybe you should clarify how you are using behavioral
> modification.
>
> MS. SMITH: I am thinking that when you write the instructions, or
> before you even start, the people with depression are going to have
> different needs than the people receiving ibuprofen, or whatever, and
> when you start writing the patient leaflets you can be more effective
> than just describing a list of instructions about side effects and how
> to take the drug. If you can give them some information about the
> disease and just guide them and convince them to take the medication.
>
> DR. SVARSTAD: Convincing them to take the medication, or about the
> disease \-- I think I earlier made the comment, and I would reinforce
> again that I think in general we didn\'t see as much information about
> benefits and I would fit that into the category. I think the Keystone
> report seemed rather silent. I know it said that there should be
> discussion about the benefits and how the patient can maximize the
> benefits but I think it was rather silent about the amount of
> information and didn\'t say anything that it is important to discuss
> the odds of having benefit. I have been kind of intrigued with what
> would happen if we started doing that because, as a consumer, I think
> it would be useful in making a decision whether or not to initiate
> therapy if you know something about the benefits and the chances are
> that you will benefit from it. But that wasn\'t included. So, we did
> not add that. We tried to adhere to the Keystone criteria.
>
> On the compliance, I am not sure \-- I am thinking, for example with
> paroxetine, my understanding of the literature is that one of the main
> reasons that persons with depression discontinue paroxetine or any
> other antidepressant, for that matter, prematurely is that they do not
> fully understand that benefits are not likely to occur until after
> some weeks. That was included in the evaluation form.
>
> MS. SMITH: Or they develop side effects.
>
> DR. SVARSTAD: Yes.
>
> MS. SMITH: Like with the antihypertensives, 50 percent of the
> population stop taking them in the first year.
>
> DR. SVARSTAD: Yes.
>
> MS. SMITH: So, it depends on the disease and the drug and the
> population.
>
> DR. SVARSTAD: Yes.
>
> MS. SMITH: But I just really think that your study has really helped a
> lot in the fact that you have pulled out the comprehension as one of
> the areas that we just need to look at a little bit more.
>
> DR. SVARSTAD: Yes, thank you.
>
> DR. OSTROVE: Any other questions? If not, I think we ought to give a
> hand to Dr. Svarstad \--
>
> \[Applause\]
>
> Well, certainly the interest level is still high. We really appreciate
> all these comments, and I am trusting that at least 95 percent of you
> are going to be here tomorrow for the small group breakout sessions
> where we can talk in more detail about essentially how we want to deal
> with these results in terms of what we are going to do the next time
> around, that is, for the evaluation in the year 2000.
>
> I just wanted to let you know that tomorrow morning there will be a
> Continental breakfast at 8:30. We will be starting promptly \--
> approximately promptly at nine o\'clock. You know how it goes around
> here; you never start on the dot. Dr. Janet Woodcock, the Director for
> the Center for Drug Evaluation and Research, will be addressing us at
> that time. We will be moving to the breakout rooms at about 9:30, and
> in those rooms \-- locked, no refreshments \-- I am joking \-- from
> 9:45 till noon \-- there will be a break. I think the way we will
> probably set it up is that when you need a bio-break, just go and take
> it, rather than having set breaks because you know what happens with
> set breaks \-- you set it for ten minutes and it goes to twenty
> minutes and we do have a lot of information that we would like you to
> deal with in a short period of time.
>
> We will have a buffet lunch at noon, and we will be reconvening to
> discuss the results at around 1:15, and then finishing by three
> o\'clock. If today is any example, we will finish a little earlier
> than that but I am not promising anything.
>
> Now, important reminders: Please bring your badges because the secret
> to your small group breakout sections is on the badge. Also, I would
> suggest that you bring your packets. The facilitator should have a new
> list of the ten criteria but we feel it would be helpful for you to
> have access to all the information in the packet in case you do want
> to kind of consult with it. A number of us will be moving around from
> group to group and we will kind of serve as information people, if you
> need that.
>
> Again, thank you very much for coming. We really appreciate your all
> being here, and we hope to see you all tomorrow. Take care.

\[Whereupon, at 4:55 p.m. the proceedings were recessed until 9:00 a.m.,
Wednesday, March 1, 2000.\]