DEPARTMENT OF HEALTH AND HUMAN SERVICES

> FOOD AND DRUG ADMINISTRATION
>
> CENTER FOR DRUG EVALUATION AND RESEARCH
>
> **CURRENT STATUS OF USEFUL WRITTEN PRESCRIPTION**
>
> **DRUG INFORMATION FOR PATIENTS**
>
> **VOLUME II**
>
> Wednesday, March 1, 2000
>
> 9:00 a.m.
>
> Double Tree Hotel
>
> 1750 Rockville Pike
>
> Rockville, Maryland
>
> [C]{.underline} [O]{.underline} [N]{.underline} [T]{.underline}
> [E]{.underline} [N]{.underline} [T]{.underline} [S]{.underline}
>
> Welcome, Nancy M. Ostrove, Ph.D. 3
>
> Opening Remarks, Janet Woodcock, M.D. 4
>
> Small Group Assignments, Nancy M. Ostrove, Ph.D. 16
>
> Results from the Breakout Sessions,
>
> Nancy M. Ostrove, Ph.D. 22
>
> Discussion of Questions:
>
> Question One 24
>
> Question Two 34
>
> Question Six 44
>
> Question Five 54
>
> Question Seven 60
>
> Question Four 65
>
> Adjounrnment 80
>
> [P]{.underline} [R]{.underline} [O]{.underline} [C]{.underline}
> [E]{.underline} [E]{.underline} [D]{.underline} [I]{.underline}
> [N]{.underline} [G]{.underline} [S]{.underline}
>
> **Welcome**
>
> DR. OSTROVE: Good morning, everyone. Welcome back. I know that you all
> probably have very big heads from having to process all that
> information yesterday. I have to tell you my brain is definitely
> tired. In fact, my husband said that I sat up in bed a couple of times
> over the course of the night. I am sure that it was revelations, you
> know, about how we should best do the Year 2000 assessment. The most
> distressing part of it, of course, is did I remember any of them? I am
> afraid not. So, we are depending on you to bring those back.
>
> We are all sure that you have many thoughts to share and today is the
> day that we want to focus on that. Before we do that just for a little
> bit of time, we would like for you to hear from the person who for
> years has been supporting the need for all stakeholders, and
> especially patients, to receive information that is critical to their
> understanding of how to use prescription medicines safely and
> effectively in a way that will minimize the risks, while maximizing
> the benefits that they receive from the medicines. So, I would like
> you to welcome someone that I am sure you all know and respect, Dr.
> Janet Woodcock, who is the Director of the Center for Drug Evaluation
> and Research.
>
> \[Applause\]
>
> **Opening Remarks**
>
> DR. WOODCOCK: Thanks and good morning to everyone. It is wonderful to
> be able to address an audience of people who are really committed to
> this idea of patient information. Many of you have really spent your
> careers and much of your effort around this information.
>
> I want to talk to you about the recent ideas that have been developing
> in FDA around risk management and how this whole idea of patient
> information feeds into risk management, and what this means.
>
> As I said, you all are probably a more receptive audience for this
> type of thinking than possibly a lot of the scientists at FDA. I was
> talking to Nancy right before this session about this. Every time we
> talk about patient information at the FDA I think more and more people
> begin to understand the concepts and buy into the concepts, and there
> is quite a bit of momentum around this, but there is an aspect of \--
> what shall I say? \-- it is hard to say this tactfully, but
> professional elitism or something like that, or professionalism, the
> people who own the information or are technically trained in
> scientific information or medical information \-- there is some
> reluctance for those folks to share that information, I think, in a
> way that is understandable for other people. Some of that is a concern
> that they won\'t understand. Some of that is the fact that knowledge
> is power, and those who have the knowledge have the power. Some of it
> is really failure of imagination, I would say, the inability to step
> into that other person\'s shoes and into themselves and understand why
> they need to know that stuff just as much as the professional does.
>
> The idea of risk management at FDA really surfaced about a year ago
> when a number of drugs had been withdrawn from the market. Some of
> them were drugs that had been approved for a very long time. Some of
> them were drugs that had been recently approved. And, there was a lot
> of concern amongst the world that FDA was reviewing drugs too fast and
> putting them out onto the market, and so forth, with little regard or
> less regard than previously for safety of the public.
>
> So, we went back and looked at our processes and looked at the numbers
> of withdrawals, and so forth and so on, to see if, in fact, any of
> that might be the case. Our conclusions in that regard were that no,
> that wasn\'t the case; that actually the withdrawal rate was lower
> than in the past, and the number of very serious new events discovered
> after the drug was on the market was somewhat lower in the past. But,
> the point is those risks are still there even though they are
> discovered before drug is marketed. Those risks still pertain to the
> drug. Even if they are in the label, people are still exposed to the
> risk.
>
> So, we went a step further and we looked at the system in this country
> for managing the risks of pharmaceuticals. How is that done? So, we
> tried to step back and look at the big picture and think how in this
> country do we make sure that benefits are maximized for patients and
> risks are minimized for patients with respect to taking medicine.
>
> And, what we found, and we weren\'t the original authors of this \--
> these thoughts had all existed out there in the world, is that over
> the last hundred years or so we have operated under a professional
> model, and that professional model has been that the healthcare
> professionals \-- the prescribers, pharmacists, the nurses, and so
> forth, will be trained and credentialed and will know everything that
> is needed to be known, and they will make sure that the patient gets
> the correct drug, and at the correct dose, and they will monitor the
> side effects, and they will adequately instruct the patient.
>
> As has been pointed out by the National Patient Safety Foundation and
> others, within that model then, if anything goes wrong, the
> professional model calls for blaming somebody \-- calls for blaming
> usually the professional who is closest to the problem and that person
> might be the pharmacist who dispensed something incorrectly because he
> couldn\'t read the handwriting. It might be the physician who wrote
> the wrong dose of a drug, or the nurse who formulated the drug wrong,
> or it might be the FDA that is blamed because the drug is causing more
> side effects than were anticipated.
>
> In other industries though what has been discovered is that most of
> these type of ill effects are, in fact, system problems. What does
> that mean? Well, that means that this whole thing is a big system and
> you can\'t just pull out one piece of the system and blame it when
> something goes wrong. For example, say if a patient\'s wrong leg is
> cut off \-- that is a notorious event. That is a terrible event that
> gets in the paper, and everything, and catastrophic for the patient,
> and everything. And, the immediate response, they will blame the
> surgeon. Right? Of course. When you read that article you blame the
> surgeon. How could the surgeon do such a terrible thing? That is a
> terrible thing. But when you go back and you try to reconstruct what
> happened and how did this all come to happen, what you find is, well,
> on the x-ray machine the technician put the wrong label on for the
> x-ray. So right and left were switched. And then, perhaps when the leg
> was prepped, the prep team prepped the wrong leg. And then, when the
> patient was put in the operating room, they exposed the incorrect leg,
> and the patient was all covered up and put to sleep, and everything,
> and there was the leg sticking out. The surgical team comes in, does
> the surgery.
>
> It is a comedy of errors; it is cumulative mistakes that occur all
> through the system. If one piece of this series had been caught, this
> whole thing wouldn\'t have happened. That is what occurs, in fact,
> with most errors that occur, preventable problems that occur within
> healthcare. So, there is a whole cascade of events that lead to a bad
> outcome.
>
> With medication errors \-- you know, problems in the dispensing of
> drugs, and so forth, most of them actually never reach the patient.
> There are enough fail-safe mechanisms in place that somebody catches
> the problem and the medication error is stopped before it actually
> impacts on the patient. But because there are so many prescriptions in
> the world written today, 2.5 billion I think prescriptions written,
> even a tiny fraction of medication errors can lead to a significant
> number of problems for people.
>
> So, my point in all this is that we stepped back and we said, okay,
> we, at FDA, approve medicines. We put them out there. Our attitude has
> always been, all right, we have studied them and the good things are
> in the label and the problems are in the label and now the system out
> there \-- it is their problem. They have to handle this. They can read
> the label. They can understand the problems. The professional system
> is in charge; that is the practice of medicine. I have heard that many
> times from the people at the FDA. That is the practice of medicine; we
> don\'t really have an impact on that.
>
> But if you look at a system\'s approach, if you step back and you
> think these outcomes, whatever they are, out in the real world are a
> result not just of one professional and their failure to read the
> label, or whatever, it is the result of a whole series of events and
> structures and processes that have been set up for healthcare, and you
> can see that every part of that chain has responsibility for the
> events that occur, whether they are for good or for ill for the
> patient, for the ultimate patient.
>
> This is where patient information comes in. I don\'t really need to
> talk to you all about the role of the patient because I suspect
> everyone here has a very good appreciation of that. But if you look at
> emergency room visits or admissions, you see that a tremendous number
> of visits, hospitalizations, and so on, are from simple non-compliance
> with medications. All right? And, if you wanted to take this
> professionalism analogy a little bit further you would say, well, that
> is the patient\'s fault. They didn\'t take the medicine. Right? And,
> then we can just blame somebody else \-- that is their fault.
>
> Well, the question is did they know the consequences of not taking
> their medicine at a level of true understanding of that? Did they know
> the consequences of drug-drug interactions? Did anybody tell them? Did
> they know they should come to their doctor if they experience whatever
> side effect it might be? Not were they informed in some way; did they
> know? And, we really can\'t blame people if the system has failed
> them, if it has failed to allow them to arm themselves sufficiently
> with information to protect themselves.
>
> This is the same with doctors I think. I am a physician so you might
> say but I feel that way about nurses and pharmacists and everyone else
> in the chain. If the system fails them, is it really right to blame
> them for that one event that occurred that was the result of a system
> failure?
>
> Now, I regard patient information as a real spectrum of stuff, and
> what I think what is being talked about here is just one facet of
> that, which is the written information that is given to patients, for
> example. Obviously, reinforcement is so important in learning that the
> prescriber should inform the patient. The pharmacist should then
> inform the patient, and that could be reinforced with written
> information \-- this is the ideal, in the language that the patient is
> fluent in, of course, and if they have trouble absorbing written
> language, which many people do in our society, it would be really nice
> to have videos and other things, and most people in our society seem
> to be able to manage to watch television. I have been saying this for
> years. I have kids. I have never seen a medium with such attraction as
> television. You know, it sucks people over to watch it.
>
> So, we shouldn\'t limit ourselves if we are really trying to address
> the actual problem, which is people are being harmed because of
> ignorance about the medicines they take, and their are failing to
> achieve the promise or the benefit of those medicines because they
> don\'t understand. Now, at some level if people are fully informed and
> they still choose not to take their medicines, or they still choose
> not to deal with the side effects, or to take interacting drugs, or
> something, then I would say at that time, well, that is their
> responsibility. They have to be accountable for that because in a free
> society those are the choices they have made.
>
> But most of the people who roll into the emergency room with their
> non-compliance, or they are using their drugs incorrectly, or they are
> waiting so long to come in while they are experiencing terrible side
> effects \-- it is out of ignorance. They do not truly understand what
> the consequences will be to them, and they have not been truly
> informed. So, I think what this is all about, as well as many other
> efforts I hope the FDA and others will be taking, is to really think
> about what are all the ways we can help maximize the benefits of the
> medicines that are available to people, and really improve their
> health, and because drugs always have the potential for harm \-- every
> one of the drugs \-- how can we minimize that harm and manage it? That
> is the risk management part \-- manage it appropriately.
>
> Now, effective communication is the key to this. It isn\'t enough, as
> I said earlier, to just give people the information. It has to be
> reinforced and it must be in a form that they actually can absorb and
> use, not just a paper handed out to them.
>
> I remain concerned about the quality of information that people are
> getting, and I guess we have been hearing that at this workshop. It
> isn\'t easy. We are not only trying to overcome the barriers that were
> identified during this whole process, the logistical barriers of
> getting this information out, and the type size, and everything, we
> are trying to overcome a century or more of the professional model
> that really people don\'t need this information and probably
> shouldn\'t have it because they might be worried, or something like
> that, or they wouldn\'t be able to understand it.
>
> I think getting technical and scientific stuff into a form that people
> can truly absorb and understand is a tremendous challenge. We have
> been trying to do this at the FDA. When we approve a new drug we put
> information up on the web. We try to make some of the information in a
> form that consumers can understand, and I can tell you it is a
> tremendous challenge.
>
> The reason I have so much passion about this, sometimes patient
> inserts are a part of the unit of use packaging of newly approved
> drugs, and so I get to see those when they come through and uniformly
> they are pretty terrible. They are terrible. I have seen ones that you
> would lie down on the floor laughing at them; you would think who is
> this directed at? You would have to be, you know, a Ph.D. graduate to
> even begin to access some of the information in these things that are
> submitted. And, our scientists are, well, you don\'t have enough of
> the information in here; you ought to add this piece, and you ought to
> \-- you know, this isn\'t quite right technically. You ought to put
> longer words in there.
>
> So, we aren\'t up against just some political difficulties or
> logistical difficulties. We are up against a huge societal barrier
> here about technology information and really the bias of experts that
> ordinary people can\'t understand this information. That is what I
> think. For me, that has been totally borne out in my experience over
> the past five years with this.
>
> So, I think the FDA needs all the advice we can get. That is why we
> have had this workshop because we need the input from the people in
> the field who are working on this, the people who care about this as
> to how to do these surveys, how to move this process forward. How can
> we determine, as best as possible, the quality of this information in
> the sense of is it really useful? Does it give the information that is
> needed to people to protect themselves and to manage their medicines
> as best as possible?
>
> So, I appreciate everyone\'s attendance here. I hope you are really on
> task today as you provide us input. I don\'t think any of us can
> overestimate the magnitude of the task that is involved and, as I
> said, I think the written insert or the written patient material is
> only one facet of this whole problem. There are a lot of people to
> whom you give written information that will not allow them to learn
> what they need to know. So, we really need to think about that as
> well.
>
> But there are many people in this country who do read and who do use
> the written word to get their information, and who can refer to that
> when they are at home or when they are experiencing a possible side
> effect, and so forth. That is something that people can go and look
> at, or when they are considering not taking a medicine anymore, for
> example, and being non-compliant with their regimen, having that
> information there and why you should take it and why you should finish
> it, and so forth, I think could be very helpful in many of these
> areas.
>
> So, we really are dedicated to moving the risk management system
> forward in as many ways as possible, and one crucial link in making
> sure that medicines are used as safely as possible is making sure that
> that information gets right down to the level of the person who has to
> take the medicine, or the parent who has to mange the child\'s
> medicine.
>
> So, this is crucial for us. I think it is crucial for the
> manufacturers of pharmaceuticals \-- and I see a number of people from
> the manufacturing sector here \-- because these drugs are only going
> to be as safe as they are used. We can study them extensively
> premarket, and at the FDA we can review them meticulously and we will
> write all the risks down right on that package insert, and we can
> write the benefits down, and so forth, and if that information is not
> used correctly out in the world, if it isn\'t maximized then the
> medicines are not as going to be as safe as promised and they are also
> not going to be as useful as promised. So, in order to really make
> drugs as safe and effective as we can this is part of the whole
> management chain. I can only promise you that the Center for Drugs at
> least will continue to make this a high priority. We are very
> interested in input on this survey and any other ideas about what can
> be done to improve this information.
>
> Thank you very much. Also, I am willing to take any questions people
> might have for a few minutes. Any questions? No? Okay, thanks.
>
> \[Applause\]
>
> **Small Group Assignments**
>
> DR. OSTROVE: Thank you, Dr. Woodcock. We really appreciate it.
>
> Now, all of you have been patiently waiting your turn to get down to
> work. So, I am going to give you instructions about how we are going
> to do that this morning. Remember I told you, you had to bring your
> badges. Did everybody bring their badges back today? There is a number
> on your badge. That indicates the room or place assignment for you.
> This was done in a random fashion. We didn\'t try to stack any of the
> groups, or anything. So, any cases where you have lots of one
> particular perspective, again, that is part of the whole random
> process and you ended up out here, you know, on one of the ends of the
> distribution.
>
> So, if your badge says group one or group six, you will be in the
> Twinbrook Room. If your badge says group three or group four, you will
> be in the Montrose Room. For both of those, when you go out the door,
> you make a right and then you make a right again. The rooms are along
> that corridor leading toward the front of the building. Group five
> will be in the front of this room near the door. If you look at that
> flip chart, you will see it says group five over there. Group two will
> be in the front of the room on the far wall, right over there. Group
> seven will be in the back of the room near the door, in that corner.
> Group eight will be in the back of the room on the far wall.
>
> Clearly, you will probably have to take some chairs and kind of move
> them around so that you are in a comfortable position. The
> facilitators that we have assigned to each of these areas will direct
> the process that we have recommended the groups follow, just basically
> to ensure that all the members have a chance to express their
> opinions.
>
> I just want to give you a few points first, at the risk of frustrating
> you; I am sure you are all itching to get started. We won\'t have any
> breaks. We will simply kind of move to the areas and start from there.
> We just decided that it would be better from a process standpoint to
> not have a set break. So, you will be in those areas from about 9:45
> to noon. If you need to take a bio-break, to right ahead. If you need
> to stand up and walk around, don\'t feel constrained. I am sure you
> all don\'t need to be told that, but just in case.
>
> Now, one thing I wanted to point out, this is clearly not the Keystone
> process but, just as Judy O\'Brien mentioned yesterday, I think it is
> important for everyone to realize that everyone in this room is trying
> to do the right thing and has reasons behind their perspectives, and I
> just think we want to keep that in mind. I heard someone this morning
> say something about the Keystone headache, that this person had that
> had gone away about three years ago, came back after yesterday\'s
> meeting. I felt bad about that, but this is not the Keystone process.
> This is not formally a consensus-building exercise. There is not
> enough time here for people to try to persuade each other of the
> rightness or any particular position. We want all of your thoughts and
> your recommendations to the extent that we have the time to get those
> out.
>
> However, we also would like some sense of degree of agreement with
> different positions. So, don\'t be surprised if the facilitators ask
> for this kind of information, just so that in terms of the
> facilitators presenting the results of the group conversations after
> lunch we can get a sense of where these different ideas come from
> because some of them may, in fact, be contradictory.
>
> Now, we are interested in specifics to the extent possible. If your
> specifics are really specific, remember that the docket is going to be
> open until April 28 and if you don\'t get everything out that you want
> to get out today, or if you want to have it in a written form, please
> submit your comments to the docket. The number is 00N0352, and it will
> be open until April 28. So, I just wanted to remind you of that.
>
> Now, the way we want to set it up is we would like all groups to
> address the first two questions. In your packet there is a single
> sheet that has a list of seven questions. The first one regards the
> minimum standard or threshold that should be determined for a piece of
> information to be considered useful \-- well, in general actually for
> the goal to be met.
>
> The second is should certain criteria be weighted more than others? If
> so, which ones and why?
>
> So, we would like everyone to address those two questions. We have
> raised a number of other issues, five other issues at least, that
> express some of the concerns that we have had as part of this process
> of the interim study. We hope, but we don\'t really expect, that we
> have addressed all of your issues, and we are sure that if there are
> oversights you will let us know about them.
>
> What we kind of figured, again from a process standpoint, is that we
> have time for each group to address two additional questions, in
> addition to the first two on that list. So, what we have suggested is
> that the facilitators take a vote at the beginning of the session and,
> just with a hand vote, determine which other questions the group will
> address. If you can get to more, more power to you. You know, we are
> not saying you can\'t do that but we feel that another two is a
> reasonable way to go.
>
> Now, to seem to contradict myself with regard to the detail issue \--
> you know, I said don\'t get too specific; we want specifics but not
> too specific \-- if that isn\'t ambiguous enough for you, let me
> suggest some details that you might want to address in looking
> especially at question one.
>
> Do you see other ways of combining these data to obtain a final score
> that would represent the standard, or the minimum criteria that should
> be used to reach this goal of usefulness? If so, what are they? If you
> go for something similar to the one-point method that Dr. Svarstad
> talked about yesterday and that is in the report, please consider
> carefully the issue of whether credit should be given for full versus
> partial adherence to the sub-criteria under each of the general
> criteria. This is an issue that I heard raised. I don\'t know whether
> it was in the public forum or if it was someone I spoke to afterwards,
> but I heard it raised yesterday.
>
> We obtained information related to three drugs, and we selected those
> drugs for a number of reasons, one of the major ones being logistics
> in that how much could we afford to buy and pay for out of pocket, and
> which ones could you pick up at the same time and not appear to be
> doing something really bizarre? We had to be able to set up a scenario
> where people could pick up three drugs at the same time.
>
> How would you combine these scores to create some kind of a single
> score? As you can see from all of the tables that Dr. Svarstad
> presented, they are all separate \-- ibuprofen, paroxetine,
> amoxicillin. How do you combine those?
>
> If you address the question of consumer involvement, which we
> certainly heard about loud and clear yesterday, please try to address
> what kind of involvement; how much involvement; what degree; on what
> particular criteria should consumers be involved?
>
> So, those are just some initial thought that I had, and you may decide
> not to deal with any of those and that is your prerogative. I have
> about 9:35. Does anyone have any questions before we move? No? Good.
> Why don\'t we move to our rooms. Some of us will be floating around
> but you will have facilitators, and please, be open.
>
> \[Whereupon, at 9:39 a.m., the proceedings were recessed, to be
> resumed at 1:15 p.m.\]
>
> A F T E R N O O N P R O C E E D I N G S
>
> **Results from the Breakout Sessions**
>
> DR. OSTROVE: If I could have all the facilitators who will be doing
> the speaking up at the lead table here, and the facilitators who will
> be helping out over at the overhead station, that would be great.
>
> Let me give you kind of the layout of how we are going to handle the
> rest of the afternoon in terms of reporting back on the results of
> your discussions which, having kind of sat in here and there on all of
> them, were definitely very interesting and there is definitely a lot
> of information there. This will certainly be a large part of getting
> out on the public record. We are all here and, of course, we will have
> all the acetates, all the overheads, everything will be looked at.
>
> We have kind of decided on a particular format for doing this. Who
> knows if it will work! If anyone comes across a way of doing this kind
> of reporting out that you have found works especially, let us know.
> Let me know, in any case. What we have decided to do essentially is to
> have each one of the pairs of facilitators in the different groups,
> except one group that had three, sit at the lead table. Each one will
> take a lead on a particular question. What I mean by that is rather
> than have us go through what every group said on every single question
> that they addressed, which can get a little bit redundant after a
> while, we will have one group start out and discuss everything that
> they addressed for one particular question and then, if other groups
> want to add to that because they have something different, then we
> will kind of go down the line and add to those. Then a different group
> will take the lead on the next question.
>
> Believe me, I had all kinds of little slips of paper, you know,
> deciding who was going to take the lead on what question depending on
> which groups had addressed which questions. Just for your information,
> of course, everyone addressed questions one and two, no one really
> addressed question three in any detail. So, no one is really going to
> take the lead on that. I will just kind of open it up at one point and
> ask if anyone has anything to say about this.
>
> Question four, two groups discussed. Question five, six groups
> discussed. Question six, all groups addressed, and question seven,
> three groups discussed. So, just in terms of what people found
> interesting and something that they wanted to discuss today, that is a
> little bit of data.
>
> So, what I would like to do is start with question one. But before I
> do that, let me thank all the facilitators because you guys have done
> a great job under a lot of pressure and I really appreciate it, Nancy
> \-- we all really appreciate it, especially those of us who didn\'t
> have to do it and got to kind of walk around. That is the best part.
> Starting with group two, to start the discussion on question one,
> start the feedback, that was Nancy Smith and Gina Choy. Nancy is up
> here; Gina is down there.
>
> **Question One**
>
> DR. SMITH: As we get started, I wanted to share a cartoon that was
> given to me a few minutes ago because I think it is extremely
> appropriate for this. This is from yesterday\'s paper. It is Beatle
> Baily, and Beatle is talking to Zero and most of you who look at the
> comics at all know that Zero is not your most intelligent person. This
> is the sergeant talking to Zero, and the sergeant says, \"follow these
> directions exactly and you won\'t get lost.\" Zero says, \"yes.\" And
> later on you see the sergeant says, \"why didn\'t you show up?\" And,
> Zero is actually crawling down the middle of the road and he says,
> \"I\'m looking for the fork in the road.\"
>
> \[Laughter\]
>
> I think part of what we are doing right now is people not
> understanding what we mean by the fork in the road. You know, it is
> not what we just ate lunch with, and that is what he is looking for.
> So, determining what is useful is really going to be dependent on the
> audience, much more so than the health professionals, I think, and
> that is why I think it is very interesting that every single group
> addressed question six. They had to choose two and everyone included
> that one. So, we know that consumers are important.
>
> What should be the minimum standards? This one was an interesting
> discussion and, to begin with our group, after some discussion,
> decided to distinguish between what were the criteria that the study
> used and the components of the Keystone report. Many of the criteria
> were components but some of the criteria were not components.
>
> So, Gina, could you put that up first? The Keystone components, what
> the Keystone report says it should include are the name, warnings,
> indications, contraindications, precautions, adverse events, risk of
> dependence, proper use, storage, general information and a disclaimer.
> Now, the things within these criteria that are really more criteria
> \-- they are not components, that what it is be scientifically
> accurate; that what it is, whichever of those, be unbiased, be
> legible, be up to date and be in enough detail. Each of those
> categories can be applied to each of these components, and this is
> sort of a distinction. It is not just one part of it, and it is not
> enough to say the list of adverse events is scientifically accurate;
> all of those need to be scientifically accurate. All of them need to
> be up to date, and so forth.
>
> So in terms of the first, we felt that a minimum standard should be
> that it addresses 100 percent of those components. Everything that was
> considered a component should be included. And, they all must be
> compared with respect to scientific accuracy, the unbiasness, and so
> forth.
>
> Our group could not come to consensus. We ranged from at least one
> individual who felt that anything less than 100 percent was failure;
> that the Keystone report stated what should be included and how it
> should be and it was either that or nothing, and had major problems
> with evaluations, like yesterday, where partial or complete was
> considered complete in some of the analyses.
>
> I think there was probably a larger group that felt that you have to
> be realistic, that these were subjective and, without really mandating
> exactly what little each part of it had to say, there had to be some
> flexibility here but, at least within some reasonable range, it
> certainly needs to meet all of these and, clearly, if any component is
> totally left out then it doesn\'t meet those. Now, that doesn\'t mean
> two of them couldn\'t be combined in the same thing or something like
> that, but they would all have to be included.
>
> We also felt very strongly that we needed to check with consumer
> groups ahead of time in order to see what they really felt was
> necessary for calling information useful, and the suggestion was made
> that we use focus groups to really get a wide variety of people to
> discuss what they felt would be useful information, and what would
> make them read the information not just, you know, throw it away in
> their trash can when they get home.
>
> We had a minority opinion that said patients should receive all of the
> information, the same information as the healthcare professionals do.
> This was a minority opinion. The majority of the group felt that there
> is information that health professionals need to know about that the
> individuals within the group do not.
>
> But, basically, our group said for number one that you have to address
> all of the components, and you have to do a reasonably good job,
> although we didn\'t define reasonably good job, of covering all of
> them to be considered.
>
> We felt that much of what we saw yesterday did not meet our standards
> for being acceptable.
>
> DR. OSTROVE: Why don\'t we, for this particular time, start with Tom
> and move down? Tom, do you have anything to add from your group?
>
> DR. ABRAMS: Sure, just a couple of things. We found in our discussion
> a lot of things that Nancy Smith went over. A couple of things that we
> also pointed out was that standards should be tied to existing
> research. There was pretty much a consensus in the group that FDA or
> an expert panel should not make up new criteria. The information is
> out there for standards and we should seek that existing thing rather
> than reinventing the wheel. Also, the feeling was that probably it
> should be more objective.
>
> The second point is that the criteria should be weighted, and I think
> we go into that in question number two so I won\'t spend any time on
> that.
>
> Then, one thing that was interesting that came up is that we need to
> consider the outcomes of this patient information. What is it going to
> do? For example, behavior modification, what is the goal of it \-- to
> make the patient more compliant taking the medication, for example, or
> if a patient should have an adverse event, the patient should know
> what to do \-- call the pharmacist or physician or how to act with it.
>
> Also, under the same thought, the group thought it would be helpful if
> this prompted some dialogue between the patient and healthcare
> professional, whether it is the pharmacist or physician. We recalled a
> comment from somebody who questioned how the pharmacists were so
> isolated from giving advice so the group thought it would be useful if
> this information could prompt a discussion as well.
>
> MS. ACKER: Yes, we had some discussions around some different points
> on this one. We talked about whether the threshold should be full
> adherence to the criteria or partial adherence, whether that would be
> good enough. The group felt that the goal should be full adherence,
> that criteria should be met fully according to the panel\'s definition
> of full adherence, but that the minimum would be greater than 80
> percent. So, that would correspond to level five, the high adherence
> to criteria.
>
> We discussed the fact that how you define what partial adherence is,
> is an issue. The point was brought up that if you were given credit
> for partially adhering to a criteria that, you know, the majority of
> the side effects that were listed in the package insert were also
> listed in your patient information, whether that was the cut-off or
> whether it was, you know, just some side effect information present
> that would meet the full adherence measure.
>
> The group thought \-- this sort of speaks to question two as well in
> terms of whether certain criteria should be weighted more than others,
> that maybe there are some criteria where partial adherence would be
> acceptable but that for other criteria we would want to see full
> adherence.
>
> There was a lot of discussion that depending on the drug there may be
> different thresholds that you could accept. If there was a drug that
> had a very significant side effect profile or there were significant
> warnings that needed to be heeded, you would want a higher threshold
> for information on that drug versus something that would have a more
> benign profile.
>
> MS. WHEELOCK: Our group got into some more details. I am looking at
> consumer input needed for the forms, such as legibility, readability,
> printability \-- also to the function which, Nancy, you addressed
> earlier, considering looking at the individualized content tailored to
> the patient. We also looked at whether the content was useful and
> practical, identification of disease, symptoms, side effects and
> management, making the scientific terms translatable to the patient
> and clinically relevant; also looking at cultural diversity \--
> written in the language of the consumer and appropriate for
> geographical area; and also recommending sticking to the action plan.
>
> MS. LECHER: Greetings from group five. We had some of the same issues
> that were already brought up but two additional things that have not
> been covered yet are that we made a distinction between items that
> could be scored as yes/no or present/not present, and those were
> fairly objectively scored. Others, however, were not as objectively
> scored. We agree with one of the previous groups which said that some
> of these are mandatory and others are desirable but not mandatory.
>
> Also, we had a suggestion that we, on the Internet or otherwise,
> publish what the expectations would be in the scoring system so that
> the public would be informed about how we are going to evaluate these
> patient information leaflets on the next go around.
>
> MS. STIFANO: Actually, group six had a hard time separating question
> one from question two since they seem to be so intertwined with each
> other. Basically, we are going to echo what has already been said, and
> we too got specific in saying that the minimum standard would be met
> if the following criteria were adhered to and had the information,
> like the name of the drug, and the benefits of taking it, and who
> should or should not take the drug, drug-drug interactions, drug-food,
> and so on, and also weighted the amount of information that is
> expressed as far as risks. We felt that as far as side effects,
> patients should know that this is not a done deal, that you may or may
> not, and the bottom line is that the information and how it is
> presented should encourage a dialogue between the patient and the
> healthcare provider, and that they just don\'t take it on themselves
> to read this and think this will happen to me or this won\'t happen to
> me.
>
> MS. NORDEN: In group three we had a lot of similar comments. One of
> that that was not brought up so far \-- I will just say a couple of
> them that aren\'t on the overhead \-- was that in order for the
> information to be useful it should be not too long. Somebody said
> limited to one page and printed out at the site. Others said that it
> could be more than one page as long as it wasn\'t excessively lengthy.
> This point, in terms of certain sub-criteria, couldn\'t be met by the
> use of a disclaimer, and the way the information is presented may be
> drug specific but certain contraindications and precautions should be
> communicated in a way that it really draws attention to them. That
> would be formatted in a question like who should not get this drug.
>
> The other point that was brought up in our group, and I am not sure if
> this falls under number one or not but I don\'t have it anywhere else
> so I will bring it up here, was in terms of what happens, and how can
> we evaluate when patients get more than one piece of patient
> information when they get their prescription? For example, there is
> certain FDA-required patient labeling that must be disseminated with
> particular drugs and often patients are getting that labeling at the
> same time they are getting the leaflet that is generated by the
> pharmacy, and a way to look at if that is too much information or
> whether they are consistent with each other in terms of the
> information that is in the two different pieces.
>
> MS. MEASE: I am representing group four, and we had a lively and
> lengthy discussion around this question but never really addressed the
> question at issue here. There were some interesting ideas brought up
> but in trying to address the question here, the group felt that an
> expert panel would be necessary to decide the information that answer
> needed, whereas the consumer should determine whether the information
> is comprehensible.
>
> We did identify and believe that consumer research is needed, and we
> did have one member who provided her input that a level five, as
> defined in the study as greater than or equal to 80 percent adherence
> to the study criteria, would be the minimum threshold.
>
> MS. BUSH: \[Comment away from microphone\]
>
> MS. MEASE: We had a very lengthy discussion on what the definition or
> whether useful should be part of this question.
>
> DR. OSTROVE: That is a lot of information, thank you. Is there anyone
> else who wanted to add from the audience? One of the reasons we wanted
> to have the facilitators do this is that, believe it or not, the
> facilitators have no vested interest in any of this. So, we felt it
> would be best, from a political perspective, to have a nice, neutral
> perspective. But if there are others who feel that something may not
> have been communicated, if you could come up to the microphone?
>
> MS. LEUNG-VEGA: I just want to make the comment on Tom\'s feedback
> from this group regarding the addition of outcomes information and
> compliance. I think that needs to be discussed further before
> something like that is put into the assessment of what information is
> currently out there because I think it will change the whole written
> information piece that will make it a little bit more complicated. We
> are actually in the middle of something right now and probably should
> continue with the initial intent before we add something else to it
> that is controversial. That is the only comment I have to make.
>
> DR. OSTROVE: That is a good point. Thank you very much.
>
> So, moving on to question number 2, should certain criteria derived
> from the action plan recommendations be given more weight than others?
> If so, which should be weighted more strongly, and why? I would like
> to start with group eight, which was facilitated by Tom Abrams and
> Branda Kiliany.
>
> **Question Two**
>
> MR. ABRAMS: The group agreed unanimously that the criteria should be
> weighted, and that is the approach that should be used. The group then
> discussed how we should approach this, and thought it would be
> impossible to rank them since a lot of these things are very
> important. So initially the group decided to discuss making high,
> moderate and low and thought that would be too much distinction, so
> went to high and then low, and then changed it to high and moderate
> because all these things are pretty important. So, that is the
> process.
>
> \[Laughter\]
>
> That took about five minutes. So, we don\'t have as much to discuss on
> this one. That went very quickly but, you know, I think it is
> important to note because what we kind of found was that nothing was
> really insignificant, and I think it is interesting to note that.
>
> So, as the group classified these different criteria into high and
> moderate, they found that almost everything was high. So, what we
> decided to do is to note where it was going to be moderate.
>
> DR. OSTROVE: Is our giggling bothering you?
>
> MR. ABRAMS: I just don\'t know if I am doing something here.
>
> \[Laughter\]
>
> At division meetings I make jokes and nobody laughs, and when I am not
> trying to make a joke people laugh. Anyway \-- no, you can giggle.
>
> So, what we did was say, hey, what are high, what are things that are
> not high. And, we said storage instructions would be considered
> moderate unless the products were storage sensitive, for example, you
> have to store in the refrigerator or there was something special about
> the storage of that product.
>
> The group, except for one member, thought contraindication information
> should be high. The one member who thought it should be moderate said
> that this is really the physician\'s responsibility to communicate,
> the contraindication information and kind of decide if it is a
> contraindication that the patient should not be taking this drug.
>
> The next point was the self-monitoring on drug interaction
> information, and the group was not real clear as far as what this
> meant. They thought if it was for the patient to report back to the
> physician about a possible adverse event, that was very important;
> that should be high. However, if it was just to get data back to FDA,
> which is the physician\'s responsibility to report adverse events and
> to MedWatch, that would be lower. That would be moderate. They thought
> the function of this patient information should be patient specific to
> that patient to help them know what to do in case something happened
> to that particular patient.
>
> Another thing that was a little unclear was up to date and timely
> information. The group specified if this is concerning safety and
> efficacy information, it should be high. They were a little concerned
> about what might be considered new and important information \-- if
> you got so much trivia it is not that important. One of the comments
> was if you put too much information in things get lost.
>
> Lastly, if you look at, I guess, the eighth criteria it includes
> storage instructions and general information. The group wasn\'t really
> sure what that general information was. So, they thought it was
> important to have a number eleven criteria. Number eleven criteria
> would be general information that refers people to get additional
> information. It was recognized that if this is a one-page piece there
> may be patients who may want to get additional information, and it
> should be referenced where they can go for more information that they
> might be seeking.
>
> DR. OSTROVE: Thank you, Tom. Mary, why don\'t we start at your end and
> come back up this time around? Okay?
>
> MS. MEASE: We did actually think that the criteria should be weighted.
> We actually went through and each individual in our group weighted the
> criteria as they thought was appropriate, which was very interesting.
> We saw generally that one and four were ranked the highest and eight
> and ten were ranked the lowest. Someone did offer the opinion that the
> ranking depends on the drug.
>
> We also ran into some differences on the interpretation of the various
> ten criteria. We generally thought that the negatives should be ranked
> higher than the positives, i.e., the contraindications, the adverse
> events versus storage and handling instructions. There were members
> that felt that nine and ten should be given, and then there were a
> couple of people that offered the opinion that if we thought they
> should be given, they should be ranked the highest.
>
> Again, on number seven, specifically we ran into some differences of
> opinion on the interpretation and also thought that it was an
> ambiguous criterion.
>
> MS. NORDEN: Our group thought that, first of all, the criteria should
> be reworked and more closely track the action plan, which is on page
> 21 of your handouts, where there were eleven categories instead of the
> ten, and to follow the report in terms of the ordering and the
> weighting that was recommended in the report.
>
> Also, there was discussion concerning the risks and benefits, and that
> it is really drug specific to determine how much of that there should
> be. Also, this kind of comes back to what was previously said, that
> having information presented is really important and some members felt
> that there was a tendency to try to present a lot of risk information,
> which I think was also brought up yesterday. Patients need to know the
> benefits of the drug as well, but there should be a balance between
> the benefits and the risks, and if there were significant risks
> associated with the drug, they should be presented up front.
>
> As far as the ADR\'s and monitoring adverse events, the group didn\'t
> think it was useful to present a laundry list of all the side effects
> associated with the drug.
>
> MS. STIFANO: We also weighted everything. You know, we placed, like,
> in order of their appearance the criteria, one being identifying the
> drug and its benefits, and the second one being kind of a combination
> of identifying the contraindications and what to do, as well as
> specific precautions and how to avoid harm. So, that would encompass
> who the should not take and any drug-drug interactions or any
> drug-food interactions.
>
> Then, how to take and receive, the third and, again, if it is to be
> taken with food or without food. The fourth would be addressing the
> proper monitoring of adverse events. Again, we felt that a laundry
> list would only be frightening to the patient. So, you would have
> those adverse events with a certain level of significance. For
> example, if the drug carries a black box warning, then most certainly
> you would want to address those concerns. Or, you know, if we had to
> have a numerical value, it would be perhaps greater than five percent
> experience but, again, that would be dependent on the class of the
> drug and the population taking it. So, for example, if the top five
> percent are headache, general malaise and everything else, that could
> be expected patient of the disease itself. That is not necessarily
> useful; that is what they are being treated for. It would be more out
> of the norm, out of the ordinary. So, if they are being treated for a
> headache and their toenails hurt ferociously, then put that
> information in there, not something that they don\'t already know.
>
> The others just kind of fell \-- you should weight them but they fell
> into more of an equal layering below those pieces.
>
> MS. LECHER: We had some overlap with what has already been said but we
> also had some additional items. At this point, I just want to go back
> to question one and say that we had some difficulty in answering
> question one. We didn\'t know whether we were to answer the question
> in terms of the data set that we had been presented with, a data set
> that might be generated using new criteria that we would develop, or
> just in general. That caused some confusion on the part of members of
> our group about specifying what we were really talking about.
>
> Now getting to question two, we did believe that some criteria and
> sub-criteria should be weighted more. The criteria that we thought
> should be weighted more especially included how to avoid risks and
> comprehensibility. It was then suggested that of the criteria on the
> list, criteria one, two, nine and ten could be applied in considering
> all the other criteria.
>
> There was also a discussion, as there has been in the other groups,
> about how to prevent adverse events. That is something else \-- never
> mind, I am reading my notes incorrectly.
>
> We called the points under the criteria sub-criteria and they must
> each meet a predetermined percentage, but this may vary for each of
> the sub-criteria.
>
> On the second page, we say weigh these sub-criteria differently then
> add or delete sub-criteria, and I do believe there were members of our
> group who had suggestions that I hope they will pass along as to what
> additional criteria there should be or what should be changed or
> deleted. Then, that there should be certain standard information in
> all of these. For example, the fact that the leaflet is important and
> you should read it carefully; storage information; sources for
> additional information about the product; and how the information was
> generated, where did this information come from.
>
> Then, on our last overhead we had a suggestion that terminology be in
> lay terms whenever possible. There was some lengthy discussion about
> the term scientifically accurate. Some of the members felt that the
> word scientifically suggests that the language would be scientific and
> perhaps not accessible to lay people. I think there was a consensus
> that when possible terminology should be simple.
>
> MS. WHEELOCK: Our group obviously is the group that doesn\'t agree
> with the rest of the groups here. We decided, unanimously actually,
> not to weight and we had a lot of reasons: that different drugs
> require different weights. That implies a priority. It was felt in the
> group that there would be a subliminal influence if you actually
> prioritized and what would be actually the priority because it may be
> different based upon the different drugs. Different patients require
> maybe different weights, and new information as it is known may alter
> the weights, such as adverse events and interactions. Again, our
> action plan suggestion was that four items are of importance and those
> are the ones to actually follow.
>
> MS. ACKER: Our group had a lot of the same points of view as has
> already been discussed. Most of our criteria came out as being highly
> weighted or heavily weighted. As you can see up there at the top, the
> legibility being very important.
>
> We had a discussion about maybe there is a tension between information
> being scientifically accurate but also concise and readable, and that
> if scientifically accurate means complete, it may be a challenge to
> present that in a concise manner.
>
> The group brought up the point that it looked like from the interim
> results that the criteria on being unbiased in content had already
> been pretty well met, and so maybe that is of a lesser concern and
> that would hold a medium weight because that didn\'t appear to be an
> issue in the interim report. Although they felt it was important that
> information be unbiased but that if that is not something that you are
> seeing in the sample maybe you could weight that lower. That is all we
> have.
>
> DR. SMITH: Our group agreed with Leslie\'s group. We took the easy way
> out and said they should not be weighted. One of the reasons given,
> and probably the most important reason \-- and, again, we separated
> the components from the ones that were more descriptions of how the
> components should be analyzed, and the Keystone report did not weight
> the components, so our group felt that it wasn\'t our place to weight
> the components and, you know, it probably would differ for different
> drugs if you tried to do it.
>
> DR. OSTROVE: Any additions?
>
> \[No response\]
>
> In that case, what I would like to do is actually skip around a little
> bit. Since all of the groups also addressed question six, I would like
> to skip to question six. That is, should the evaluation panel include
> consumers with varying educational backgrounds? If so, how should they
> be involved in the evaluation process? I would like to start with
> group six, Tony Stifano and Spencer Salis \-- you notice there is an
> abundance of women doing this? There need to be more men doing
> facilitation. If you guys would start, that would be great.
>
> **Question Six**
>
> MS. STIFANO: Obviously the answer to this question to our group was a
> resounding yes, that scientists absolutely should not be involved in
> assessing the usefulness or understandability of information that is
> going to the patients. Basically, what we did was to suggest that we
> keep the existing process in place \-- the writing on the slide is a
> little out of order since we really just had so many ideas going
> around, but to keep the existing process in place but then also to
> establish a protocol to evaluate the usefulness of information by
> getting input from consumers, and vendors, and industries and
> pharmacies so that you really could come up with everybody\'s point of
> view on what they consider to be useful, and then put that information
> to use in how you would evaluate it.
>
> One of the other ideas we came up with on getting patients more
> involved in looking at the actual piece of paper that they are handed
> when they get to the pharmacy is to perhaps do, in a non-threatening
> sort of way, a mini-comprehension test after they have had a chance to
> look at the information, and ask them so that you really do get an
> idea of did they comprehend it or not, in a group situation the
> chances of a person just nodding in agreement with everyone else
> because they don\'t want to admit that they didn\'t understand it are
> very high.
>
> One of the other things we suggested is to perhaps conduct focus
> groups to get some answers, and to have these focus groups be
> comprised of consumers, and vendors, and pharmacies and the industry.
> There was a lot of interest on the part of the industry to make sure
> that patients get good information, and to publish the results of the
> outcomes of these focus groups and then ask the question of ourselves,
> well, what do we do that we find that information is, in fact, getting
> in the hands of consumers, and we see that it is; the numbers are
> rising, but if the consumers are getting it and they are not reading
> it or, worse yet, they don\'t understand it, then what good is it?
>
> So, basically, the bottom line that we came up with is that, you know,
> if the information being disseminated is not understood or used, then
> how useful is it? So, we think that the more consumers you can get
> involved, not only from different educational levels and backgrounds
> but different language skills since the two are very different, the
> better off we will all be.
>
> DR. OSTROVE: Thank you. Leslie, we are going to start with you and
> then move this way.
>
> MS. WHEELOCK: We had four major areas about the consumer and my group
> also agreed that the consumer should be involved. In fact, in all the
> questions input of the consumer was very important.
>
> The first thing was involving the consumer just by asking the
> consumer, starting right out and, you know, assessing or asking the
> questions, things such as what is really their method for preferred
> information dissemination? We have been speaking about written but
> perhaps the spoken word, the oral communication, as well as
> considering future technologies where things may be on the web. Also,
> asking them questions about will they take the med? Looking at
> assessing the lifestyle issues, and also what is their initial source
> of information? Is it the written; is it the oral?
>
> The second major area was what types of consumer input would be
> important? Actually, the group decided a consumer would be a very
> valuable asset to the expert panel with input into the design and
> actual evaluation of studies, helping the expert panel develop
> questions and also analyzing those questions, looking at giving
> feedback about format and content, as well as efficacy of oral
> counseling, and also oral information and feedback about the oral
> reinforcement of the written. Also to help evaluation the sample and
> differentiating it \-- the sample form, differentiating it and
> comparing it to the expert panel and what they came up with. Then,
> also looking at evaluating outcomes \-- is there really a return on
> investment with all of this?
>
> We also looked at the consumer in recommending the various
> characteristics about the consumers that would be involved in this
> process, and really wanting a diverse sample, looking at the largest
> user, which is right now the non-college elderly, and also to include
> diverse socioeconomic, cultural, ethnic and educational so that the
> literacy levels, interpretation and comprehension are actually
> addressed.
>
> One of the suggestions was to refer a model, the over-the-counter.
> Studies, again mentioning focus groups, be involved and inclusion in
> those focus groups would be chronic users, first-time users and also
> new users because their feedback maybe is very different. Then,
> looking at the types of consumer, expanding from more than just the
> end-user who is the patient but also to include end-users such as the
> mail- order people and companies that are involved and also the chain
> store pharmacies.
>
> DR. OSTROVE: Thank you, Leslie. Tracy?
>
> MS. ACKER: Our group also undoubtedly thought that consumers should be
> part of the process here, also that the panel include, you know,
> people from various groups; that it be multi-cultural, mixed
> demographics according to age, rural versus urban, etc., different
> educational levels.
>
> The suggestion was brought up that consumer advocacy groups could be a
> source to both spread information but also as a source of volunteers
> to participate in this process.
>
> We also felt that rather than, as was done with the expert panel where
> they had sort of a questionnaire to check off did it meet certain
> criteria, that the consumer actually be asked about comprehension of
> that information and to read the patient leaflet but then be asked
> questions about how do you take this medicine; when should you take
> it, etc. That would be more meaningful and wed speak to many of the
> criteria there in terms of how easy is it for the patients to actually
> use these leaflets.
>
> The suggestion was also brought up that consumers be involved in the
> process through focus groups, also as individual consumers being asked
> to rate these leaflets but also to do a survey among people at the
> point of service at their pharmacy when they get the prescription \--
> to actually get a leaflet but a survey as well to fill out and mail
> back.
>
> MR. ABRAMS: Just to emphasize what Tracy and Tony said, the group
> thought it should go beyond education. Tony mentioned language and
> Tracy mentioned demographics such as age. Just to add to that, the
> group said economics and comprehension levels should really be
> addressed. So, you really want varied backgrounds.
>
> Leslie referred to her group saying that we should look at other
> models, such as OTC models. My group said the same thing, and they
> pointed out to the Australian model and the European model to look at
> how consumers are involved.
>
> Lastly, they stated that panel members should evaluate different
> sections of the patient information.
>
> DR. SMITH: Our group also had a resounding yes, that consumers need to
> be involved and, you know, a random sample which means all cultural,
> all educational levels, and so forth.
>
> We felt that consumers needed to look at all the components but that
> first, you know, the experts would do an evaluation, similar to what
> was done in the study that Bonnie described. That would come first and
> if all of the components weren\'t there, in our estimation it had
> failed already. But, if all of the components were there, then the
> consumers would look at it.
>
> One suggestion was made, that the consumers and the experts actually
> look at it together and have some discussions, which might be very
> interesting, but the consumer would have primary input on is it useful
> or understandable. We talked a lot about whether the best way was
> focus groups or something like a one-on-one interview, something other
> than the study that Bonnie described, the brief study which seemed to
> require a certain education and comprehension level in order to even
> participate very much. So, we were trying to get ways so that people
> with lower education levels could be involved because, you know, we
> feel that there are a lot of people who probably would not necessarily
> be able to even do an evaluation as it was described.
>
> An interesting point was brought up, that maybe we ought to target, if
> possible, consumers with the target disease, whatever the drug is
> indicated for. There were some people who felt that maybe that wasn\'t
> a good idea because those people would have more knowledge than
> someone else but, on the other hand, if you are looking at people that
> are just getting the new prescription for the very first time they
> probably wouldn\'t have that much knowledge and they would obviously
> be more interested in it since they would need the drug than just
> someone off the street. So, we felt it might be interesting to try to
> target them.
>
> MS. MEASE: We did believe that consumers should be involved, and we
> grappled with defining what a consumer is. Somebody offered the
> definition that a consumer is anyone who takes medicine independently
> or gives medicine to others. We thought that was an important point to
> make.
>
> We believe that we should have a large number of consumers on these
> panels, and we generally felt that we should start with people that
> would represent the majority of the population and then, after
> establishing those criteria, we could move on to subpopulations \--
> multi-cultural, age groups and also health status and the blind as
> well. We felt that we should involve consumers in the comprehension
> and testing portions of this, but we did feel, again, that the experts
> should tell us what consumers need to know.
>
> We felt that keeping an eye on technology for the future would be very
> important as that may be the venue in which consumers are going to
> receive more and more information over time.
>
> USP could also be a resource for setting the standards because they
> already do that for product so maybe the could do that for information
> as well.
>
> MS. NORDEN: Our group also strongly believed that the consumers should
> be involved. Our points are almost exactly the same. These were kind
> of a little different concerning involvement of consumer groups and
> consumers in the whole process \-- our group believed that an outside
> group should be the ones that conduct the study; maybe get competitive
> bids; that we should ask Congress for enough money, and to mention the
> IOM and medication errors; and that before the protocol or the
> evaluation process is implemented it should be shared with the
> stakeholders for comment before the implementation process.
>
> MS. LECHER: We have two main issues. They were primarily covered by
> other groups but we have a slightly different slant. We also all
> agreed that consumers should be involved. There was some interest
> among some members that their participation be limited only to issues
> on which they are experts, which is things that would affect informed
> consent decisions, legibility and comprehension.
>
> Also, we should utilize consumers as a valuable resource in conducting
> studies with these labels, such as label comprehension studies or
> cognition studies. We also mentioned actual use studies where they
> would take the product home and see if they followed the directions.
>
> I would also mention that someone in our group did mention Australia
> as another source of information for us. Apparently, they have a
> history of evaluating patient information and perhaps we can learn
> something from that.
>
> DR. OSTROVE: Does anyone in the audience have anything to add to their
> groups? If you could give your name and affiliation, that would be
> great.
>
> MR. FOSSEN: I am Merle Fossen, with the Academy of Managed Care
> Pharmacy. In looking at question six and looking at how consumers
> should be involved in the education group, I would like to suggest
> that maybe there be a pilot study where the study focuses on a return
> on investment in terms of the outcomes, be they economical \-- dollars
> and cents, clinical \-- does the patient get better or worse, or
> humanistically \-- does the patient improve as far as the quality of
> life goes.
>
> MS. HARE: Doris Hare, American Foundation for Maternal and Child
> Health. I don\'t think that consumer involvement should be limited to
> informed consent and that type of thing. There are many consumer
> groups who are extremely knowledgeable on the drugs that their group
> would be likely to take.
>
> MS. DAY: Ruth Day, Duke University. I just wanted to comment on what
> is meant by cognition studies. This group is well aware of
> comprehensibility, the ease with which individuals understand
> information. That is really important. There are other cognitive
> functions that are important that can be tested with these leaflets as
> well \-- the extent to which they perceive and know that the
> information is there; whether they can remember it because, after all,
> in the real world they don\'t always have the leaflet handy; and then
> the most important thing is problem solving \-- can they do what needs
> to be done when circumstances arise. So, the call there for cognition
> studies is for comprehension but more broadly-based cognitive
> processes.
>
> DR. OSTROVE: Thank you, all, for your additions. We appreciate it. I
> would like to move to question five, should there be more detailed
> assessment of factors affecting readability and legibility for
> consumers, for example, type size, style, spacing and contrast? I
> would like to have group four, which was facilitated by Linda Brophy,
> Kathryn Aikin and Mary Mease, take the lead. Mary?
>
> **Question Five**
>
> MS. MEASE: We did believe that there should be a more detailed
> assessment of these factors, and generally the overall format was a
> very important thing to consider.
>
> This is very difficult for me to read. Let me walk up here. There are
> several models that we might be able to use as a resource that were
> presented. Someone mentioned some OTC consumer studies in the Final
> Rule, as well as some FTC proceedings on OTC advertising in the 1970s,
> and maybe we don\'t have to reinvent the wheel here; maybe we can look
> at some other resources that have some well-established processes in
> place.
>
> One person did offer the idea of looking at a recent studied that was
> commissioned by the USP, entitled, \"Optimizing Patient Comprehension
> Through Medicine Information Leaflets,\" which really seems to be very
> applicable here and might be very helpful.
>
> Another important point that we had was we keep hearing about these
> one-page patient information leaflets and maybe we really need to get
> past this mind set. This is an assumption perhaps that has been made,
> and patients really do want more information, and in doing this we
> recognize that the one-page mind set may have come from, say, the
> technology limitations, i.e., the printers or the software, and that
> perhaps this is a systems issue and maybe the software companies need
> to be involved in some of these discussions to iron out this detail.
>
> DR. OSTROVE: Thank you, Mary. Karen, can we start with your group and
> then move in that direction?
>
> MS. LECHER: We had some lively discussion on some of these issues and
> where you see question marks we had especially lively disagreements.
>
> There was a suggestion that the length of the leaflets be limited to
> avoid information overload. However, some other members thought that
> this would not be a good idea. If there were a limitation on length
> there would be clear information about where to get more information
> about the product.
>
> Then, there was a suggestion that there be a summary of the most
> important information up front. Again there was disagreement because
> some thought that this might raise questions of liability or patients
> may not read further into the information if they had a summary up
> front.
>
> There was an interesting suggestion about having gradations of
> information. The pharmacy would be capable of producing, for example,
> three different forms of information that would vary in terms of the
> amount of detail and that consumers could be given samples of each of
> these and then they would choose which type they would want to have
> generated for them for the product they were purchasing.
>
> There was also a suggestion to add some additional factors to the
> analysis, for example, cognitive factors. We had room only to put a
> couple of these examples here. These would include chunking or
> grouping information that is similar and coding, which is providing
> appropriate headings for the information.
>
> There were some other suggestions too. If we have time perhaps those
> could be discussed further today.
>
> MS. STIFANO: Our group came had a lively discussion too about the
> suggestion of having vendor standards, vendor/pharmacy standards,
> since we did find that apparently not only are the vendors providing
> information to pharmacies but pharmacies can, in fact, themselves
> pound out whatever information they wish. So, the suggestion was to
> have industry play an active role in setting these standards, to work
> together with these vendors that are providing the information to
> pharmacies. The specific suggestions included the 10-12 point Aerial
> font, lots of white space, preferably one page but not at the expense
> of skimping on certain information, and we also batted around the idea
> of having printer quality standards for the pharmacies so that you
> don\'t have these God-awful dot matrix printers that are chopping
> things off or are obliterating parts of letters.
>
> Part of the rationale for keeping companies involved is that one of
> the other issues was that the most updated and current information be
> made accessible, and this would be one way of assuring the most
> updated and current information be made accessible.
>
> The only other point that had little to do with this but something
> that might need clarification is it seems there is some confusion
> right now on terminology in terms of med guides versus a PPI versus
> what the printout is called from the pharmacy. So, perhaps we could
> start with something very simple and clarify what all those mean.
>
> MS. NORDEN: In terms of evaluating for readability, our group felt
> that it should be sort of two-pronged, that there should be both a
> consumer evaluation as well as an expert evaluation and the experts
> should be experts in communication and readability.
>
> Basically, they thought that the factors in Appendix G of the action
> plan were very good to follow in terms of laying out criteria for
> fonts and things like that.
>
> As far as the evaluation that the consumer should be doing, they
> thought that they should look at actual leaflets \-- well, also maybe
> at prototypes too but at actual leaflets that are being passed out,
> looking at them for format and readability as well as
> comprehensibility and also usefulness. This came up a lot of times,
> that these are really not that helpful if they are not actually useful
> to the consumer, and ask the questions like what kinds of information
> do you want included, and is the amount of content adequate. Not have
> the consumer necessarily evaluate the scientific content of it, but is
> the amount of information presented adequate.
>
> MR. ABRAMS: Group eight thought readability was sufficient as far as
> assessment of factors. They, however, pointed out that readability
> needs to be more detailed, and pointed out two situations which are
> very related, situational dependency \-- it depends on how complex the
> disease state it is and how complex the drug is, the mechanism of
> action and things, how hard that is to translate. Going along with
> that, they point out it depends who you are communicating to. We
> can\'t just group consumers into one big group.
>
> One group member related an interesting story where they designed an
> HIV brochure which was at a lower educational level, grade level, and
> they thought it was pretty good but, in fact, a couple of consumers
> evaluated it in their focus group and they became really very annoyed
> with it because they felt it was demeaning, like insulting to their
> intelligence. So, it is a hard situation that they pointed out.
>
> Also, just a comment \-- and I don\'t know if the group made this
> remark during this question but Mary mentioned that we shouldn\'t be
> restricted to one page because there might be a lot of information.
> Group eight made a point that if you have one page, no matter how much
> you have, how short or long it is, you really need to always have a
> reference where you can get more information because some people will
> seek more information and it should be made easier for them.
>
> DR. OSTROVE: Did any of the other groups discuss this?
>
> DR. SMITH: I just wanted to make a comment.
>
> DR. OSTROVE: Please.
>
> DR. ACKER: One of the members of our group is from Sweden and he
> shared with us that their country has some experience with evaluating
> the readability and legibility of patient information as well, and
> that may serve sort of as a model. I just wanted to point out to the
> group that Sweden may be a model as well.
>
> DR. OSTROVE: Thank you. Is there anyone in the audience who would like
> to add?
>
> \[No response\]
>
> Moving right along, three groups I believe addressed question seven.
> Question seven is this report collected patient information from
> United States retail pharmacies. Are there ways to expand sampling to
> include mail-order or other non-retail pharmacies? Tracy Acker and
> Mary Kremszner, I would like you to start.
>
> **Question Seven**
>
> MS. ACKER: We had several members in our group who are from the
> non-retail world, from mail-order pharmacies. So, we had some
> interesting discussion around this point. The group felt that, yes,
> some other settings would be important to look at in terms of
> information patients are getting.
>
> Some ways to sample that universe, of course, would be to pose
> patients and mail a prescription to a mail-order pharmacy or send a
> prescription to an Internet pharmacy, through those mechanism. The
> issue came up though for some of these mail-order pharmacies that they
> only fill prescriptions for their clients, for the insured members.
> So, in terms of study methodology that may be a barrier, to have a
> fictitious patient. You wouldn\'t get the prescription filled and
> billed through the insurance company. Some patients who get their
> prescriptions filled in that manner, they pay cash sometimes if it is
> a non-formulary but in terms of using that as a setting for a study,
> that might be a barrier.
>
> The point was brought up that through Internet pharmacies that may be
> easier to use in a study because you could pay cash that way for a
> prescription.
>
> The idea was brought up to select some of the test sites through
> working with their trade organizations for mail-order pharmacies, that
> they might serve as a source for pointing out the sites that work well
> for this study.
>
> Then, the settings we came up with for what might be interesting to
> look at would, of course, include mail-order and Internet but also
> institutional settings, such as outpatient when a patient might be
> started on a new medication in a hospital but then be discharged on
> it. That might be the first time that they are on that medicine and
> that would be a way for them to get the patient information, and we
> should include that. Also, HMO clinics, if the HMO requires the
> patient to fill a prescription at their pharmacy, that would be a
> setting. And, also not to forget the setting of physician dispensing.
> There is a small minority there but we should look at all the places
> where patients might be getting an introduction to a new prescription
> and that would also include physician offices. Also, the idea was
> brought up for the weight loss clinics, for instance, that are
> prescribing diet drugs. That would be a setting as well.
>
> DR. SMITH: Our group also discussed this fairly extensively, and we
> felt that they must be included because they are increasing. More and
> more people are buying their prescriptions, especially over the
> Internet. We thought the Internet part would be relatively easy
> because there are currently only four Internet sites that are approved
> by NABP.
>
> We just had an extensive discussion and nobody in our group had
> ordered a drug from the Internet, and I am curious how many people in
> this room have. Could we have a show of hands? We have a few; great.
>
> The question was, does the information that you get \-- when you send
> in an order and you get an e-mail confirmation of your order, do you
> get the information then or do you get it with the drug, or both or,
> of course, there is always the possibility of neither. But we felt it
> would be good if you would get it right after. Perhaps both was the
> ideal way since it would be so easy here. If you got it with your
> confirmation, maybe you would read it because you still didn\'t have
> the drug but you would have a little bit of time to read the
> information before you got the drug. So, perhaps you might be a little
> more apt to read it. But we did think we would not be sampling from
> these sites, that all of these sites would have someone buy the drugs
> we are particularly interested in.
>
> Managed care \-- we discussed the problems because in most cases you
> have to be a member. We also talked about the acute care setting,
> patients starting drugs in the acute care setting, and we felt that we
> needed to check on what percent of new prescriptions are given for the
> first time within a hospital or other acute care setting, and if it is
> a high enough percentage we would need to include them, or if it is
> not too high the other alternative would be to put the caveat that
> those weren\'t included in the discussion. But we felt that, no matter
> what, the Internet and the managed care needed to be included because
> even though at this stage they may be in the 15 percent range of all
> prescriptions, they are rapidly rising.
>
> MS. MEASE: Like the other two groups that looked at this question, we
> did believe sampling should be expanded to the non-traditional
> pharmacies \-- the Internet, mail-order, managed care. When looking at
> what type of patient information should be included, we went back to
> the Rule 104-180 and thought that the term useful written information
> to 75 percent of individuals the term individuals should be defined
> because we do have children that do take their own medication,
> especially if it is a chronic condition. So, we thought that
> individual should be defined in the Rule.
>
> DR. OSTROVE: Did any of the other groups address this?
>
> MS. NORDEN: We didn\'t specifically address this question but almost
> the exact same kind of points came up when we were talking about
> another issue and, basically, there was a need to sample other venues
> besides just what was sampled in this, and look at more drugs also,
> besides just the three.
>
> DR. OSTROVE: Thanks. Is there anyone from the audience that would like
> to add \-- okay, name and affiliation.
>
> MR. HOEY: I am Doug Hoey, from the National Community Pharmacists
> Association, and we are a trade association for retail pharmacies.
> Just as an added note on what Tracy presented, on using trade
> associations to examine any of the managed care settings, I guess I
> would just like to caution that there is a randomized sample. Just as
> there weren\'t trade associations used to identify retail pharmacies,
> it introduces a certain amount of bias. So, I would recommend that
> trade associations be advised but not used to select the pharmacies.
>
> DR. OSTROVE: Thank you. Actually, none of the groups addressed the
> last question, at least specifically. Did I miss question four? I
> skipped question four. Let\'s go to question four. Question four is,
> should the assessment include additional criteria or types of in and,
> if so, what? I would like to have Leslie Wheelock and Dorothy Ballmann
> start with that.
>
> **Question Four**
>
> MS. WHEELOCK: Our group looked at two major areas, the study
> considerations and evaluation of the leaflet. With regard to the study
> evaluations, a larger sample size was actually discussed, and large
> meaning not only in size but also in its diversity as well. Also,
> asking some questions to assess are there some differences with drug
> classifications, and also looking at patient comprehension, and some
> more detailed criteria evaluation, and also how patients are using and
> interpreting the data they are getting.
>
> With regard to the evaluation of the leaflets, were there any pharmacy
> modifications, comparing the FDA\'s approved patient package insert
> with what is distributed by the vendors; also looking at who and how
> the leaflets are developed; and also does the information in the
> leaflets provide sufficient information. So, that is what our group
> came up with.
>
> DR. SMITH: We basically just had one comment but I think it was
> important, and this was that the Keystone criteria makes the
> distinction that bolded boxed warnings need to go up front but there
> was a strong feeling that all bolded warnings, anything that had to be
> bolded as a warning, really needed to go up front and get more play
> right from the beginning.
>
> DR. OSTROVE: Did any of the other groups have additions to this kind
> of area, or any comments from the audience? Okay.
>
> MS. DARCY: My name is Mary Lynn Darcy and I am with Micromedex, a
> vendor. On the last slide that was just put up there, there was a
> comment on the boldness and the black box warnings, which we didn\'t
> specifically address as an issue in our group but several people, as
> the discussion went around, did discuss this and maybe there were
> about four of us in the group who felt that this was the one thing we
> differed on with Keystone, and some of that was due to the USP
> research that had previously been done that talked about chunking
> data. Of course, we could have more research, however, prior research
> does support that when you separate out the black box warnings section
> you then have warnings in two different areas. You no longer have
> chunked the data. We felt that wherever the precautions section would
> be, that is where the black box warnings should be. They should be
> incorporated in that area.
>
> DR. OSTROVE: Thank you.
>
> MS. COHEN: My name is Susan Cohen, and I think that in conjunction
> with this we should encourage, particularly HMO\'s where the
> physicians are allowed ten cents \-- ten minutes \-- that is
> interesting, and that was subliminal \--
>
> \[Laughter\]
>
> \-- that they have someone who can assist the patients as they come
> through the office. After all, this will be as good as a physician in
> the long run and people need the physician. I would like to encourage
> industry to have some enlightened self-interest in doing some
> interesting things in terms of labeling. As I understood, the label
> that I would have loved to have seen on bottles is not possible but if
> you use the black box on a piece of paper why can\'t you use something
> on a bottle too?
>
> I think whatever program is developed, it should be reevaluated in a
> certain number of years to see if it is really working, and go back to
> the consumers \-- that is what it is about \-- and find out if it is
> really working for them, or are there things that they really would
> like to have.
>
> I am delighted that you have had this, and I hope that we can go
> forward and we can all get our medicine and understand what we are
> getting.
>
> DR. OSTROVE: I think we all agree with you on that. As I was starting
> to say before, none of the groups formally addressed question three so
> we are going to wait and see what comes in with the written comments
> for that. I know that group three didn\'t get to be the lead on any of
> these, which is Janet Norden and Melissa Moncavage. So, I just want to
> acknowledge that they are here, and Karen Lecher and Rebecca Redman
> also did not get to be the lead \-- not that they were unhappy about
> it, but I want to acknowledge their presence here and their
> contributions.
>
> In fact, Karen sent me a note and said that their group had some
> additional comments that didn\'t seem to fit clearly into any of the
> questions addressed formally. So, I figure we have a little bit of
> time, Karen, if you would like to present those as a part of your
> group\'s process, I think this is a good time to do it.
>
> MS. LECHER: We didn\'t put these on the overheads because we didn\'t
> know if they would be appropriate for presentation. The first few that
> I am going to give you are primarily about the content of the
> information that members of our group suggested be in the leaflets
> and, therefore, they would then be reflected in the evaluation
> criteria.
>
> There was a suggestion that there be more balance of the benefits and
> the risks; that current leaflets often emphasize the risk information
> and don\'t give enough information about the benefits so that patients
> might be scared off from taking it or may not understand the
> implications of they don\'t take it appropriately.
>
> There was also a suggestion that the leaflets tell when the product
> was approved to give the consumer a sense of how long it has been in
> use. Related to that is information about the number of patients who
> have taken the medication.
>
> There was a suggestion that giving percentages of adverse events may
> be misleading to consumers. There may be under-reporting of adverse
> events, or they may not accurately reflect the rate of adverse events.
> There was not consensus on any of these items that I am telling you
> about, this one in particular.
>
> There was a suggestion to group the adverse events as more common,
> less common and rare, and define what those terms mean, perhaps giving
> numerical ranges rather than giving percentages associated with each
> of the adverse events. And, also to tell whether the adverse events
> are being reported from clinical trials or from postmarketing
> experience.
>
> There was one other suggestion that doesn\'t deal with the content of
> the leaflet. If there are low ratings, and one of our members has
> looked at each of the ten criteria to see which ones seem to be rated
> higher and which are lower \-- if there are lower ratings, those
> particular criteria should be examined in more detail, perhaps looking
> at the inter-rater reliability and the sub-criteria that were used,
> and perhaps those could be worked on before the next study.
>
> Finally, ideally each of the leaflets would be individualized to the
> condition for which the medication is being dispensed so that
> information that may not be applicable for that use would not be on
> the leaflet. Thank you.
>
> DR. OSTROVE: I said that my brain was already full yesterday \-- yes,
> Dr. Svarstad?
>
> DR. SVARSTAD: May I make one comment?
>
> DR. OSTROVE: If you could come up to the mike and give your name and
> affiliation, yes.
>
> DR. SVARSTAD: Bonnie Svarstad, University of Wisconsin. Several people
> made a comment so I want to raise a question here and get your
> feedback on this because I struggled with how to answer it. One
> suggestion that was made a couple of times was that the actual
> evaluation form with the sub-criteria be circulated in advance for
> comment. I can see all kinds of good reasons for that \-- to give the
> public, vendors, pharmacy, etc. a chance to have input into that. I
> wonder if those people that made that suggestion have thought of the
> downside of that?
>
> The downside of that is one that a university, a high school or a
> teacher often faces when exams start to circulate. The one possibility
> here, for example, is that you do circulate the ibuprofen, the
> amoxicillin and the paroxetine evaluation form and then whoever is
> responsible for that information makes sure that those three were done
> well but that all of the other medications were not attended to. Does
> anybody have \-- do you get my concern?
>
> My concern is that, on the one hand, we want feedback, as much
> feedback as possible \-- we, meaning whoever is responsible for
> evaluating. On the other hand, the concern is that by circulating a
> particular evaluation form for a particular medication may lead to
> some attempt to improve only those sheets rather than the sheets that
> are necessary for all drugs. And, I don\'t know the answer to that.
>
> DR. SMITH: I would to address that. To some degree, those will
> probably get the most change, but I also think a lot of groups may use
> them as samples. So, it could be useful. I also think, and correct me
> if I am wrong, Nancy, that they will not be the same drugs in the
> next. So, these three drugs, I guess, are the only ones that will not
> be included in the next go around because they have been done. So, if
> anyone is trying to correct a few labels in order to maybe score
> higher next time, those are the absolute worst ones to correct right
> now.
>
> DR. OSTROVE: And, I think it is possible that Dr. Svarstad\'s point,
> in regard to some of the comments in the groups, is that before we use
> it for the next time that they be sent around.
>
> DR. SVARSTAD: What I thought I heard was several suggestions to
> circulate the evaluation forms \[comments away from the microphone\].
>
> DR. OSTROVE: To repeat what Dr. Svarstad said, what she had heard in
> several of the groups was that, in fact, the test forms that will be
> used in the next evaluation be circulated, and that may have been a
> mis-perception on her part but that is what she was responding to with
> regard to this particular comment.
>
> DR. SMITH: I think certainly if the criteria are somewhat different,
> then it would be good that those be circulated but not the specific
> details for the specific drugs probably.
>
> DR. OSTROVE: Again, I think it is important for everyone to understand
> that we, today, have heard a wide range of suggestions, and I am not
> going to make this suggestion because actually legally we are not
> allowed to do this but if I wasn\'t constrained I would say go lobby
> your congress person for money because a lot of, you know, what we are
> hearing today would constitute a fairly expensive undertaking. I mean,
> we will have funds but I don\'t know if we will have those kinds of
> funds.
>
> So, we are going to have to sift through all the information that we
> have heard today and all the information that is going to be submitted
> to the docket within the next couple of months and decide what we want
> to focus on. Where obviously there seems to be some consensus, clearly
> we need to go in that direction. In the many areas where there isn\'t
> complete consensus, then we have to make some decisions as to what the
> right thing is to do given the constraints that we are working under
> in terms of resources and funding. So, I think, you know, that is
> something for everyone to take back. It is invaluable to have had you
> all come today and to give us your feedback, and it will all be taken
> into account.
>
> I think that we also need to consider the whole issue of continuing,
> even before we do the next evaluation, getting some more feedback and
> if we make any drastic changes to what we have here, certainly getting
> back to the Keystone participants and the people who are involved in
> the process and letting them know what those changes are. Of course,
> the biggest question comes down to, well, what I think is a
> substantive change may not be what someone else thinks is a
> substantive change but we can deal with that. We have dealt with those
> things before.
>
> None of that undermines the importance of what we have done here today
> and the importance of your submitting your comments because we will be
> considering all of them. I can\'t stress the importance \-- I feel
> like I am repeating myself but I think it is okay to sound like a
> broken record in this particular instance because I am really being
> sincere about it, and really do want to hear from you. Yes, ma\'am?
>
> MS. BUSH: Pat Bush, USP. I have a question. It must be that you
> considered not buying the drugs. That, of course, would make the study
> vastly less expensive. So, why did you turn down doing exit surveys or
> observational survey type work, which has been done in the past, I
> gather, rather successfully?
>
> DR. OSTROVE: If you go to a pharmacy and simply say give me what you
> are going to give \-- you mean exit surveys \--
>
> MS. BUSH: Yes, you stand and you watch what happens a patient and \--
>
> DR. OSTROVE: And you ask them to give you the piece of information
> that they have gotten?
>
> MS. BUSH: And, then when they exit you ask them, yes, and then you
> look at it and talk to them but you see do they get something.
>
> DR. OSTROVE: You are going to have a huge range of drugs to do
> something like that.
>
> MS. BUSH: Yes, but you ought to have general criteria.
>
> DR. OSTROVE: But the general criteria need the drug-specific
> sub-criteria in order to be assessed. So, you would have to have
> drug-specific sub-criteria for who knows how many products. We did
> think about other ways of doing this, and we really felt that from a
> methodological perspective that this was the best possible. If you
> have other ideas, please send them in.
>
> MS. BUSH: Bonnie says the Hawthorne effect is so big there.
>
> DR. SVARSTAD: \[Comment away from microphone\].
>
> DR. OSTROVE: This effect arose from some very, very old studies that
> show that when people know that they are being observed, that alone
> changes their behavior.
>
> MS. BUSH: We all know that but there have been a lot of this kind of
> studies. Are they all then terribly biased? They meet the people
> outside after they leave the store, or they go to their houses, or
> other ways.
>
> DR. OSTROVE: I hear what you are saying and, as I said, we are
> listening and that is why we are here.
>
> MS. BUSH: You are right, the pharmacists would probably pretty quickly
> catch on when the patient says, guess what happened to me after I left
> last time. So, okay.
>
> DR. OSTROVE: Thanks.
>
> MS. RUCKER: Lee Rucker, National Council on Patient Information.
> Having participated in the action Keystone committee a few years ago,
> what is exciting about today, revisiting this, is that I think we have
> a real opportunity with the timing for two factors. One is the fact
> that just a few months ago we had the Institute of Medicine report on
> medical errors, which I think did a great deal for raising the public
> consciousness about the problem and solutions. So, I think we can
> capitalize on perhaps that raised public consciousness for the short
> term.
>
> The other action that is coming down the pike, and every day it seems
> to be more imminent, is the expansion of an outpatient Medicare drug
> benefit, which certainly affects the people who use the medicines the
> most. However, opening the access floodgates could bring with it, of
> course, more problems.
>
> DR. OSTROVE: Sure.
>
> MS. RUCKER: And, therefore, more practicality and appropriateness and
> importance of our effort here over the past two days, and down the
> road in terms of research. So, I think that for each of our
> organizations that we are representing we need to embrace this
> opportunity and see it as an obligation to communicate to our
> respective stakeholders what has transpired here, where we are going,
> and how we can all do our part.
>
> DR. OSTROVE: Work together. Absolutely. Absolutely. I second that.
>
> MS. COHEN: Here comes a grade school question. If you are going to
> spend money on all these wonderful suggestions, why does it have to be
> drug specific? That is what I don\'t understand. It is expensive to do
> these things, and I think every bit of information would be valuable
> to consumers, of course, depending on the medication and how you
> tailor it to the medication. So, I am a little puzzled why you have to
> do specifically and just not do it in terms of what is best for
> everybody and that gets the myriad of medications that are available
> to consumers. I have missed something here. Help me; I really need it.
>
> DR. OSTROVE: Do you have an answer to that in the back? Sandy, could
> you come to the mike?
>
> MS. ESKIN: Susan, in answer to your question, the whole plan is an
> attempt to improve the actual information that consumers get. What
> Congress asked is that the Secretary has to evaluate compliance with
> this voluntary plan. She has to assess what is out there in the
> marketplace. So, obviously there can be differences of opinions on how
> many drugs should be looked at and how many leaflets should be looked
> at, but there needs to be evaluation of actually what is out there,
> otherwise you are not doing what is being asked.
>
> MS. COHEN: \[Comment away from microphone\].
>
> MS. ESKIN: Well, of course, but there has to be some way. There was a
> methodology applied in this study and, hopefully, all of us who can
> comment on the specifics of the methodology can, hopefully, improve
> the methodology of the next set. You can\'t look at every single
> pamphlet; you can\'t look at every single drug. So, you sit down
> perhaps and look at the whole marketplace and look at perhaps the most
> widely prescribed drugs, or you take a whole range of drugs \-- one
> that has a black box warning, one \-- these are all possibilities. You
> may wind up at the same place you are right now, which is three drugs.
> We don\'t know but I think you clearly have to do real pamphlets
> involving real drugs. I don\'t think you need to buy them. That is
> another issue to discuss. Maybe I am thinking something that is
> obviously faulty but what if you asked every pharmacy to submit
> samples of what they actually put out?
>
> DR. OSTROVE: I think that this is a discussion that could go on for
> half an hour, hour easily, and we only have a few minutes left. I
> would suggest that any of you who have suggestions concerning the
> methodology, please submit them. Believe it or not, we actually have
> reasons for why we do things and so we can respond, and we can give
> you the reasons for why we make a certain decision. But if you have a
> good rationale and if it is something that we haven\'t thought about,
> we are absolutely happy to consider it and see whether that would work
> better. We are not plugged in to any particular way of doing anything.
> Nothing is set in concrete. So, please keep that in mind but I don\'t
> think that this is really the place for a discussion about methodology
> right now. So, with that, Ruth, if you are not going to talk about
> methodology.
>
> MS. DAY: A very brief and important thing, there is a small point that
> may cause a lot of confusion, and that is, we have been using
> criterion numbers and I wanted to point out that the criterion numbers
> you may have in your notes from yesterday are from Dr. Svarstad\'s
> report where, for example, number nine is scientific accuracy. In all
> the criteria numbers that have been on the screen today, they have
> been based on the handout that we got here where scientific accuracy,
> for example, is number one. So, if you have in your notes criterion
> number something or other, make sure you keep track of whether it was
> the first day from Dr. Svarstad\'s report or from today from the
> handout so that you know but, also, if you submit something to the
> docket don\'t use the criterion numbers or at least identify that they
> are from the report. Use their names rather than the numbers.
>
> DR. OSTROVE: That is a very good point, and I really appreciate that.
> Excellent. Thank you.
>
> Well, I think we are at a good point, at this point, to break. Again,
> I want to thank all the people who have worked so hard in putting this
> together, and especially Ellen Tabak and Marcia Trenter, and the
> Office of Training and Communications, all the people from the Office
> of Training and Communications and from the Division of Drug
> Marketing, Advertising and Communications, and all of the rest of you
> who came \-- Dr. Svarstad, who flew all the way in from Wisconsin.
> Thank you, all, and submit your comments, please. Thank you very much
> for coming.
>
> \[Whereupon, at 3:00 p.m. the proceedings were adjourned.\]

\- - -