UNITED STATES OF AMERICA

DEPARTMENT OF HEALTH AND HUMAN SERVICES

FOOD AND DRUG ADMINISTRATION

\+ + + + +

RISK MANAGEMENT PUBLIC WORKSHOP

PHARMACOVIGILANCE PRACTICES AND

PHARMACOEPIDEIOLOGIC ASSESSMENT

\+ + + + +

FRIDAY,

APRIL 11, 2003

\+ + + + +

The Workshop was called to order at 8:07 a.m., in the NTSB Board Room
and Conference Center, at 429 L\'Enfant Plaza, S.W., Washington, D.C.,
by Steven Galson, Deputy Director, CDER, presiding.

PRESENT:

DR. STEVEN GALSON Deputy Director, CDER

DR. MARK MCCLELLAN Commissioner FDA

DR. SUSAN ELLENBERG CBER

DR. JULIE BEITZ CDER

DR. MILES BRAUN CBER

DR. MIN CHEN CDER

MS. AILEEN CIAMPA CDER

DR. EDWARD COX CDER

PATRICIA GUINN CDER

DR. VICTOR RACZKOWSKI CDER

DR. PAUL SELIGMAN CDER

DR. JUDY STAFFA CDER

DR. JOYCE WEAVER CDER

DR. CAROL HOLOQUIST CDER

PUBLIC COMMENT:

DR. GERALD FAICH Pharmaceutical Safety

Assessments

DR. JOHN FERGUSON Biotechnology Industry

Organization

GREGORY GOGATES CRF Box, Inc.

DR. STEPHEN GOLDMAN Stephen Goldman Consulting

Services, LLC

LINDA HOSTELLEY Merck, PhRMA

DR. SIDNEY KAHN Pharmacovigilance & Risk

Management, Inc.

DR. ROBERT C. NELSON RCN Associates

DR. ANNETTE STEMHAGEN ISPE Membership Committee

DR WENDY STEPHENSON PhRMA

DR. ANSHU VASHISHTHA Watson Pharmaceuticals

C-O-N-T-E-N-T-S

Welcome and Introductions 4

Session I - Overview and Good Pharmacovigilence

Practices

FDA Presentation by Dr. Julie Beitz 8

FDA Presentation by Dr. Min Chen 14

Oral Presentations - Public Comment

Dr. Robert C. Nelson 21

Dr. Stephen Goldman 36

Dr. Gerald Faich 53

FDA Presentation by Dr. Mark McClellan 67

Oral Presentations - Public Comment

Dr. Anshu Vashishtha 92

Mr. Gregory Gogates 102

Ms. Linda Hostelley 111

Panel Discussion 120

Session II - Pharmacoepidemiologic Assessment

FDA Presentation by Dr. Judy Staffa 157

Oral Presentations - Public Comment

Dr. Annette Stemhagen 172

Mr. John Ferguson 186

Dr. Wendy Stephenson 193

Dr. Sidney Kahn 201

Panel Discussion and Q&As 212

Closing Remarks by Dr. Julie Beitz 256

Adjournment 258

P-R-O-C-E-E-D-I-N-G-S

(8:07 a.m.)

DR. GALSON: Welcome. I hope to have a good group today, and I am sure
that we will have more people filtering in as time goes on. I wanted to
welcome you to the third day of our Risk Management Public Workshop, and
what I am going to do first is just ask the FDA staff who are here to
quickly introduce themselves.

And first up at the table here, if you folks would just sequentially
push your buttons and say who you are and what you do.

DR. MILES BRAUN: My name is Miles Braun, and I am the Director of the
Division of Epidemiology in the Center for Biologics at FDA.

DR. SELIGMAN: Good morning. I am Paul Seligman, and I am the Director of
the Office of Pharmacoepidemiology and Statistical Science in the Center
for Drugs at the FDA.

DR. BEITZ: I am Julie Beitz, and I am the Deputy of the Office of Drug
Evaluation III.

DR. COX: And I am Ed Cox, Deputy Director, Office of Drug Evaluation IV,
CDER, FDA.

DR. GALSON: Okay. And then we have got at the table at my left is the
working group that focused on this concept paper that we are discussing
today, and if we could just go down the table.

DR. WEAVER: Joyce Weaver, Office of Drug Safety

DR. HOLQUIST: Carol Holquist, Office of Drug Safety.

DR. CHEN: Min Chen, Office of Drug Safety.

DR. STAFFA: Judy Staffa, Office of Drug Safety.

DR. GUINN: Patrick Guinn, Office of Drug Safety.

DR. GALSON: And that is Lee Lemley, and thank you, Lee, for doing all
the hard work to organize this 3 day meeting. All right. First, I want
to thank the members of the CDER/CBER Executive Oversight Committee,
none of whom are sitting here, but I think we will have some a little
bit later.

Going quickly over this, particularly for those of you who may be here
for the first day today, we have had \-- this is our third day of
meetings to discuss three concept papers that corresponded to three
groups working in this area, and today\'s meeting is focused on
pharmacovigilance.

MS. LEMLEY: We will be with you in a minute.

(Brief Pause.)

DR. GALSON: So, let me go over the schedule for today. We are in the
introductions now, and then we are going to be split up into two general
areas today. The first session this morning is overview and good
pharmacovigilance practices.

We are going to have a number of FDA presentations, and then oral
presentations from the folks sitting at the table in front. We will then
have a break at 9:45, and Dr. Mark McClellan, the Commissioner of the
FDA, will be here at \-- we should have some time for him to ask and to
respond to questions from the audience.

Then we will continue with oral presentations and have a discussion
among the people who spoke, and questions and answers from the audience.
At 11:30, we will have lunch, and then we will start on the second
session, pharmacoepidemiologic assessment, and FDA\'s presentation, and
then presentations from speakers, and then questions and answers from
the audience, and then some closing remarks at the end.

Some logistics. We are going to distribute cards. If you don\'t want to
go up to the microphone and you just want to write down your questions
on the cards, hand them to the people who will be in the aisles, and we
will bring them up and we will open mikes.

The transcripts from these sessions will be available within 30 days on
the website you see, and as well, we have got a general website for the
meeting, and we are going to try to post all of the slide presentations
that you are seeing today and for the last few days on that website
address. So look for that if you are interested in copies of the slides.

No food or drinks are allowed in here. The restroom is out in the lobby,
and please note the fire exits from the room. Thank you very much.
Before we move on with the program, I just want to have the speakers for
this morning start, and those of you that are sitting at the table, just
introduce who you are, and we will get around to each of you. Just real
quickly on the mike.

DR. NELSON: My name is Bob Nelson, and I am a consultant in Drug Safety
and Regulatory Affairs.

DR. FAICH: I am Gerry Faich, and I am a second consultant in Safety and
Regulatory Affairs.

DR. GOLDMAN: Steve Goldman, and I am the third consultant in Safety and
Regulatory Affairs.

DR. GALSON: Okay. No shortage of consultants.

DR. NELSON: The order means only administrative order.

DR. GALSON: Okay. I am now going to turn the mike over to Dr. Julie
Beitz, who has done a great job managing the work involved in creating
this third concept paper. Julie.

DR. BEITZ: Good morning, and welcome to day three of the CDER/CBER Risk
Management Public Workshop. Today, we will be discussing the concept
paper entitled, \"Risk Assessment of Observational Data.\"

This paper was written in a general way to stimulate discussion at this
workshop regarding what constitutes good pharmacovigilance practices and
good pharmacoepidemiologic assessment.

Comments that we receive today and in the docket will be considered as
we prepare a longer and more comprehensive draft guidance document in
the coming weeks and months.

The CDER/CBER pharmacovigilance working group represents a broad
spectrum of talented individuals at FDA who came together to produce a
concept paper in very short order. Drs. Mark Goldberger and Miles Braun,
served as group leads for CDER\'s Office of New Drugs, and for CBER.

Our support staff, notably Aileen Ciampa, provided regulatory input; and
our project manager, Patrick Guinn, helped with scheduling meetings and
organizing today\'s workshop. Many group members are in attendance
today, and you will have an opportunity to interact with them.

Dr. Ed Cox will be filling in for Dr. Goldberger, who could not be with
us today. You have already seen the agenda presented, and I will just
point out that we are currently in the overview for the morning session.

The concept paper under discussion today encompasses the following:
Important pharmacovigilance concepts; safety signal identification;
pharmacoepidemiologic assessment and interpretation of safety signals,
and a development of pharmacovigilance plans.

Today\'s discussion will focus on risk assessment based on observational
data sources, including case reports, case series, pharmacoepidemiologic
studies, registries, and surveys.

Risk assessment, in the context of clinical trials, was discussed on the
first day of this workshop. First, let me begin by defining the concept
of pharmacovigilance. As we have stated in the concept paper,
pharmacovigilance includes all post-approval, scientific, and data
gathering activities relating to the detection, assessment,
understanding, and prevention of adverse effects, or other product
related problems.

This includes the use of pharmacoepidemiologic studies. Risk assessment
occurs throughout a product\'s life cycle. At the time of approval,
available clinical trial data involve limited numbers of patients
treated for relatively short periods of time.

In drug approval, large numbers of patients may be exposed to a product,
including patients with co-morbid illnesses, patients using a variety of
concomitant medications, and patients using the product chronically.

After approval, new safety information may become available from a
variety of sources through use of the product either domestically or in
other countries, through use of other drugs in the same class, from
preclinical or pharmacologic studies of the product, or from controlled
clinical trials.

Many definitions have been put forth to characterize a safety signal.
For the purpose of today\'s discussion, we have defined a signal as an
apparent excess of adverse events associated with the use of a product.

Small numbers of well-documented case reports, even a single case
report, may be viewed as a signal. In addition, we believe that
pre-clinical findings or experience with other products in the same
class, may be sufficient to generate a safety signal even in the absence
of case reports on patients.

Thus, a products risk profile may be characterized by several safety
signals. Throughout the concept paper, we have tried to clarify the
following activities as they relate to safety signals.

We say that signals are identified from case reports or other sources.
That signals are evaluated in pharmacoepidemiologic studies, registries,
or surveys. That signals are interpreted in the context of all available
safety information, and that signals are monitored through enhanced
pharmacovigilance efforts.

This paper also introduces the concept of a pharmacovigilance plan. This
would be a plan submitted by a sponsor for the ongoing evaluation of
safety signals identified by the use of a product.

The plan could also include monitoring for at risk populations which
have not been adequately studied to date. The plan may be developed at
the time of product launch, or after a signal is identified.

For a product without safety signals identified pre-or-post approval,
and for which at-risk populations have been adequately studied, routine
post-marketing reporting may suffice as a pharmacovigilance plan.

For a product with safety signals identified pre-or-post approval, or
for which at risk populations have not been adequately studied, a
collection of additional safety information beyond routine
post-marketing reporting may be desired.

Examples of the kinds of activities that could constitute a sponsored
pharmacovigilance plan include expedited reporting of serious adverse
events of interest, submission of adverse event report summaries at more
frequent pre-specified intervals.

For example, on a quarterly basis rather than annually; conduct
additional observational studies or control trials; implementation of
active surveillance activities to identify as yet unreported adverse
events.

We recognize that many of these activities are already being carried out
by sponsors, even though the term, pharmacovigilance plan, has not yet
been applied to them.

Active surveillance efforts may involve data gathering activities that
are focused on specific products of interest on specific health care
settings, such as emergency departments, or on specific events that are
often considered to be drug related.

FDA looks forward to hearing public comment on active surveillance
strategies using electronic health information systems or other
databases. In particular, we would like to hear your thoughts regarding
the circumstances for which these efforts would prove most useful.

We recognize that emerging new safety data may result in ongoing
revisions to the sponsor\'s pharmacovigilance plan for a product.

While additional safety information is being generated, the FDA will
work with sponsors to communicate information about safety signals, and
minimize events occurring in the users of a product through risk
management programs as appropriate.

This concludes my presentation. We will now move on to the next FDA
presentation on Good Pharmacovigilance Practices.

DR. GALSON: Thank you, Julie, and our next speaker is Min Chen, who is
the Associate Director of the Division of Drug Risk Evaluation in our
Office of Drug Safety.

DR. CHEN: Good morning. I am here to talk about Good Pharmacovigilance
Practices and our current thinking at the FDA. As Dr. Beitz mentioned,
the pharmacovigilance definition is generally regarded as all
post-approval scientific and data gathering activities relating to the
detection, assessment, understanding, and prevention of adverse events,
or any other product related problems.

This includes the use of pharmacoepidemiological studies, too. We think
that good pharmacovigilance practice starts by acquiring complete data
from spontaneous adverse events or reports. It is critical for sponsors
to actively seek information on an adverse event by direct contact with
the initial reporter so that a thorough assessment of the event can be
made expeditiously.

Good reporting practices are available in several guidances that we
have. We are looking for the following information in a good case
report, whether reported spontaneously or published in the medical
literature.

The adverse event details, description, patient\'s baseline
characteristics, and concomitant drug therapy details, time to onset of
the signs or symptoms, diagnosis of the event which hopefully can be
supported by the lab data stated here, and the clinical course of the
event, and the outcome; not just regulatory outcomes, but also the
patient outcome in the end of this event.

Any other relevant information can help us in a good case report. For
reports of medication errors, a good case report would also include a
full description of the following; the products involved, the sequence
of events leading to the medication error and the work environment in
which the error occurred.

Also, types of personnel involved with the error. We encourage sponsors
to include in the case narrative all of the data elements outlined in
the NCC MERP taxonomy. That is National Coordinating Council for
Medication Error Reporting and Prevention.

When case reports are identified in adverse events where there are bases
that associate a similar event with a product, a case series could be
developed. In FDA\'s experiences, identification of all clinical
relevant cases depends on thorough search strategies based on measured
terminology.

Generally, case definitions would be developed to provide consistent
characterization of the adverse event of interest, and to facilitate the
retrieval of all clinically relevant cases from the database. In
addition, data mining techniques may be applied to the databases to
identify some cases of interest.

For any individual case report the FDA has found that without complete
details, it is rarely possible to know with absolute certainty whether
it was product induced.

However, a number of features as follows are generally recognized as
supportive of an association between a product and an event. If the
event occurs in the expected time frame \-- for example, allergic
reactions occurred within a few hours, or a few days

\-- most likely cancer will develop after long term years of therapy.

If there are no symptoms related to the event prior to exposure, and if
there were no other use of a concomitant drug that could contribute to
the event or co-morbid conditions. Of course we are looking for the
positive challenge, or positive rechallenge scenarios if we have them.

Also, the event is consistent with the established mechanism of action
of the product. However, we always keep that in mind, that idiosyncratic
reactions that are not necessarily consistent with the mechanism would
be considered in the assessment.

For medication error related events, the FDA would expect the sponsor to
evaluate each event to identify the root cause of factors that led to
the actual error, or even possible error, the NCC MERP that taxonomy
that was mentioned before would be used.

When appropriate the case reports may be grouped into probable,
possible, or unlikely for our assessment purpose. In general, a safety
signal may be described as an apparent access of adverse event
associated with the products used.

Occasionally, however, even a single well-documented case report,
particularly if the report describes the positive rechallenge, may be
viewed as a potential signal. In addition, technical findings with a
product, or experiences with other similar products in the class may be
sufficient to generate the hypothesis for a safety signal.

Safety signals may be further assessed in terms of their magnitude,
population at risk, changes in risk over time, biological possibilities,
or other factors. A product\'s risk profile may be characterized by
several safety signals.

Data mining can be used as a signaling tool by using the Bayesian
methodology, and data mining uses raw case report data, and all drug
adverse event combinations. It calculates the expected number of events
for all drugs, and then compares each drug\'s observed to expected
count.

There are many methods that may be used to generate the relative signal
scores observed and expected count. This may be a useful adjunct to our
routine safety signaling methods.

However, our current thinking is that further exploration and validation
is still needed to use as a tool. Safety signals may be able to tell us
any new unlabeled adverse event or observe an increase in the severity
or specificity of a labeled event, and an increase in the frequency of a
labeled event, and new interactions, from drug interaction, drug food
interaction, or drug dietary supplement interactions.

Of course, where there is a confusion with the product\'s name, package,
or use, actual or potential, then that is a signal to us. After
detecting the potential safety signals from the case series, this
descriptive analysis of the cases are very useful in characterizing the
adverse event and the population at risk.

These analyses should include demographics, including age, gender, or
race information, the effect of dose exposure duration, is it short or
long term duration exposure, and the dose effect. If there is any
concomitant medication, then what is the potential interactions between
the drug and the suspect, and the rest of the event.

And if there is a co-morbid condition of the patient, such as baseline
hepatic or renal impairment, in the risk event. We are looking for a lot
to lot differences in product formulation, and the risk of the event,
and if there is a potential for an access adverse event given the
disease being treated, such as in an advanced cancer condition or
immunocompromised patients that we have to consider all these conditions
in the analysis.

Finally, we would like to do some estimates of the magnitude of risk or
differences from the known background rates of the events that are
interesting. All these risk assessments of post-marketing data will be
considered if further pharmacoepidemiologic studies may be recommended
to further characterize the risk.

Dr. Judy Staffa will present this afternoon the various aspects of
pharmacoepidemiologic studies. The questions we are asking for public
comments for these sections of the guidelines for pharmacovigilance
practices are how can the quality of spontaneous reported case reports
be improved.

What are the possible advantages or disadvantages of applying data
mining techniques to spontaneous reports databases for the purpose of
identifying safety signals.

Finally, what are possible advantages or disadvantages of performing
causality assessment at the individual case level, or even at case
series level. Thank you.

DR. GALSON: Thank you, Min. We are going to move on now to oral
presentations from members of the public who have preregistered, and the
first of those is Dr. Robert Nelson.

DR. NELSON: Good morning and thank you once again for the opportunity to
comment. And it is always a pleasure to follow my old colleague, Min
Chen. I know that she is a cornerstone for all the activities that are
happening here.

For the record, my name is Bob Nelson, and I am a consultant in drug
safety and epidemiology. Previously, I was a commissioned officer in the
Public Health Service, having spent 20 of my 23 years at the Food and
Drug Administration, both in the Office of New Drugs and in the
Post-Marketing Surveillance Activity, with my last position being the
Assistant Director in what is now called the Office of OPaSS, I guess.
It changes so rapidly.

What I want to begin with is just putting up my set of definitions, and
not to argue with the definitions in the concept documents, but just to
point out two things.

In general, the term, pharmacovigilance is considered a much more narrow
term, usually focused on spontaneous reports and not being the umbrella
term which I would call risk assessment.

And the second point is that the only term that is in the regulations is
post-marketing surveillance, and that is again an umbrella term that
encompasses all safety activities, not just the ones that we are talking
about today, but also advertising and quality of the products, field
inspections, and so on. So that is a term that should not be left out.

But I will just leave those for you guys to look at. But what I will
spend most of my time talking about is just commenting as you have
requested to the questions that you have laid out, and then I will end
by discussing some popular myths.

The first one, of course, is how can the quality of spontaneously
reported cases be improved. And I think that the most important thing
that could occur is the philosophy and the position that the agency
takes on them.

And I call that the public health focus philosophy, meaning that when a
report comes from a health professional, the agency should consider it
true for what the health professional has reported it as, unless the
company does sufficient research to show otherwise.

I think that is an important position to take rather than the Congress,
and that is a public health position, true unless shown otherwise. That
encourages to obtain as full data sets as possible, and rule out
alternate explanations, or rule in alternate explanations as the most
likely cause.

And I need to remind everybody that confounding, or in other words,
multiple indications or medical conditions, and drugs, does not rule out
a causal relationship, which I see so often done.

Secondly, I think we have to focus on those reports that are likely to
be important. Those reports that are for medically significant outcomes,
commonly known in the regulatory vain as serious, but in addition there
are a number of other ones.

And the most important way to increase the quality of those important
reports is at the query at the point of initial contact, and you will
see that the new proposed suspected adverse drug reaction regulations
also says that, and again I have to point to my colleague, Min Chen, and
I worked on those for so many years, and I am sure glad to see them out
finally.

And I think that once they are out and the comments are received, I
would hope that the agency moves as rapidly as possible to finalize
those regulations, and as soon as they are finalized, that they revise
the client\'s documents so that the compliance officers focus more on
the quality of the cases rather than the timeliness of the cases.

I think that is going to be a very important thing in turning the tide.
So that is my comments for question one, and again I want to focus on
this public health focus philosophy, because without it there is a
temptation among some companies \-- very few, but some \-- not to
acquire the needed data, because they know that the FDA will disregard
the case as not being credible. And that is a bad position to be in.
Question Number 2, advantages and disadvantages of data mining. I think
data mining is a value for secondary alerting, and what do I mean by
secondary alerting?

I don\'t think that any computer system or any statistical technique of
any kind can tell you what a signal is. A signal is something that a
person must judge. But a computer system is useful in telling you things
that are disproportionate, discordant, or whatever.

So they are useful, but they must be kept in context, and I call them
secondary because of course primary alerting is just recognizing good
cases, and so on. So it has a role, but it is not a primary role.

And it also has the ability, of course, to identify previously
unrecognized relationships amongst the data that would not be intuitive
by looking at an individual case. And they are most valid when you do
the internal proportional analysis.

The third question, is advantages and disadvantages of performing
causality assessments. I really think that it is useful? I think it is
important, especially when we have high volumes of reports as we do in
2003, to sort the good quality reports from the vast majority of reports
that are possible at best, and another large group of reports that are
just not \-- just don\'t contain any information value at all.

Because I believe that unless you have some good quality reports that
would lead to probable criteria, you really don\'t have very much
evidence, at least from the individual cases. Maybe when you get enough
possible cases you can do a content analysis, and a case series mode,
and you can extract a little more evidence.

But it is really the quality of the cases that are important in the
spontaneous reporting schemes, and not the quantity. Remember, these are
qualitative data.

And then when you do a case series review, you can add the possible
cases back in, and as my first myth is the myth of a definite case. I
think that is the unicorn of spontaneous reporting. There is no such
thing as a definite case in a single spontaneous report, even with the
positive dechallenge and rechallenge, because the background disease is
very common, and it could still be an artifact.

Four, what circumstances would a registry be useful. I think that we
need to be careful with registries. Just getting cohorts of people and
calling them a registry is not of much utility in and of itself.

In order to interpret the data from a cohort of individuals, you need to
have a valid comparative group, and registries I see being formed do not
have this.

Also, the safety data extracted from these prospective cohorts need to
be collected with the same methods and the same data collection
instruments that are used in the clinical trial environment, and not
just spontaneous reports rising from this cohort.

It does not give you interpretable data. And always remember that
prospective cohort studies or registries, which are the same thing, are
not powered to find new adverse reactions.

Many of you will recall that there was a big effort if you have a
history of this field in the late \'70s on prospective cohorts, and
there was quite a number of them assembled by industry. and actually
they were defused by a report by an FDA official, Alan Rossi, that
basically says that they weren\'t worth anything because we didn\'t find
any new reactions.

Remember that that is not what they are there for. They are best in
assessing risk factors for known reactions so that you can subsequently
prevent those known reactions, and so it gets to the root cause question
of why these are occurring.

And they are also a good source as I mentioned yesterday of quantitative
data to aid in defining the safety profile of a drug in those assembled
cohorts, but without comparator cohorts, you can\'t interpret them.

So be careful in just assembling registries to say that you have a
registry, because it is very misleading. What circumstances would active
surveillance strategies be useful in identifying yet unreported adverse
advents?

This is a tough one. Many of you also will recall the active case
control surveillance methodology that Sloan and Shapiro had in effect in
the 1970s. That was usually a hospital-based nurse-extracted dedicated
drug safety data, mostly funded by the FDA and the drug industry, that
has kind of disappeared.

I think it should be revisited now that we have modern computers and a
lot of other things, and I think we can better address this issue. I
think it is a powerful methodology, and the agency should consider
seeding the reinstitution of this kind of thinking.

Also, I think the MedSun program in the Center for Devices could be
extended to the drugs area to have representative samples of \-- not use
the facilities, but clinical practice environment that could report a
more concentrated from of reporting.

Thirdly, we have a resource in this country that is tapped, and that is
our Poison Control Centers and our Drug Information Centers, and our
university hospitals, and our regional systems.

And for those of you who don\'t know the French system of
pharmacovigilance, those regional pharmacovigilance centers, are only
secondarily pharmacovigilance centers. Primarily, they are poison
control and drug information centers.

So we have this network in the U.S., and we are not tapping into it. I
see that as an incredible resource for getting very high quality reports
from an institutional setting, and basically you can leverage your FDA
staff through contracts to these groups, and I think that really needs
to be explored.

That would be particularly powerful for drug-drug interactions, as well
as medication errors, and things that are going to be very difficult to
get anywhere else. So I think that those are three strategies that need
to be given some attention.

Now, pharmacoepidemiological studies, I think they are useful only when
they can be replicated. I am often stunned by the fact that we need
replication in clinical trials to get a drug on the market, but some
observational research that they can\'t be replicated can do great harm
to a drug once it is on the market.

I think that it needs to be replicated, and I think they need to be
validated, which is another problem, and they need to provide as I said
yesterday quantitative data, so that you can measure baseline risk, and
you can measure change from that baseline after an intervention.

If you can\'t do those three things, then you shouldn\'t be doing
pharmacoepidemiology. Now, let me change after looking at those
questions, and address some things that may also be helpful and may not.
I have been involved in working with a number of companies over the past
couple of years and getting them supported for advisory committees to
the agency and so on.

And in a recent advisory committee, I heard this statement, and I will
paraphrase. Today\'s FDA error system isn\'t of value because due to
under-reporting, incidents cannot be reliably calculated. And I heard
that statement and no one objected to it when it was stated, and I find
it amazing that a statement like that can go uncontested nowadays.

First of all, as a \-- and again along with Min, as a developer and
creator of the FDA error system, I take personal affront to people
banging on it like that.

What they are really talking about, of course, is spontaneous reporting
schemes and not the system itself, which is the computer software. But
the whole concept of under-reporting, which I will address in a minute,
is absolutely a fascinating myth that continues.

But before I get to that, let me talk about the confounding myth. We all
know that spontaneous reports seldom contain all the history data and
concomitant conditions and drug data that we would need for assessment.

But this absence may be due to one of two factors; either a true absence
\-- they didn\'t occur \-- or poor data acquisition. Conversely, the
presence of con factors does not automatically clear a drug from
causality.

I have seen in many cases where large numbers of confounded cases in
complex medical conditions nonetheless were dismissed because they were
confounded, and I think that is a terrible error.

Because unconfounded cases, especially in complex medical conditions,
may most likely be due to

poor data acquisition, and so you are actually relying on the worst data
collection techniques in those cases, and throwing out the better cases.

There is a lot of information to be gotten from those confounded cases
when you do content assessment of case series, and we need to stop
diagnosing individual pieces of paper and look at them more
epidemiologically.

Again, you need to apply this public health focus philosophy. In other
words, a report from a health professional is true unless otherwise
shown by good data collection techniques, and of course the way that you
look at case series is by Popperian logic of relative likelihood of all
qualitative explanations, because you cannot prove any one of these
things. All you can do is rank audit the likelihood of the different
alternate explanations.

The second myth that I would like to address is the reporting rate myth,
and this is one that always makes me crazy. The statistical assumptions
in Finney\'s 1971, The Class Paper, which talks about the random
distribution of non-causal events in spontaneous data, is no longer
valid in the U.S. spontaneous reporting scheme.

It may still be valid in the U.K. data, or the French data, because
there are much more refined and defined population. In the U.S., we have
all the non-health professional reporters, and we have non-series
reports, we have solicited reports. We have a whole bunch of stuff.

So all the statistical assumptions in that paper don\'t apply, and so
therefore we don\'t know the sampling scheme from which these reports
come from, and therefore, they are qualitative. And if they are
qualitative, they cannot be true, and they are not enumerators.

And if you don\'t have a true enumerator, there is no way to calculate
any kind of rate. So I think that these extrapolations that I hear from
the agency, as well as from the industry \-- oh, we think only one
percent, or 10 percent of these are reported, and therefore, we will
extrapolate \-- I think that is doo-doo, and we have got to stay away
from that, and we have to stop torturing these data.

On the other hand, there is an exception. Reporting rates can provide
some contributory evidence for regulatory decision making when the
outcome reaction is exceedingly rare, and there are a number of good
cases that show that that has been beneficial.

Otherwise, a calculation of reporting rates, if it is to be done at all,
can only serve to confirm that the signal may actually be real, and that
we ought to do some more research, and that is all that it can be
interpreted as.

It is much more valid to look at the internal proportional analysis
across the pharmacological class using the internal data as your
denominator than it is to use external data as a denominator given the
kinds of reports we have.

Now, my favorite one of all is the under-reporting myth. Spontaneous
reporting schemes were put into place and were designed to signal new
rare, unusual, and serious adverse reactions, and there is a very
powerful tool for doing that.

They were never designed to collect all adverse reactions. We all know
that. That is not a limitation or a flaw of spontaneous reporting
systems as it is a characteristic.

You know, you can\'t blame a car for it not being able to fly. It just
is not the kind of data that should yield the kinds of things that we
try to make it yield. I actually think given the initial intent of the
spontaneous reporting scheme that we have gross over-reporting in the
U.S., and not under-reporting.

We have much too many reports coming in and a high percentage of those
have no information value at all, and I know that we have taken some
steps in the new regulations, the new proposed regulations, to sort
between those that are most likely to be important, and those most
likely not to be important, and I think that is a good step forward.

The only true under-reporting in spontaneous reports is the
under-reporting of important data in important reports. Otherwise, I
think we have to get off this thing of saying that under-reporting is a
problem with spontaneous data, because it doesn\'t make any sense.

Lastly, I will just briefly talk about the secondary data myth, because
I did go into this a little bit yesterday. We have to be very careful
when we are doing any kind of observational research using medical
record and billing data, ICD-9-CM codes do not denote adverse reactions.

When you do a study in here, you are actually telling the differences in
disease or diagnostic occurrence between two groups after drug exposure
in one hand, and without drug exposure in the other.

And the differences may be due to the drug, but it may now. And they are
not adverse reactions. They are subsequent diagnoses. Plus, we have in
many of these data bases gross misclassification often can\'t be
assessed. So these data are not of the caliber or the kinds of data that
we need for risk management, and I urge the FDA to seed new data sources
with their contract vehicles, with the PDUFA money, let\'s move beyond
the current limitations.

And let\'s get to some dedicated drug data sources, and again I
encourage that the FDA seeds their creation. We need to stop torturing
existing data sources to yield answers that they cannot possibly
provide, and actually that should be a violation of the Geneva
Convention. Thank you very much.

DR. GALSON: Thank you very much, Dr. Nelson. Our next speaker is Dr.
Stephen Goldman.

DR. GOLDMAN: Good morning. Unlike Bob, I am focusing on one topic in
particular, and I enjoyed Bob\'s presentation as usual. I am going to
talk about the quality of the data, garbage in, garbage out; spontaneous
reporting systems that handle the quality of work that you get in.

And so that you know my background, I am first and foremost a clinician,
and it has been my privilege to treat thousands of patients over the
years. I do feel having been in academic medicine, in regulatory
agencies, I have treated patients in every possible health care system,
having been in industry as a consultant, and I think I have seen almost
all sides.

So I feel actually qualified to speak for my fellow physicians in saying
that I think I know what it is like to be in an office trying to decide
what to do with patients, and trying to decide which meds to use, and
what to do about adverse events.

One thing that I did want to point out for those of you who don\'t know,
I am also the former medical director of the MedWatch Program. So the
people of the United States paid me to get better reports into the
agency as one of my purviews.

I am sort of an amateur historian, and I always like to start with a
particularly apt quote, and I tried to figure what historical figure
could I use to start this talk, and I came up with the obvious one, Bob
Marley.

And Bob Marley sang V. Ford\'s famous words, \"And in this bring future,
you can\'t forget your past.\" There are lessons as Bob pointed out, and
I will point out, and I think Jerry will also, that there are things
that we know already that we seem to have either forgotten or the
program seems distant in our memory. We know a fair amount, and I am
going to run through some of those.

Actually, I am going to go back one. In 1969, the year that the New York
Mets won the World Series, there was a very nice study done at
Koch-Weser and colleagues, and they assessed the factors that impacted
physician reporting of ADRs at MGH, and they were the certainty of the
reaction, the mechanism of the reaction, the morbidity associated with
the reaction, prolongation of hospitalization, and the onset of
reaction.

Don\'t these things seem familiar in terms of what we look at. They also
looked at quality. Now, they also looked at quantity, which I will talk
about in a few minutes, but they noted that the program that they had
not only increased the quantity but frankly more importantly the quality
of the physician ADR reporting since they initiated the studies.

There was an ADR definition they applied that was easily applied, and it
was felt to be meaningful operationally, and what is so important for
practicing doctors and other health care professionals, it excluded what
seemed to be trivial side effects that did not seem to have true
clinical significance.

Secondly, feedback clearly showed that those reports were being both
carefully monitored and assessed. Feedback is crucial and we will talk
about that further, and they also thought that emphasizing the
importance of ADRs, in terms of monitoring, heightened the visibility of
the program throughout the institution.

All right. Now that is going on 34 years ago. Let\'s fast forward to
1988. Did the Dodgers win the World Series that year? I forgot. Okay.
They asked why physicians in Rhode Island were hesitant to report
suspected ADRs. Notice ADR, not ADE.

Thirty-eight percent didn\'t have the form. At that point that was the
famous 1639, and not the MedWatch form. Approximately 30 percent were
unsure the drug caused the reaction. And 24 percent said the reaction
was expected, and one of my favorites, 21 percent didn\'t know how to
report.

And in 21 percent, it never occurred to them, and close to three-fifth's
were unfamiliar with the FDA forms an guidelines for ADR reporting.
Following hot on that was what I thought and still think was the famous
Rhode Island ADR reporting project, in which an intensive program was
put into place designed to increase physician reporting of suspected
ADRs via sustained education utilizing several formats.

They found after two years a greater than 17-fold increase in direct
reports to the FDA by Rhode Island docs, while at the same time there
was no increase in the overall reporting rate in the United States. This
predates MedWatch by the way.

Now, the important part of that, which I always include now in a slide,
is the trend on serious events, and to reiterate what Bob was saying,
these systems are designed to pick up where unknown, generally
idiosyncratic events. Not the common events.

And you can see that they went from before the program, 0.4 percent of
total reports to the FDA from Rhode Island; and after the program, 3.6
percent. That is a remarkable increase and of that, 31 more armed
reports on unlabeled reactions.

I would consider this program a success. In addition to that, they did
pre-imposed intervention surveys, finding significant gains in both
knowledge and attitude towards the ADR reporting system.

That is, before the intervention, approximately half were familiar with
the ADR reporting system, and afterwards, 85 percent were familiar, and
similarly, approximately 40 percent were familiar with the FDA forms and
guidelines, and 70 percent, or 69 to 70 percent after the intervention.
I think that this is important information that I am bringing back to
memory.

MedWatch. MedWatch came into existence in June of 1993. To give you an
idea of a temporal association versus causal association, I was at the
FDA in that time, the temporal association, but I had nothing to do with
the founding of MedWatch. So no causal association. I came later.

MedWatch was launched in June of 1993, and Tony Piazza and (inaudible)
Kenney looked at both trends and reporting of serious adverse events
from \'92 to \'94, and the quality of the reports before and after
launching MedWatch.

What they found was that a proportion of serious adverse event reports
went from 34 percent in \'92 to 49 percent in 1994. By the way that
figure went up. When I left MedWatch in 1999, we had approximately
two-thirds of all drug reports to the FDA through MedWatch. That was a
voluntary reporting system, and two-thirds were on serious events.

And that is what the program was designed to do. MedWatch was never
designed to increase the total number of reports to the agency. It was
designed to increase the proportion of serious reports to the agency.

Secondly, they found the overall quality of the reports went up between
\'94 and \'93, and they determined quality as a significantly greater
proportion, say whether the drug was a newer entity, and they talked
about the seriousness of the event, and they also supplied the crucial
data that Min went through, lab data, clinical data, supporting your
diagnosis.

Pharmacist reports increased both in quality and in number, and
physician reports, although of high quality before and after, the
numbers did go down. The highest percentage report to the MedWatch
program were pharmacists and not physicians, and I believe that still
holds in terms of that.

That should not be a surprise, because that doesn\'t mean that the
pharmacist was the first one always to pick up the adverse event, but
they are often the first one to report it, particularly through P&T
committees and other systems they have.

All right. This is from Spain, and I thought that this was a nice study
that I found when I did my literature review for this presentation. This
is a case control study that looked at docs who had reported ADEs and
those who hadn\'t.

And they found that the probability of reporting an adverse drug event
increased with increasing prescription volume, and what a shock, it
decreased with increasing patient load. Not a surprise. But look at the
things that they found would lower the likelihood that you would report.

The belief that truly serious ADEs were well known by the time of
marketing. Well, we all know how true that is. The belief that the
determination of whether the drug was responsible was nearly impossible.

Reporting only when they were sure of the reaction was to the product in
question, and the belief that an isolated case seen by one doc couldn\'t
make any contribution to medical knowledge. I will talk about those
myths as I follow up and summarize.

In Germany, in 2002, they took a look very similarly at docs who
reported and docs who hadn\'t. Anywhere between three-quarters and close
to 90 percent never submitted a report tied to the government or
professional programs, and yet the reporting was much better from
pharmaceutical companies.

A whopping, almost two-thirds, or more than two-thirds, suspected an
ADR, but did not report it. The major reasons, once again, is that they
thought it was a well know ADR; they thought it was a trivial ADR, and
they were uncertain of causality.

In addition to that, they found that the highest probability of
reporting an ADR were for serious unknown ADRs for new entities, or an
older drug, or a serious known ADR to a new drug. Along with that,
however, 20 percent acknowledged no awareness of the spontaneous
reporting system in Germany, and 30 percent didn\'t know how to report,
and 54 percent would report if offered therapeutic advice.

There are common themes in these studies. I didn\'t have a complete list
at the time, and I did a literature search like I said. I looked at
several countries, and I don\'t know if this is good news or bad news.
The attitudes tend to be the same.

The complaints tend to be the same, and the reason why docs don\'t
report tend to be the same in Europe and in the United States. There
aren\'t much differences there. So what do we do about it?

Well, once again I went back to history. Abraham Lincoln said we must
use what tools we have. This was an assessment of the intervention used
to stimulate ADE reporting. This is from the Mississippi adverse drug
reaction, the ADR program.

They noted that docs, particularly health professional docs, didn\'t
like the fact that they got no feedback. They send in a report, and they
don\'t find out what happened to that report.

So what happened? They put together a newsletter that summarized both
the number and types of the reports, and the drugs involved, and they
also felt that this enabled docs to get educated about new drugs,
warnings, and careful observation. There was a very good response to
that.

Secondly, they felt that maintenance of awareness about ADR
surveillance, and a concomitant effect on quality of care was important,
and they found that posters strategically placed in the hospital, and
ongoing in-services in hospitals were effective. This is the Mississippi
program.

I might add that this preceded MedWatch. One of the things that we did
at MedWatch was make the attempt to tell people what we were doing with
these reports, which led to the first conference that we ran.

This was a conference that we ran at FDA, myself and Ron Leiberman,
actually working with Bob through the staff college when Bob was
directing that. We put together what we think at the time and still
think was the first conference designed specifically to talk about the
recognition and management of drug induced disease.

We used things that we had heard were effective. We used multiple
formats; straight lectures, panels, small group discussions. We used an
underlying clin-pharm approach, and we found that knowledge was improved
by pre-imposed testing. And in addition it was very well received.

That led to the first MedWatch continuing education article, the
[\"Clinical Therapeutic Recognition of Drug Induced
Disease]{.underline}.\" It was part of that conference, and we
distributed nationwide through the MedWatch partners, which is over 150
health professional associations who signed on with MedWatch to get
information.

And the carrot that we used was free continuing education credit; two
hours of contact for pharmacists, and two hours of CME for docs. The
results were very positive.

We got a 2.2 response rate. My understanding from the marketing
professionals that I discussed this with when I published this was that
that is a very good response rate for a blind mail out.

We got over 15,000 docs and pharmacists asking for the CE credit. You
will also notice that the majority were docs, in terms of that. Over 99
percent agreed that it met their objectives and was relevant. Obviously
a halo effect.

If you didn\'t feel that it was relevant, you would have sent it in the
first place, but they did

say that. What we didn\'t expect was spontaneous comments. People wrote
in the margins of the answer sheet. I got e-mails. I got calls.

This was so successful that led to the other MedWatch articles, such as
the next one[, The Clinical Impact of Adverse Event
Reporting]{.underline}, which I wrote from scratch rather than
conference, and others that were done on vaccines, devices, and they are
all still up on the MedWatch website.

The research program basically took that idea. This is up on the FDA
website, and this is a new learning module that was developed through
what was the Georgetown CERT, and now the CERT at Arizona.

And as you can see, it is about preventable ADRs, and it was based on a
need survey that was sent to all third-year medicine clerkships, and
residency program directors.

It was sponsored by AHRQ. I have no data from this yet, but we are
looking forward to seeing how effective this is. Other successful
interventions. Very briefly just as an example of a multidisciplinary
ADR committee, they have a form that they developed, a 24-hour hotline,
an ADR newsletter, broad range in-service.

Pharmacists and investigators sit on the committee, and increased
numbers of reports. Switzerland, a clinical pharmacist making rounds,
soliciting information from docs and nurses, doing chart review, and
what a surprise; improved ADE, ADE identification and reporting.

Two other papers looking at the morning report as a way of facilitating
detection of AEs, and as a form for reporting AEs. Both of them were
very effective, and the references are there.

We talked about form completeness. I am a tremendous believer in
directed questioning and a check-off list. The MedWatch instructions for
both the voluntary and the mandatory go item by item. I think they are a
good guide frankly, in terms of utilizing it.

The idea of adverse event-adverse reaction specific questions is
crucial. If you know that you have a problem in the liver, or a
perception that you do, if you put together a list of questions specific
to liver, and those are the questions that should be asked when somebody
calls.

Homicide detectives will tell you the first interview is crucial. It is
no different when someone calls in to give you an adverse event report.
You get one shot for the most part, and that first shot is the best shot
in terms of that, because people don\'t want you to call them 2, 3, and
4 times, and it becomes diminishing returns.

So I really advocate for this, and I mention two examples, both liver
functions and the dermitities. These are the kinds of data that we
talked about getting, and I will put in a particular plug. When you ask
somebody what medicines they are taking, ask them everything that they
put in their mouths or on their skin during the day.

People don\'t think OTC drugs or dietary supplements are drugs. It is
like if you ask someone if they drink alcohol, and they say no. How many
beers did you have? Seven.

People sometimes don\'t perceive that as being alcohol, and the same
thought with taking histories. These are the things that you have to
ask. Temporal information, biopsy, or god forbid, autopsy results, and
dechallenge, and rechallenge.

This last slide before I go to my summary is to point out that when we
look at medication errors, take a look at the relative percentages of
what causes them.

Eighteen percent were errors, correctable errors, in calculations,
decimal point placement, unirate expression, but 60 percent were
deficiencies in knowledge. That is not remedial, per se, by changing a
package label. That is a knowledge deficit, and that is a harder issue.

Lessons learned. I fully believe that when we educate health
professionals that it should not be product specific. It must be
clinically oriented. The goal should be to increase the awareness of
medical product induced disease.

The only way a doc will diagnose an adverse event is to consider it in a
differential diagnosis. You must enhance knowledge of the application of
pharmacotherapy, and the impact of the individual factors that patients
have on pharmacotherapy.

Thirdly, you just explain clearly not only how, but why, busy health
professionals should be reporting adverse events to the regulatory
agency or both. Feedback is crucial, and I believe the more clinical the
feedback should be the better.

Education, education, education, all levels, in professional schools,
training programs, post-graduate continuing education, and in the
clinical care settings; hospitals, clinics, and other settings.

That includes all health professional disciplines, and it must be
ongoing. One shot programs will not work. The ADR program in Rhode
Island was not a one shot. It was a two-year sustained program, and look
at the results.

The programs that I mentioned about having a pharmacist on rounds. These
were not one-shot deals. They were ongoing. You have to demystify the
process of adverse event reporting and assessment and what goes on in
the FDA and industry, and I am going to do the same thing that Bob did,
exploiting some myths.

Obviously all serious adverse event reactions are not known at the time
of marketing. That\'s why we have post-marketing surveillance. Causality
not being a requirement for reporting. This is crucial. How many of
those studies that I just showed, people felt that if they couldn\'t say
it was due to the product absolutely, they wouldn\'t report it.

Wrong. That\'s completely wrong, and we tried so hard with the MedWatch
Program to make it clear that you didn\'t have to know that. It was not
a requirement, and it even said that on the form.

A single case can lead to general knowledge. Somebody somewhere has to
be the first health professional or consumer to report that particular
adverse event. So someone somewhere was the first one.

And, fourthly, even if a known adverse event is known, accumulating good
cases on serious known adverse events can definitely increase general
knowledge, can look at possibly patients at greater risk, and certainly
specificity and severity does make a labeled event unlabeled in terms of
that.

We need ongoing evaluation of methods to solicit information. I
mentioned directed questioning, and specialized questions for known
adverse events in AERs of interest. The brand new proposed rule category
of always expedited reports, the known bad actors, the reason that we
have post-marketing surveillance.

The bad news is that we have no quick fix. There must be a commitment of
resources, and that means financial commitment of resources, personnel,
involvement of multiple sectors and partnerships.

This is not just the FDA\'s problem. This is not just industry\'s
problem. What we wrote in the 1999 task force to the commissioner made
it clear that this is s a shared responsibility from all sectors,
including health professional associations. And that includes doctor
health professional associations.

I will leave my last slide with a historical statement. I am always
concerned about people making the correct attribution between the
methods used and the results. We have talked about this a lot the last
three days.

I have what I believe is one of the greatest single line correct
attributions in history. When George Pickett was asked why his attack on
Cemetery Ridge failed at Gettysburg on Day 3, what he answered was, \"I
think the Union Army had something to do with it.\" I will stop here.
Thank you.

DR. GALSON: Thank you very much, Dr. Goldman. Our next speaker is Dr.
Gerald Faich.

DR. FAICH: Good morning. I am Gerry Faich. It is a pleasure to be here.
While I am here as yet another consultant, I also will be drawing on my
background as FDA\'s office director. I was at the agency when the 1985
NDA rewrite was put in place.

That indeed was or did form the structure for adverse drug reaction
reporting as we now know it, in terms of defining serious, and 15 day,
and the like. I am also here as a member of the board of directors of a
safety search company, and so I need to say that, and that is Centrix,
by way of full-disclosure.

My opinions, however, are my own. I found the concept paper as outlined
this morning to be thoughtful and useful. I think it attempts to
carefully balance concepts involved in spontaneous report collection, as
well as analytic epidemiology studies, which we are going to hear about
this afternoon.

So I salute those who wrote the concept paper. I, too, will stick to the
questions that were put before us, and I would like to comment on
spontaneous report quality, data mining, causality assessments,
registries, and then I would like to comment on large simple as an
aside.

Even though that was mentioned two days ago, I think that is pertinent,
and so you will notice that I am not going to comment in two areas.
Let\'s talk about quality of case reports for a moment. We have over
many years now concentrated on coding issues, standardization issues,
and report submission issues.

I strongly believe that the key in the goal of going forward ought to be
to improve the content. So if that is true, the question is how does one
improve content, and I would submit that the way to do it, and Steve
Goldman just mentioned this, is to get as much information on the first
encounter with the reporter in a structured manner as possible.

If that is true, the question is how does one do that. Some of the
guidelines talk about using a physician for primary intake, and I think
that is not a feasible approach. I think that it is not cost effective.

However, I do strongly believe that a trained health care individual
should be, if not the first answer of the phone call, the first referral
point for in taking information.

The other thing that I would submit, and we have learned this from
experience, is that the intake point should utilize computer assisted
technology with built-in logic.

As Steve Goldman just said, if you know the problem is in the chest,
then you ought to have an algorithm which branches the questioning to
get you to lungs, and from there, to shortness of breath, in a
structured way that guides both the questioner, but also the intake of
the data.

It is perfectly clear that you will improve the quality of the data, the
quantity of the data, and you will contribute mightily to
standardization of the data, and the database, even in the intake
process.

Moreover, if that system is automated, it is clear as well that at the
end of the interview you will know which data elements are missing, and
you can make an arrangement or a contract with the reporter at that
point in time for call back on those specific items.

I again a hundred percent agree \-- and anybody who has been in this
business knows, that if you called back the reporter 2, 3, and 4 times,
by the fourth time, forget it, that you are not going to get further
data.

And by the way, as an aside, the new safety tone which came out on March
14th, suggests that sponsors will have to document their due diligence
for the multiple phone calls, and submit that due diligence to the
agency.

I think that is wrong-minded frankly. I think that is a regulatory
burden that does not have a payoff. I do think, however, that
establishing right at the outset with a reporter what data elements are
missing, and how those will be obtained, and when the call back will
happen, is dependent upon using automated telephone-based systems in
large part.

What we normally do when we intake case reports is that there is a
reception point. We log it in, and we write it up. We code it. We enter
it. We validate it. We review it.

Then somebody finally gets around to submitting a form, and what you
hear me saying here is a plea for using more direct electronic immediate
data entry. Again, I would emphasize that I am not talking about mere
wordprocessing.

We are talking about a built-in logic in the script that an informed
health care recipient will be receiving and intaking information. It is
sorely lacking. This technology does exist. It is being used
increasingly, and I strongly believe that this is the way, the single
most important way, to improve adverse drug event report quality.

In January of this year, I and Hugh Tilson, and Ralph Edwards, and
others, participated in an International Society of Pharmacovigilance
meeting in Paris.

We, among other things, set out some 10 or 12 points of interest that we
are going to be discussing at the International Society of
Pharmacoepidemiology meeting this August.

But I cite it here only to tell you that one of the points was relative
to signal detection, we noted that despite report volume issues that the
reporting rate again must be seen as not a true measure of the risk
rate.

The reporting rate is just that. It is the number of reports,
spontaneous reports, over some denominator. It is a signaling device
that needs to be used with caution.

Similarly, you will see here at the bottom that we noted that when
observational studies, epidemiologic studies are done to investigate a
report and a signal, those should be seen when well done is largely
trumping signals that arise from spontaneous reports.

So I would submit that in terms of both report quality and in report
interpretation, once a signal occurs, if it is important, there ought to
be great urgency in getting a pharmacoepidemiologic study mounted.

And when it comes in, in general, it should be seen as outweighing the
signal value of spontaneous reports. Let me turn to the second area, and
that is data mining. The FDA database in particular has evolved over
time as you well know.

That database has millions of reports in it, and is now receiving
250,000 reports a year, as opposed to the 5,000 when we started in 1983,
and the 50,000 in 1990 when I left the agency.

Because of that the database overall has marked variation in terms of
the quality of reports, the requirements that were present at the time,
waivers have affected it, and the like.

This is simply a way of saying that there are numerous non-biologic
influences in the database, and that is bound to perturbate any data
mining effort. You can make efforts to control for that, but my biggest
concern about data mining is the generation of false signals, which
inevitably will happen.

And I don\'t think that is necessarily so bad. I think that we can
interpret false signals as long as we know that they are there. The
question is what is the overall value.

Two quick examples of false signals that come out of data mining. One of
them was a letter published by Ralph Edwards, in fact, in
[Lancet]{.underline} about the association of Claritin, a
non-arrhythmogenic antihistime, and arrhythmias, and in fact it was
based largely on U.S. ADR reports that flowed through the air system.

And in fact FDA reviewers wrote a letter to [Lancet]{.underline} saying
totally wrong. What we are looking at are people who had pre-existent
arrhythmias, and that this is a clear example of channeling bias. Those
that were at the highest risk of having the outcome were assigned the
safest disease, will still manifest some of them, that outcome, and that
is obviously a non-causal selection bias.

We are bound to see that in data mining, and of course one of the other
issues is that you can\'t know that if just use the data that are in the
electronic database. You may well need to have much more data that would
be available in the appendices and the like.

So I would refer you to that glaring example of a false signal achieved
by data mining. Another one that one will find is if you look at prozac
and suicide, you are going to find out that prozac and suicide has the
highest rate certainly in the neuropharmacologic agents, clearly
confounded by indication.

It should surprise no one that the press people commit suicide and that
there has been a large number of reports that came in after publicity
about this. Also, not easily controlled in a data mining exercise. One
way to go at this is that I would recommend that data mining be
restricted to within a therapeutic class. I think that comparing
antibiotics or cardiovascular agents to neuropharmacologic agents makes
no sense at all to me.

And I certainly since Steve was quoting Abraham Lincoln and the like, I
will quote Yogi Berra, who has of course said that if I hadn\'t believed
it, I wouldn\'t have seen it. And I think that is a clear thing that
goes on in data mining.

Let\'s move on and talk about causality assessment quickly. We all know
that there is no standardized way to do causality assessment. Navarro,
or algorthims, or others, doesn\'t do it. Koch-Weser recognized this in
1968.

In fact, what hasn\'t been said is that Von Koch-Weser showed that if
you gave a panel of five physicians 30 reports, you would get virtually
no agreement between the physicians as to whether it was possible or
probable.

But more interestingly if you shuffled the reports, and gave them back
to the same reporter, you wouldn\'t get any agreement the second time
aground. So you have inter-observer and intra-observer variation.

And one has to say wait in terms of the value of doing what is a time
and energy consumptive process. When I got to the FDA in 1983, causality
assessment, report by report as reports came in, was the routine. One of
the first things that we did to improve the process was to stop that.
And we stopped it because it was backlogging the system, and it was not
all clear what the value of that was. So I am going to suggest that we
should not do routine causality assessment, with two notable exceptions.

One is in the conduct of clinical trials for serious adverse events not
in the investigator brochure, and the other one is that once a signal
surfaces, however it surfaces, then I do believe that you have to lay
out the reports and do this arduous process.

But I would hate to see us retrogress to the point where we expended
energy and resources on the front end, when it is not clear or useful at
all. I am going to skip this, because I think that I just said it.

But I would point out again the new safety tone calls for submission of
serious reports from trials that cannot be where causality cannot be
ruled out. Those reports are to be submitted in an expedited fashion.

I suspect that that is wrongheaded, and by the way, the change is that
the current regulations say that if you have a serious AE not in the
investigator\'s brochure, that is reasonably and possibly due to the
drug. Notice the language. That is the standard, and then it should be
submitted.

This one says that if you can\'t be sure that it might not be associated
with a drug, then you do have to submit it. I think that is going to
clog the system, and I think that we are going to lose our ability to
discriminate between those reports, and I would submit that we should
revisit that issue.

DR. GALSON: Dr. Faich, you have got one minute.

DR. FAICH: Thank you. I will speak quickly. In terms of registries,
follow large numbers of patients. I have three problems with registries,
both on the demand side from FDA, and also on the feasibility side.

We are asking registries to be too large and to last too long. They
can\'t do it. You can\'t follow patients much beyond 5 years, except for
using national death index, and I would urge caution in doing that.

I also think that data collection needs to be highly focused, and by the
way, I would say that we ought to think more and more about directed
patient interviewing in these registries.

Large simple safety trials. I just put this out here because I think
that is going to be the fundamental tool. If we depend on classical
pharmacoepidemiology, we are going to have to wait two years or more
until there is enough population penetration.

And instead of doing that, we ought to do registries in large simple
trials if we need to know answers quickly. There is technology, again,
that facilitates both registries and large simple trials. This is taken
from a smoking cessation study, and it just is using telephone
interviewing to accrue the data on the front end, and to output the data
on the back end, something that needs to be given higher priority.

If I can get past this, I will summarize my talk under a minute
hopefully. No, I won\'t. Well, let me summarize then. I have talked
about the need to impact improved quality by improving intake of
spontaneous reports.

I question the value of data mining as you have heard me say, and I
think that registries do have a role. I think that we need to be careful
about their sizing and which data gets collected, and I think that we
should harness new technologies to get our data faster and in ways that
make more sense to us. Thank you.

DR. GALSON: Thank you very much. We are now going to take a break until
9:45, when we will have Dr. McClellan. Thanks a lot.

(Whereupon, at 9:31 a.m., the workshop was recessed and resumed at 9:46
a.m.)

DR. GALSON: Okay. Thanks, and if folks could come in and get seated. It
is my great pleasure to introduce now our Commissioner of Food and
Drugs, Dr. Mark McClellan. I am not going to go through his whole bio,
because I think that his background is very well known to most of you.

But just a couple of notes. From 1998 through 1999, Dr. McClellan was
the Deputy Assistant Secretary of Treasury for Economic Policy, putting
him as part of a small club of people who have been political appointees
in both the Clinton and the current administration.

During 2001 and 2002, Dr. McClellan served in the White House. He was a
member of the President\'s Council of Economic Advisors, and he advised
on domestic economic issues. He also served during this time as a senior
policy director for health care and related economic issues.

Dr. McClellan\'s research interests and studies have addressed measuring
and improving the quality of health care, the economic and policy
factors influencing medical treatment decisions and health outcomes,
estimating the effects of medical treatments, technological change in
health care, and its consequences for health.

And medical expenditures and the relationship between health and
economic well-being. During the few months that I have had the privilege
of working with Dr. McClellan. I am, and the other senior managers at
the agency have been incredibly impressed with his energy, his interests
in detail, and the complexity and challenges of the issues that we are
facing at the agency.

And as well, his clear focus on the public health parts of his task. So
now I am really happy to introduce Dr. Mark McClellan.

COMMISSIONER MCCLELLAN: Thanks, Steve, for that introduction. That
actually made my day. It is very much appreciated. I am very glad to be
here with you all today. This is a great conference room and I hope that
you will appreciate what I have to say.

I am a little nervous with these drug experts behind me, and so if they
start making faces, or groaning, or something like that, I hope that you
all will let me know.

But again I am very glad to be here. This is an important meeting that
is going to be of great value in helping us do our job more effectively,
and that is very important today with the tremendous and growing
responsibilities that this agency has, the growing volume and complexity
of our responsibilities, both for medical products, and for food, and
even national security issues, with limited resources.

We have got to prioritize, and we have got to focus on using the most
effective methods possible for carrying out our mission, and so that
means that we have to think critically and carefully about how we can do
the most good for the public health with the resources and authority
that we have.

In some cases this is requiring a reexamination and an updating of the
way that the FDA is doing its job. To bring all of this together, the
agency has undertaken a major strategic action plan that is moving
forward right now within FDA, and we are going to have much more to say
about soon.

One of the central themes in our strategic planning is the concept of
efficient risk management, and I know that risk management is the topic
for much of what you all are discussing here, but I want to emphasize
that efficient risk management goes way beyond the risk management
programs for certain pharmaceuticals that are the immediate topic of
this in related meetings.

The broad principles, however, are very much applicable to your efforts
here, and that is one reason we are so interested in hearing from you on
this topic.

What we mean by efficient risk management is first, using the best
scientific evidence to assess and manage the risks facing the public.
This includes both the safety risks of the medical products that we
regulate and the risks from the diseases that they are intended to
treat.

This is one of the many areas of regulatory science where the FDA has
long led the way in assessing risks to the public health and identifying
ways to manage these risks.

Second, now more than ever we need to do this efficiently and in the
least costly way from the standpoint of the agency, and especially from
the standpoint of society.

Lower costs of risk management means lower cost of medical products, and
that translates into a greater input to do on the public health.
Efficient risk management in all of our regulatory activities is more
important than ever given the expanding complexity of our challenges,
and the need to reduce the health risk facing the public at the lowest
possible cost.

As drug costs continue to rise, and as health care affordability is a
front page issue right up there with the war and national security
concerns, steps to reduce the cost of drug development and to increase
the value of prescription drugs in use are more critical than ever.

A major new technology development initiative has been included as part
of our risk efficient risk management program. This has focused on the
pre-market side of drug development, and many of you I am sure are
familiar with it already.

It includes an analysis of the so-called multiple cycle product
approvals that have occurred in the agency in recent years to identify
whether at least in some cases those multiple cycles might be avoided,
making it possible to make a safe and effective new drug treatment
available more quickly and at a lower cost.

Second, the implementation of quality systems in our review procedures
to make sure that we are using the best possible review practices and
rating ourselves on how we are doing.

And third, new guidance that we intend to develop collaboratively with
outside experts in key areas of disease treatment and emerging
technologies. Another part of our efficient risk management initiatives
include an overhaul of the pharmaceutical good manufacturing practices
based on the latest science of risk management and quality assurance in
order to encourage innovation in the production, and manufacturing, and
distribution of pharmaceuticals.

From what we have seen and in talking to outside experts, there are many
opportunities for producing more efficiently, with fewer errors, with
fewer defects in production, and that, too, can translate into lower
costs and perhaps even safer medical treatments.

Finding ways to help ensure that drugs we review are used safely once
they are approved, and to reduce preventable adverse events through
better information, better systems for monitoring, assessing and
reducing adverse events, is also part of this broad initiative. And that
is why we have asked you to participate in this meeting, to help the
agency consider new concept papers that stem from risk management
activities growing out of last year\'s important legislation
reauthorizing the Prescription Drug User Fee Act.

In this area of mitigating adverse events the agency is taking many
steps to reduce the too common occurrence of adverse events involving
the products that we regulate.

Our new regulation requiring bar codes on pharmaceuticals will be
important in reducing adverse events due to dispensing and administering
pharmaceuticals improperly. It will help make sure that the right
patient gets the right drug and the right dose, and in the right form at
the right time.

We are also working on developing information technology tools to
identify adverse events automatically in a timely way, and to provide a
two-way street so that we can provide feedback quickly and effectively
on ways to mitigate the risks that we identify.

We have entered into partnerships with hospitals, such as Columbia
Presbyterian in New York, through the Marconi Project, and in our
devices center, we are expanding our so-called MedSun initiative, which
relies on the web for two-way communication to identify and respond to
adverse event problems with devices, and hopefully that can be expanded
to drugs and other treatment soon as well.

But adverse events that can be prevented aren\'t just caused by medical
errors. Some are caused by rare problems, such as liver toxicities, or
adverse interactions in certain groups of patients that may only become
apparent after a product has been on the market in use with real world
patients for some time, rather than in the somewhat idealized conditions
of a clinical trial.

Finding better ways to monitor the safety of new drugs could improve the
drug approval process, as well as give patients and doctors greater
confidence that the drugs that they use will work as expected. And if
they don\'t, will give them greater confidence that the problems will be
identified quickly and addressed quickly and effectively.

We have an unprecedented opportunity to implement these mechanisms now,
thanks to the important new post-market authorities in the PDUFA 3
legislation, and thanks to the agency\'s and the department\'s strategic
focus on improving patient safety and improving the use of modern health
information systems.

We are looking for better information in this process, and not more
information. We are looking for more useful information for guiding
decisions about the benefits and risks throughout the process of
developing and using new technologies, not burdensome additions to the
requirements for approving new products, and monitoring their use after
approval.

But this kind of creative thinking isn\'t easy, and we need your help.
That\'s why workshops like this are absolutely critical, and as we
consider use of the drug development process, we need constructive
suggestions from all sources.

This is the best way to reduce the time and the cost of the complex
process of developing medical technology to fulfill our shared goal of
giving patients more value in the medications that they use.

We hope that industry will work with us in this effort, and I am very
encouraged by the way this discussion has progressed so far in today\'s
and yesterday\'s workshops, and we hope that the product developers will
work closely with us to plan and implement these risk management
programs.

We also hope that the result, better Phase IV studies, can potentially
reduce the amount of information we need to collect during the
preapproval process, and I believe that this will be possible.

But to achieve this goal, we need to identify specific opportunities
where it is possible. We can also use guidance on the broader framework
from improved Phase IV activities. That is, on how we can through our
approach to pre-approval and post-approval studies, achieve the greatest
reductions in the health risks facing the public.

This risk management includes both risks of poor outcomes from adverse
events caused by treatments, and risks of poor outcomes that can be
prevented by the use of new treatments.

In addition, I believe, and I think that many from yesterday\'s
discussion, many of whom were here for yesterday\'s discussion, also
agree that the FDA can encourage safer and more effective, and high
value use of prescription drugs by helping patients and health
professionals get better information.

That includes better package inserts. They need to be more effective
benefit risk communication tools, and we think that there is room for
improvement. For example, package inserts may not be designed so that
health professionals can get the information they need, the most up to
date information they need to treat a particular patient.

And many package inserts today have become so laden with legal
considerations that they are often hard for the average consumer to
understand, and they are often ignored by many physicians.

Finally, we aren\'t yet taking advantage of the possibilities provided
by modern electronic health information systems for accurate and easy
communication of the relevant information contained in a product label.

In all these areas, the FDA intends to make progress in the months
ahead. As we approach the FDA\'s century mark, it is important to
remember that the FDA\'s hallmark has long been its ability to adapt to
keep up with the best new science and to use new opportunities to
fulfill its mission more effectively.

This meeting is a very important addition to that tradition, and I am
sure that it will help us find better ways to fulfill our mission in
light of new technologies and new statutory authorities.

I want to thank all of you here today for your commitment to help us
find the best way to protect and promote the public health. Thank you
for coming, and thank you for participating, and thank you for engaging
in this very important constructive dialogue.

I am looking forward to seeing the results of this meeting translated
into improvements in our guidance, and improvements in our efforts on
pharmacovigilance, risk management, and other critical responsibilities
for the agency.

Again, thanks very much, and I would be happy to take a few questions if
you have got any. Thank you.

DR. GALSON: Thank you very much, Dr. McClellan. If folks have questions,
please come up to one of the mikes in the audience there. And while
people are thinking about it, Dr. Sullivan, did you have a question?

DR. SELIGMAN: Since I am being put on the spot. This particular workshop
was often focused on sort of the risk side of the risk benefit balance,
and I wondered if you would like to say a little bit more about what the
agency can do sort of on the benefit sides of medications, and ways and
thoughts you have about promoting both the beneficial use of medications
that may be under-utilized, or are not fully utilized in our health care
system.

COMMISSIONER MCCLELLAN: I think even though as you said the focus here
is on the risk side, obviously in the decisions that we are going to
make based on the results of a meeting like this one, in particular
cases of pharmaceuticals the benefit side is very important, too. I have
often said that there is no such thing as a completely safe medication.

It depends on the conditions under which they are used, and the benefits
and the risks facing an individual patient, and we need to do more. We
need to do all we can to try to make sure that medications are used
where the benefits outweigh the risks by the greatest margin.

So while we are focusing on the risk side today, obviously in any
particular decisions about pharmaceuticals, we will consider those risks
against the potentially important benefits of the medication as well,
and that is what I meant by considering all the risks.

Diseases that medications are intended to treat, as well as risks of the
medications themselves, in our approach to efficient risk management
throughout the agency. And the things that I think can help us work on
risk can also help us work on benefits.

Better information for patients and for doctors on labeling to help them
use the most up-to-date evidence in their treatment decisions. Programs
to help prevent medication errors and adverse events may also prove
handy in supporting higher quality care if we have got these kinds of
information systems and mechanisms in place.

So I think that there are a lot of opportunities to work on both
together, and while the focus here may be on the risk side, you can rest
assured that we are thinking about the same kinds of insights and
opportunities to improve the benefit side as well.

DR. GALSON: Great. Please identify yourself.

DR. NELSON: Dr. McClellan, thank you for your comments. My name is Bob
Nelson, and I am a consultant, but a former FDAer. I apologize if this
puts you on the spot, but since you are here I will let it go.

COMMISSIONER MCCLELLAN: That\'s all right. It comes with the territory.

DR. NELSON: Dr. Galson mentioned the DailyMed program. From what I know
of it, I think it is very exciting and a collaborative effort with VA
and the National Library of Medicine to put bar-coded labels.

But I understand that that has not really been funded. If that is true,
can you verify that, and if it has not been funded by the agency, can
you comment on why?

COMMISSIONER MCCLELLAN: Well, let me take a step back and talk about the
agency\'s overall funding levels. The FDA is bigger than it has ever
been, both in terms of personnel and in terms of resources available,
and I might add to that in terms of statutory authorities with things
like the new post-market authorities in PDUFA-3.

We had a big legislative year last year. The problem is that although
our resources are expanding, so are our responsibilities, and so are the
complexities and difficulties of the public health issues that we are
charged with addressing, and so we do have to think very carefully about
prioritizing.

We can\'t do everything, and we can\'t devote unlimited amounts of funds
to each individual product, and Congress, and the rest of the
Administration, also have a say in how these priorities are set, and we
try to work with them to make sure that we are focusing our efforts and
resources on where they can do the most good for the public health.

I don\'t know the exact details of the current funding status of the
DailyMed program. I can tell you that a lot of work is proceeding on the
development of that program. That is a good example of what I mean by
how we can improve the medical labels, the pharmaceutical product labels
in a way that should lead to better information being available to
physicians.

And as a result, better care for patients, and so that effort is
proceeding. I think that there are a lot of opportunities to implement
steps in the DailyMed program without a major new expansion of funding.

For example, these collaborations with the VA are ongoing, and if we get
established some ties with health care organizations like the VA,
whereby they can send us information on potentially important adverse
events that we may need to take steps to help address, we can make it a
two-way street by giving them access to updated information that is
relevant for the product labels.

And that is moving forward, and so we will keep working as hard as we
can to identify how we can use our limited funds as effectively as
possible, but I don\'t think we are in a situation where DailyMed is not
moving forward at all.

In fact, it is one of the more important and I think promising efforts
in our whole set of patient safety initiatives.

DR. GALSON: Great. Thank you.

DR. SZARFMAN: My name is Anna Szarfman, and I am a medical officer at
the FDA, and have implemented both the patient profile viewer that is
now being incorporated into that routine, and I have worked together
with Dr. Demichele to implement the information on data mining tools.

COMMISSIONER MCCLELLAN: Yes.

DR. SZARFMAN: And the questions that I have is how \-- you know, with
the patient profile viewer, in the beginning it was difficult, because
people were accustomed to things such as lots of paper and equivalency
in other ways until we started these studies, that we were gaining speed
tremendously and in a tremendous way.

And then there were no questionable difficulties in implementing that
piece. We think that data mining is more difficult because people think
that we are torturing the data, and like you have heard this morning,
and we don\'t systematical approaches in place.

How can we set up like in the clinical pathology lab in the clinical
pathologies. How can we move from one place to the next one that is
(inaudible). You know, we have systems in place where we can test things
objectively, and I think that we have to do it.

And that helps everybody to have objective methods and recently in a
(inaudible) committee, data mining provided some information that was
critical to understand where are the specific drug, and if it was
hepatoxic, or if it was more hepatoxic, or less hepatoxic, and so I
would say it helps with the answers.

COMMISSIONER MCCLELLAN: I think that those are all good points.
Obviously in any activities that we undertake in this whole risk
management strategy, we need to think carefully about the most efficient
ways to move forward. You know, where are the greatest needs, and where
are the greatest opportunities for improving the way that we do what we
do.

And absolutely we need to test whether the changes that we implement
have an objective impact on important performance measures, such as
review times, and review costs, and whether we are identifying risks
more efficiently.

That really is the core goal here. There is a tremendous amount of
information out there, and there is getting to be more and more every
day, and the question is can we use that information efficiently and
effectively in guiding our review decisions.

I think the work that Dr. Demichele and others are doing on data mining
and related innovative statistical techniques is very important, but in
some ways it is harder than the patient profiler.

You know, it is not just a matter of implementing a software program
that presents the information that is there more efficiently and easily
for people to use. It is a matter of drawing inferences about and sort
of extracting signals from very complex data that is not completely
straightforward.

There is not a yes/no answer. There are different degrees of competence
about the conclusions that can be reached from data analyzed using any
statistical method.

And so that is going to be a more complex technical process to improve
upon. But I think that the data mining holds a lot of promise, and I
think that there are a lot of other techniques that have been suggested
to the agency by other outside statistical experts like Dr. Demichele
that hold tremendous promise for helping us make more or do more with
the information that we get.

And again as the information that we get continues to grow in
complexity, both in terms of the information coming in and what it
means, we need to make sure that we are up to date on our statistical
methods.

And we need to have methods in place to evaluate whether those
statistical methods are having an impact in a positive way on what we
do. So I think that these are the right kinds of issues that we need to
think about, and we need to couple it with making sure that as we move
forward on some of these ideas that we are seeing an impact on our
agency\'s performance.

DR. GALSON: We have time for one last question.

DR. YODRIN: Sam Yodrin, head of drug safety, Millennium Pharmaceuticals.
Thank you, Dr. McClellan, for your talk, which really highlighted two
significant issues that we have to confront in this new era of risk
management during productive life cycles.

The first is efficiency, and it is very good to hear a regulator talk
increasing of cost effectiveness, and also the early identification of
significant signals.

And then also in terms of effectiveness, and the issue of data quality.
Increasingly we are being asked to have an evidence base for our
decision making with regards to our risk management.

And the current situation is characterized by increasing partnerships
that are being established primarily by the FDA with health maintenance
organizations.

There is the CERTS program, but the challenge there for industry and
companies, whether they are a big pharma or a small pharma, alike, we
all cannot establish these types of relationships to ensure that we have
the available sources of data to enable the appropriate risk assessment
that we are now faced with.

And I know that we have made this representation to you in the past
through a different medium, but we strongly propose here that there is a
collaborative effort between the agency and industry towards looking to
establish an integrated national pharmacoepidemiology database that
would address all the disadvantages that are currently inherent to the
fragmented approach that is being taken right now.

And we believe that this integrated national pharmacoepidemiology
database would align with the current risk management framework, in the
sense that it would involve all relevant stakeholder parties in
collaboration, and all the issues that relates to data quality certainly
we could address and maximize the benefits of information technology.

And so I just wanted to use this opportunity again to make this proposal
to the Commissioner. Thank you.

COMMISSIONER MCCLELLAN: I think that idea, such a national high quality
system, is a very interesting one, and a very nice vision for where we
would like to end up.

And as you pointed out, we are exploring expanded partnerships with
health care organizations and the like, and basically we are going where
the data are, and we don\'t have such an integrated national system now,
and I don\'t want us to wait until we finally get one before we move
forward on some of these initiatives.

I think that this is going to be an incremental step-by-step process,
but certainly the product manufacturers and product developers have an
important role to play in both helping to develop this kind of
information and helping to interpret the data that are there.

So the idea of moving towards more of a partnership as we move towards
better data systems sounds like a good one, and I think it is something
that we should keep in mind.

DR. YODRIN: Thank you.

DR. GALSON: Dr. Sachs.

DR. SACHS: Thank you for letting me ask my question. Susan Sachs from
Roche, and since Bob said since you are here, we have a chance to ask
the question, and since this is about risk, and it is not about drugs,
but I am interested in your thoughts the fact that the FDA is not able
to regulate supplements and things like that, which actually have been
in the news in terms of risk, and I would be really interested in your
thoughts on that.

COMMISSIONER MCCLELLAN: Well, we actually do regulate supplements. We
just don\'t regulate them as drugs. We have a different statute, like
our food regulatory processes, has a presumption that they are safe, and
therefore, they can go to market without having to demonstrate safety
and effectiveness up front, and we have to prove a safety risk in order
to get them off the market, or in order to restrict their use.

And that is something that we highlighted back in February with respect
to Ephedra that we are undertaken and we have asked for comments, and in
a public comment period that just closed, for a review of all of the
evidence on Ephedra, which by far accounts for the largest share of
adverse event reports that we get about dietary supplements.

And we are evaluating that evidence from the comments right now on
whether the evidence is such that Ephedra as currently marketed presents
an unreasonable risk to the public health.

That is the standard under the statute and we have to demonstrate that.
So we will have more to say about that soon. So it is not that we don\'t
regulate it. It is a different regulatory structure, and it is one that
presents some different challenges than being able to ask product
developers to demonstrate up front that their treatments are safe and
effect.

The other important way in which we are moving forward on some
additional regulation enforcement action on dietary supplements is that
we published a new proposed rule last month that would impose some
national standards for manufacturing, national good manufacturing
practices for all dietary supplements sold in this country.

And I think that will give people a higher degree of confidence that the
products that they are buying were actually produced using good methods
without impurities or problems in storage, and so forth.

And it also includes new labeling requirements that the active
ingredients in a dietary supplement need to be listed on the label and
clearly identified. And this is also a good way to help give people
confidence that what they are taking is what is represented.

We would like and are trying to work on developing better evidence on
both the safety and effectiveness of dietary supplements. We have got a
partnership going with the National Center for Complimentary and
Alternative Medicine to help us with that, but I keep an eye and hand on
that.

And one other area of concern is to get them all out there, and that
many dietary supplements are making claims about their impact on patient
health that are misleading, and that are not based on good science.

And back in December outlined in a white paper a more comprehensive
strategy for enforcing the law against those claims. It is not legal to
make claims about the health effects of a product, and even the
structure function effects of a product, that are not based on good
science.

And you have seen a lot of enforcement actions where we have actually
seized products, and in some cases companies have shut down as a result,
where they were making misleading claims.

And I think you will see more of that. I do view it as a public health
threat that as we try to encourage people to do more, and to take
control of their own health, because that is the best way to improve the
public health in this country, through people\'s own actions and diet
lifestyle choices and the like, have a bigger impact than most of the
medication that we regulate.

It is a public health threat if they end up spending their money and
wasting their effort on products that are making claims about effects
that are just not right. So we will be doing more on that, too.

It is a different regulatory standard, and it is one that is more
challenging in some ways, but I don\'t agree that we can\'t do anything
there, and we are going to be trying very hard to improve things on the
dietary supplement side.

Thank you all very much for taking this much time in hearing some
answers that are a little bit afield of the topic. The topic is a very
important one, and we truly appreciate your input and your willingness
to work constructively with the agency both in this workshop, and I hope
going forward as we actually move into implementing some of these I
think quite promising new efficient risk management activities. Thank
you again.

(Applause.)

DR. GALSON: Thank you very, very much, Dr. McClellan. We appreciate you
coming. We are going to go on with our program now, and it may seem like
we are behind, but we are actually not, because we have in contrast to
what is in the printed program, we have three speakers in the next
section, rather than four.

So we are just fine and on time, and the first oral presentation in this
next session is Dr. Anshu Vashishtha, from Watson Pharmaceuticals, and
Dr. Vashishtha, if you could give us your talk now, and then when you
are done, go ahead and sit at the table, and the other speakers can
speak up at the table, too, whenever you want.

DR. VASHISHTHA: I thought Dr. McClellan wanted to listen on some views
to causality, but we will do that another time. So today I just wanted
to give some comments on causality assessment, and I think some of them
have been presented.

Some of the comments will be along the lines made earlier by other
speakers this morning. But as it was one of the questions that was
addressed to give the benefits, considerations, and causation
assessments, I think I will explain some of the pros and cons as I see
them for performing causality evaluations, and what criteria to
consider.

And beyond causality assessment, if we look at the data mining as a way
of computing causality, or causal relationship, some of the limitations
of that.

And then some proposals and thoughts on going forward in the area of
assessing risk, and relationships of risk to pharmaceutical drugs.

So I think we can break it up into as has been mentioned the causality
evaluation at the individual level, and at the case collection level,
and then clearly either way there is a very important rule of the case
definition.

That when we start looking at causality, the first thing we do is that
we confirm that what we are talking about is a report of the claimed
event. As an internist in drug safety, I have seen reports that claim to
be aplastic anemia, and which has turned out to be only anemia, and not
meet the case definition as my standard of criteria.

And again I think as some of the speakers earlier had mentioned, the
importance of case quality, beginning with good data capture, and
defining the case is really what the event being reported is, and what
it is stated to be.

I do see it as a necessity that we have to look at causality at some
level or the other. I think the question is whether to do it for every
case or to wait until there is a case series, or a signal, a potential
signal is detected.

And as has been outlined I think very well in the concept paper, it is
because \-- and in Dr. McClellan\'s remarks, the reported adverse
(inaudible) concomitant medication that are known to have that same
safety risk or are not known to have that same safety risk (inaudible)
in disease.

So I think we cannot validate from it because the current system is to
cast a wide net, and the consumer can call the company and say
(inaudible) and by the way does your title now cause it, and then it
becomes an adverse event according to current requirements.

So since the letter is some broad, just data mining or just counting
adverse events will give us a lot of false positives. I think that some
of the pros of doing a case assessment are that it can be to better case
quality and follow-up, because as we start trying to do a causality
assessment, we can think of the confounding factors, and we can think of
the follow-up needs, and get those questions addressed while the case is
still fresh and the data is available.

Also, it will enhance ongoing signal assessment by breaking down the
reports from just counting them as reports that may be incidental,
rather than to the categories, and ranking the categories, and we can
discuss the ones outlined in the concept paper work, as well as any
others.

But I think that those are some of the advantages of doing it. I also
think that, and again as Dr. McClellan also mentioned, that if you want
to make the package inserts more useful to prescribers and patients that
the assessment of causal relationship can provide a tool to do that in
keeping events which are incident reported out, and not considering them
as adverse reactions merely related to the drug.

And on the other hand, events which are assessed as adverse reactions
truly belong in the risk information on the label, and perhaps linked to
the management programs and could get in there.

The points of course have been pointed out that it indicates that it
does take time, and it does take resources, and again if we do it on the
industry side that it takes them on both ends.

It may be that there are some approaches which are not all the way down
the road of distinction between a probable and a possible, but at least
outlined in a case whether it is a substantial case or not, and the WHO
criteria which I will go into at least indicate as an example.

It is limited of course by a lack of all relevant data in several
instances, and it can be misused as was mentioned again in the morning,
that just because something is confounded does not mean that it is not
related.

I think that one has to take the whole picture into account. There is
also the issue from the industry side of what is the legal liability of
the sponsor who has assessed certain cases as possibly related, but not
made the decision to put them in the package insert.

The criteria that can be considered I think are well outlined in the
concept paper, and I do not need to review them all. We can just go to
the slides, and I think the CIOMS working group has indicated them.

Again, the most important being that if there is a rechallenge after the
drug, and as was mentioned in the morning even that is not a certain
one, and so as for most of these criteria, whether it is rechallenge,
dechallenge, class effect, biological plausibility, a lack of a
alternate explanation, or a typical ADR type event, like a
Stephens-Johnson, with a low background rate not related to drugs.

All those response factors lack further concomitant factors in a clean
subject patient, or a consistency of time to onset, or a high frequency
of reported cases of similar findings in doctors case studies.

Or perhaps positive individual or individual tests, like IG antibodies,
which are only available in a small minority of adverse events or
adverse reactions; or if there is a clear identification of an
identified subset, where there is a clear risk identified, and where the
reaction is related to the drug.

And again I think the protopathic bias has been mentioned earlier, that
a drug which is administered to treat a symptom may be temporarily be
associated to the illness which manifests after the drug is started,
even though the drug was started to treat the symptom of that same
illness.

And also it has been mentioned the issue of stimulated reporting, and in
particular I think for causality assessment when there is stimulated
reporting, there may be a recall bias of not recalling perhaps and a
bias towards false attribution towards the suspected drug.

One thing that I think that I would recommend to the working group is
that we consider the spectrum of causality assessment methods available,
and evaluate them in a way to go forward, and I think to some extent
that there are some literature reports which given the limitations of
these, as well as (inaudible) products ability back I think from \'68 I
don\'t think should be our guidance going forward.

I think we need to look at what the current status of these is. There
has been research on them and extensive use of them in Europe which has
continued after the FDA discontinued causality assessments from \'83.

And I think that it would be helpful to draw on this using criteria like
applicability, explicitness, explanatory, culpability, completeness,
biological balancing, and lack of constraints, and these are reviewed,
of course, in several writings on the topic.

So one of the methods that can be not all the way to categorizing it
into probable or possible is the literature of Dr. Edwards criterion of
using three index cases.

And where cases can be divided, I think, with relatively less time and
effort into substantial index case or feasible case, and a substantial
index case has information on the first six \-- on all the 11 items, and
there is a feasible case as on the first items.

So just having the items on each of these could be one way of
categorizing an adverse event report into a feasible or substantial
report which then can make the basis for a more detailed causality
evaluation.

And these sort of rely on the fact that for rare events the probability
of picking up three cases is very low, or in a short period of time it
is very low, and so that is the strength of that.

And just to mention some of the risk data mining approaches, where the
approaches are based on the proportion of the reporting rates, and some
published opinions that this remains as only one aspect of causality
assessment, subject of course to the usual biases.

And I don\'t think we can get away from global clinical assessment or
more case detail based approaches, as opposed to again counting and
doing status tests on events as reported.

And so in terms of the categorization into broad categories, I think the
one in the concept paper is one, and there is also this proposal from
the \-- based on the review of causality classification at

pharmacovigilance centers in the European community, which Meyboom, et
al., had published a couple of years ago, almost 10 years ago, into
Category A, B, or O, where there is either good reason and good
documentation to presume a causal relationship in Category A.

Or in Category B, where the connection is uncertain, and even doubtful,
perhaps because of missing data, and maybe because a case does not meet
substantial or case, or category, which is not assessable because of
missing or conflicting data.

I think that some sort of categorization into such broad categories
would at least make causal evaluation more efficient in terms of
assessing risk. So at the individual level, then I would recommend that
the working group consider the researching experience of the European
agency.

The French agency, for example, does require causality assessment by
sponsors for a long time, and so that there might be some data there to
base our decision going forward on as we try to get evidence based on
this, and not just go by our impression of the utility or lack thereof.

For the case collections, I do think that it remains necessary for
signal evaluation. Again, the working group may consider really looking
at the different alternative approaches and perhaps testing them on the
FDA database.

It remains important to define a case for the cases confirmed and it is
of good quality, and algorithms can be evaluated I think in different or
on a worldwide basis, perhaps categorizing into these broad zones,
either A, B, or O, or something that is more amenable to perhaps less
label intensive, or medical review intensive methods.

But nevertheless lays a foundation for a fully detained risk assessment,
and perhaps these can be compared by really doing a retrospective
evaluation for the utility of these methods using the existing data from
the FDA database.

So just seeing if the two have been applied what would we have found,
and comparing maybe some of the tools that are used in Europe, and see
if that would have been helpful in identifying the risk which we now
know in an earlier way.

One may consider a pilot on certain drugs and in certain instances, and
one can then determine the overall guidance based on the results of the
assessments. Thank you very much again to the whole working group and
the FDA for this portion.

DR. GALSON: Thank you very much. Our next speaker is Gregory Gogates, or
Gogates.

MR. GOGATES: Good morning everyone. My name is Greg Gogates, and I
represent CRF Box. We are a clinical data collection company using
remote data collection tools, namely PDAs and cell phones. So I thought
that this would be a good venue to sort of tell people what is going on
in the world as far as the status of collecting this type of information
using portable data collectors.

Okay. The overall overview I think, and everybody has said that a
million different times, but I guess the thing that I am trying to state
here is that some of this data that we are talking about collection with
AE data could certainly be done using PDAs and mobile phones.

One of the things that we are talking about doing is getting this
information through the doctor-nurse and all the different organizations
that exist, but one thing that I think we are missing is direct
collection of data from the patients directly into databases and this is
happening today.

So I am going to discuss some of the pros and cons of doing this, and
show you a case in which and where this is actually happening right now
in Europe. So the typical compliance that we have out there, and this is
from some ad hoc interviews that I have done with some physicians and a
few people in the industry, and we have relatively good compliance with
the clinical trials, hospitals, and pharmacies.

And I think the answer to that is typically because these people are
salaried people, and they are being paid to be in their office all day
long. So if they have to answer a few forms, that is just part of their
daily jobs.

So you have relative good compliance. Now, the quality of the data, and
other people are talking about the quality of data, but I think we have
good compliance there, because it is part of their job.

Now you move into the citizens, or you move into the private physician
area, and citizens don\'t necessarily know that this avenue exists, or
know and don\'t want to bother. I think that is what you are finding
there.

And you have private physicians who say, well, I am being squeezed by
all means. I am a broke, fixed type of a person, and a patient comes in,
and I fix them, and they leave the office, and what is my incentive to
fill out all these forms.

And I think you find that a lot of times they say I am not going to
bother. I fixed my problem, and I am going to move on to the next
patient. So by its nature, it is voluntary on that avenue, and I think
that we can improve some of that by making it a little easier.

So you have your traditional MedWatch methods using the 3500A and the
3500 forms, but on the 3500 forms, it is all not mandatory. So there are
some custom on-line tools which I believe have been discussed a little
bit earlier that do exist, but they are not or they don\'t travel home
with the patients.

So what I am saying is that maybe we should consider getting some of
this information and making it available for the patients to input this
information from their homes.

So the current challenges of course is to get this collected and report
this spontaneous patient level safety data per the CFR as referenced.
And of course we want to get the timeliness of the data that has to come
in there, and so we want to make this data more timely.

And of course we have to reside within the HIPAA regulations, and I
understand that AES are pretty much \-- that they don\'t have to worry
about HIPAA, but at the same time, we still have to worry about some of
the fringes, because it really is not that well defined.

So we have to get increased public awareness of the public safety
concerns, and make sure that they understand that there are concerns
with these products, and as we said before, these products aren\'t
perfect. We understand that there are risks in taking them, and that
people have to understand that.

So how do we respond to these things? Well, I think the key to this
whole thing is spontaneity. If you don\'t get it up front, you don\'t
get it at all. So let\'s use some electronic tools.

Some of these tools as you see up there on the screen, you have PDAs,
and the PDAs right now, you can buy wireless ones that are actually part
also cell phones, and you can buy them with cradled modems so they can
come up with data.

You can also use normal cell phones, and there are of these more
expensive cells phones, but the bottom line is that this information
could be brought directly into data bases, and viewed on-line by anybody
who has an access, and it can be stored centrally for review by anyone
who wants to look at it. It is very simple.

Implementation issues. Patients can use the proprietary electronic
devices; i.e., the mobile phone. So we don\'t have to give them
anything. They already own a mobile phone.

Mobile phones not only can talk voice, but they can also transmit data,
and the SMS messaging system, which is on most of everybody\'s cell
phones here in the audience, is usually not that well used in the
States, but it is very pervasive in Europe and the rest of the world.

But if you have a budget, if this happens to be a trial, and there is
some budgets, you could give them a proprietary device, say a PDA, or
something else to collect data with. But, in reality, I think the mobile
phone is probably a better venue for this.

So the only thing that is recorded is that you could record the
patient\'s cell phone number at the point of prescription. So you could
say, okay, Mr. Patient, I am giving you this drug, and would you like to
volunteer to provide some information on your experience with taking
this product.

If the man answers yes or if the person answers yes, you collect their
cell phone, and then a week later, or two weeks later, or a day later,
or whatever frequency you want, the system would call up the person\'s
cell phone and say, hello, this is the drug company, this is the
pharmacy, how do you feel.

Have you had any experiences with this drug. They could ask 3 or 4
questions and you answer the questions, and there is no voice going on,
and they don\'t have to worry about doing it. It is not intrusive at
all. It\'s just that your phone beeps, and you look at it, and if you
don\'t want to answer the question now, you answer the questions later.

You just push the buttons, and you say have you had any diarrhea,
yes/no, enter. Have you had any vomiting, yes/no, enter. Headaches,
whatever your questions want to be, you enter the information, and it
goes right back to the database, and it is immediately available for all
people to view.

So these periodic safety questions can be sent and answered and sent
just as I mentioned. There is text space, and there is no voice
involved, though there is one side benefit of having the voice.

If the right question is answered yes, then there could be a follow-up.
Someone could call this person back and say hello, you answered yes on
this question. We need some more information. So it would be nice and
straightforward, and very simple to manage.

Pros and cons. They don\'t necessarily need to visit the physician. They
don\'t need to get involved. It just shows up on their cell phone. And
the sponsor and anyone else who wants to, has the ability to see this
information in real time, and they can combine it with any other safety
data that they need.

So the timeliness is improved, and the questions can be changed on the
fly. So if you decide that the questions being asked aren\'t correct,
you can tell the system to ask new questions.

So the cost, which is the cost of this technology, it does add to the
budget, which is a deficit. But if you are using the patient\'s
proprietary cell-phone, you don\'t have to buy their equipment. You just
have to deal with the server side.

So don\'t get me wrong. I am not saying this is an end-all, but it
certainly could be used in some situations. You may have the disabled,
or the elderly, or someone who doesn\'t have a cell phone, or maybe
something that just wouldn\'t make sense, and of course you wouldn\'t do
it.

But this is a system that could be used in a lot of areas. So there is
detail as I said earlier, and there would be detailed analysis that
still would need to be conducted with the physician or the nurse, but in
this case at least you would get that up front spontaneous information,
which everybody understands is the best way to get the information.

There are some examples out there that I can state that our company is
doing at this point. There is a Phase III trial going on in Europe at
this time, and there is 12,000 children across Europe, and at 75
different sites, and they have been given a vaccine, and this is a
situation where they need to collect data for 2 years every 4 weeks,
from 12,000 parents.

And this would be a phenomenal expense, and so we came up with this idea
of collecting the parents\' cell phone number, and then every 4 weeks
the system calls the parents\' cell phone.

They get a beep, and it says they have received a message that says,
hello, this is the study and how does your child feel. And the
responses, there are maybe 4 or 5 questions, and they answer yes/no, and
the system answers back, thank you, and I will call you back in four
weeks.

Now, we are not getting a hundred percent response out of this, but at
least we are taking at least a 90 percent response on this information,
and now the study nurses only have to call up 10 percent of those 12,000
people.

So it is a significant cost savings in doing all of this follow-up work,
and there is no reason why this technology, or these concepts, couldn\'t
be moved into the AE realm of people taking medicines, and just taking
it from their physician or their pharmacy, and not necessarily in a
clinical trial.

So in conclusion, basically I think that I would like the organization
to consider using mobile phone infrastructure. It is in this country,
and it is global, and it is very simple to use, and it would be a very
good venue to collect patient data.

I would say that in Europe, and in the rest of the world, you have very
good \-- just about everybody and their brother, and children, all have
cell phones, and it is becoming just as pervasive here in this country.

As I said, the EU system are mature, with good inoperability, and one
issue that we do have in the States is the inoperability between cell
phone providers is not perfect, but it is increasingly getting better.

So considering that this is going to be a long term permanent effect,
that this is something that we should consider. And that is really my
simple message that I have to say. Thank you very much.

DR. GALSON: Thank you very much. Our last speaker for this morning is
Linda Hostelley. Linda. And she is going to tell you who she is speaking
for.

MS. HOSTELLEY: Good morning, ladies and gentlemen. I am Linda Hostelley,
the executive director of worldwide product safety in epidemiology at
Merck Research Laboratories.

I would like to first thank the FDA panel and the entire working group
for the opportunity to provide public comment today on behalf of the
Pharmaceutical Research and Manufacturers of America, concerning concept
paper number three, risk assessment of observational data, good
pharmacovigilance practices in pharmacoepidemiologic assessment.

As mentioned previously over the past few days, PhRMA supports FDA in
the following concepts. We support FDA\'s efforts to provide guidance
for developing pharmacovigilance in risk management plans.

We support the concept that risk management is an ongoing process
throughout a product\'s life cycle to optimize the benefit risk balance.
We agree with the agency that the ultimate goal of any plan in its
evaluation is to ensure that the efforts and costs involved in an RMP
are expended on effective processes that achieve a positive benefit risk
balance.

PhRMA believes that risk management programs and interventions should
balance access to drug with level of concern. We agree with the FDA that
decisions which are made concerning individual drug benefit and risk,
and RMPs, must be based on scientific evidence.

We believe that any risk management program should rely on systems based
interventions, and specific tools should be used consistently across
RMPs. We recognize that efforts to identify all risks prior to drug
approval may not be feasible without delays in drug development.

And we recognize that collaboration among all the stakeholders,
including FDA, the academic institutions, help care providers,
pharmacists, professional organizations, and patients, is absolutely
critical to achieve significant improvements in the benefit risk
balance.

My comments today will focus on the first three questions posed by the
FDA in concept paper number three concerning good pharmacovigilance
practices. Dr. Stephenson will address the questions on
pharmacoepidemiologic assessment during the afternoon session.

The first question. How can the quality of spontaneously reported case
reports be improved. PhRMA supports FDA\'s efforts to improve the
quality of individual spontaneously reported cases which serve as the
foundation for identification of safety signals, and are the critical
building blocks of a post-marketing safety surveillance system?

We recognize the importance of obtaining complete information for these
reports, but recognize that a balance must exist between the
significance of the clinical event reported, and the intrusiveness of
follow-up with treating health care professionals.

The more onerous, intrusiveness, or threatening the follow-up, the more
likely the number of reports will diminish. Such an outcome is exactly
the opposite of what is needed.

It would be useful to all stakeholders if the agency provided a
prospective on what aspects of case reports, what sources of reports,
and what sort of follow-up has been most helpful in signal detection and
regulatory actions.

It would also be useful to know that where there is waste in a reporting
system, and without a formal study of these questions, it is speculation
as to how to improve the quality.

The provision of adequate clinical information is a shared
responsibility with all stakeholders. PhRMA supports the education of
all partners involved in post-marketing safety surveillance to improve
the understanding of the need for reporting and the quality of
reporting.

To that end there should be education of practitioners to communicate
high quality data with specific clinical details, and education of
patients to recognize signals and report specific details.

There is also a need to increase awareness of the importance of
pharmacovigilance in clinical pharmacology for health care professionals
during their medical, nursing, or pharmacy school curricula.

Education of these health care providers on a continuum throughout their
careers will improve the understanding of the need for reporting and the
quality of reporting.

PhRMA and FDA should continue to partner with medical, nursing, and
pharmacy schools to increase the awareness of the importance of
pharmacovigilance in clinical pharmacology.

Partnership with health care organizations and academia is also needed
to reinforce these principles. This effort to improve report quality
will also benefit from continued collaboration with the National Patient
Safety Foundation.

Question Number 2. What are the possible advantages or disadvantages of
applying data mining techniques to spontaneous report databases for the
purpose of identifying safety signals?

One of the greatest challenges in post-marketing pharmacovigilance is
determining what constitutes a safety signal. Post-marketing adverse
event databases, which are based on voluntary adverse event reporting,
lack formal controlled data, and exposure data, and until recently have
been difficult to use objectively and efficiently for signal detection.

Traditional pharmacovigilance methods have consisted primarily of
individual case review and empiric analysis of crude frequency data.
During the past 5 years, tremendous progress has been made in the
development and application of statistical methods for systematically
screening post-marketing adverse event databases to efficiently identify
adverse events that occur at a higher than expected frequencies in the
absence of exposure data.

Examples of these data mining or disproportional analysis methods
include the proportional reporting ratios developed by the U.K.
Medicines Control Agency, the multi-item gamma poison shrinker,
developed by William Demichele, of AT&T, in collaboration with Anna
Szarfman, of the FDA; and the Beijiang (phonetic) Neural Networks,
developed by the WHO.

Recently these sophisticated tools for data mining have become more
available to regulators in industry through packaging and deployment of
modern easy to use web-based interfaces.

Data mining may therefore become a useful adjunct to traditional
pharmacovigilance methods, and that it provides a systematic, efficient,
and objective approach to screening large adverse event databases for
safety signals.

It may be especially useful for detecting safety signals involving rare
events that are unlikely to be observed in clinical trials or
observational studies. These rare events may also allude detection by
traditional pharmacovigilance methods.

Other potential applications include screening for signals associated
with polypharmacy. For example, drug interactions, and for demographic
factors associated with adverse events.

Data mining analyses compliment data from well-designed, well-monitored
clinical trials, and observational pharmacoepidemiologic studies, and
can be used to generate hypotheses that can be further tested in such
studies.

Furthermore, in some instances, the information gleaned from data mining
can be used to enhance risk management and clinical development
planning. The limits, however, of the underlying data and the data
mining techniques must be fully appreciated to avoid false positive
causality conclusions with costly interventions and inappropriate
product restrictions.

Research efforts should be targeted to reduce false positives. To this
end a collaborative working group of statistical and medical experts
from both the industry and FDA was recently formed.

The goal of the working group is to define best practices for the
application of data mining technology that will optimize use of these
tools in the areas of pharmacovigilance and risk management. The group
will seek to better understand the comparative performance
characteristics of various data mining methods.

The optimal specifications and configurations for various uses. Data
quality issues and the limitations of these methods, particularly as
they affect the interpretation of the results.

It is also hoped that this joint effort will provide an opportunity for
FDA and industry to develop a common language, and share systematic
approaches to the detection and assessment of signals from
post-marketing adverse event data.

Question Number 3. What are the possible advantages or disadvantages of
performing causality assessments at the individual case level? Given the
inadequacy of spontaneous report data, the application of causality
algorithms to a single case is fraught with misinterpretation.

The ability to rule out the likelihood that the suspect drug may have
contributed to the adverse experience in most instances becomes
impossible. Therefore, most adverse experiences at the individual case
report level end up with a possible association.

With the exception of cases involving a positive rechallenge, there is
little or no advantage in performing causality assessment on individual
case reports.

In addition, a lack of consistency among experts in the evaluation of
causality assessment for individual cases strains the already limited
resources.

Although a series of cases may be used to generate hypotheses concerning
the association between an adverse experience and drug exposure, there
is no methodology determined to date that is reliable and reproducible
for individual causality assessment.

Thus, causality assessment at the individual case level is open to a
high likelihood of misinterpretation. In summary, PhRMA supports the
development of best practices for improving the quality of spontaneous
reports and for developing the methodologies for application of data
mining techniques.

PhRMA also supports education of all partners involved in post-marketing
surveillance. In closing, PhRMA would like to pose an additional
question for the FDA. What criteria should be used to determine whether
a pharmacovigilance plan describing pharmacovigilance efforts above and
beyond post-marketing spontaneous reporting is warranted.

Thank you for this opportunity to provide comment on behalf of PhRMA.

DR. GALSON: Thank you very much. We now have about a half-an-hour for a
discussion session and question and answers, and if all the six speakers
from this morning, the six presenters, want to sit at the table, that
would facilitate this.

I don\'t know if everyone is still here, but we have got six seats, and
there were six of us. And so we will take questions from cards and
people are walking around to pick up any other questions on cards.

People can come up to the microphones and we will just have a free
exchange between those of us up at the front of the room and people who
want to come up and talk. I think I will just start with a question
while people get themselves going.

And this is one that came in on a card and it says please address the
effects of MedRA on label and safety signals, and I think that is one of
our working group members, Min Chen, is going to address that.

DR. CHEN: It is the ADR proposed rule that requires all adverse events
in the case reports coded in the most recent version of MedRA.
Currently, however, the ADR individual events, or the organ class in the
drug labeling has no specific guidance as to what to use.

But in a safety signals detection in post-marketing, we use the measure
terminology to store and to retrieve clinically irrelevant cases in the
databases.

Once risk is identified, currently it is the clinical or medical concept
of the identified risk information that will be considered for
incorporation into the labeling.

In the future, I think there may be some considerations in the agency to
have a consistent labeling on the specific ADR information in the
labeling using MedRA terminology.

But I think that still has some way to go, because the clinical trial
data should be coded in measure first before they use the information in
the proposed labeling. So that is the current situation right now.

DR. GOLDMAN: Are we allowed to comment on that?

DR. GALSON: Yes.

DR. GOLDMAN: Okay. A couple of principles if I may, and I say this to
the other people who basically helped develop the measure sections. The
tail should not be wagging the dog. Labeling is designed to convey
clinical concepts.

And there have been several studies done, such as work Ali Brown
(phonetic) has done, and been published, and others. That you could
actually have terms that when the MedRA version changed, and went from
labeled to unlabeled, and I have \-- I am putting out a cautionary note
in public, and as we have done in print, that MedRA was not designed for
that in particular.

And that that has to be borne in mind when we talk about labeling,
particularly when we would be coding measure in terms of clinical trial
data, and I just wanted to have that point.

DR. GALSON: Okay. Let\'s move to the mikes now on this side. Please
identify yourself.

MR. STANG: Paul Stang, Gold Associates. A couple of general comments.
One is that when you read through the document, although there is
separation between signal detection and evaluation, I think it is very
important to make that absolutely crisp and clear, because I think that
a lot of people confuse signal detection with signal evaluation.

I think that there are just many opportunities to clarify that. A second
comment has to do with data mining, and that is whether or not we have
progressed to the point where we have complimentary rules for when we
don\'t find a signal, inasmuch as when we do.

And the point here is that the fear would be that there would be
multiple looks at the data many times over time, and without proper
statistical adjustment, or other ways of accounting for the fact that
you keep going over time looking for something, and eventually you will
find it, is my sense if you look enough times.

So I think that this is just a plea to, as others have said, to develop
some methodology around it. The third point is that I work with a
non-profit group of physicians, a grass roots organization, and we
developed just on a whim a self-training module in safety surveillance,
just because we had heard from a few of the 10,000 members that they had
an interest.

And we developed a little CD-based training system, and a thousand of
them were handed out \-- well, actually, people had to request them, and
all thousand of them went in two weeks.

So there is great thirst for this information at the primary care level.
It is a primary care based organization, and I actually shared an early
version of this with members of the agency.

So I do want to point out that primary care is screaming for this, and
that as someone has pointed out, they don\'t get this kind of training
in medical school, and they certainly don\'t get it subsequent to that.
They themselves are looking for more information.

And finally along those lines is the training for the actual consumers
of this information, and I specifically have been struck in a couple of
meetings that I have attended with the agency with their external
clinical advisory boards for various products that are coming up under
review post-marketing for certain issues.

And I think that there is a great opportunity to educate some of the
external clinical experts that are brought into these meetings along
safety issues, and the resources that are available, and with what their
strengths and what their limitations are. Thank you.

DR. GALSON: Thank you and I think I will turn to Dr. Beitz now, who had
a question.

DR. BEITZ: I had a question actually that I would like to direct to some
of our industry representatives here today. And it has to do with the
application of an FDA newsletter of adverse event drug combinations.

And there is really some sensitivities around publishing such a
newsletter, and how would we choose which events and which drugs might
be included in a particular issue.

And I would just like to get a sense of how people would think about
publishing on particular adverse events or combinations even though it
has not been totally worked out as to causality. Labeling may not be
fully worked out or discussed, or negotiated.

Do people feel that a newsletter approach could or could not occur in
parallel with the more traditional labeling negotiations, or does
something have to be in labeling first before folks in industry would
feel comfortable about FDA putting it in a newsletter?

MS. HOSTELLEY: Linda Hostelley, representing PhRMA, and perhaps others
from the organization will also comment, but I would say that this is an
interesting concept, and I think PhRMA would certainly welcome an
opportunity to have further dialogue on this topic.

Again, you know, the concerns from an industry perspective would be, you
know, how much can be said before an actual label is constructed, and of
course there are the legal implications for us as a regulated industry.

But certainly it is an interesting concept, which I think would warrant
further discussion.

MS. STEPHENSON: Can I add to that?

DR. GALSON: Sure.

MS. STEPHENSON: Wendy Stephenson from Wyeth. I actually \-- my first
impression is that a newsletter of that sort would be very problematic.
I think it would be much better to have fully vetted the issue, and
evaluated data, and come to a conclusion, which then would usually
result in a labeling change, or result in the conclusion that there is
not an issue.

And then perhaps it might be reasonable to publish. We have had some
very sort of strange experiences with a similar type of newsletter that
the Upsalla monitoring center of WHO has issued.

They generally issue these reports of signals \-- they are called
signals \-- and then after the publication, they will ask the company
for a comment. And in every case where we have had one of those signals
referred to us, it was a completely spurious association, and there was
generally a pretty good explanation for what they were seeing.

And there was no causal association. So I would just caution against
publishing preliminary information before it has been fully vetted.

DR. VASHISHTHA: Anshu Vashishtha from Watson. I think another issue that
of course comes up is the issue of the importance of sequence reporting,
the stimulated reporting, and the recall by issues that have come up
with such a publication.

I think that is where we have to clear the package inserts and they have
professional letters and other ways to have established lists
(inaudible) risks.

And anything done prior to that in my mind to the general public would
distort the evaluation of that risk.

DR. GALSON: Alan.

DR. GOLDHAMMER: Yes, Alan Goldhammer, PhRMA. I would echo Dr.
Stephenson\'s comments on this. I think that you can go back and look at
the literature being rife with examples of preliminary assessments of
risks, which later are disproved.

Unfortunately, companies in many cases have incurred significant legal
bills defending themselves in court. I mean, we would be more than happy
to look at the proposal, but I am not terribly optimistic that the legal
side of PhRMA would be encouraged by it.

DR. GALSON: Okay. We are done with follow-up on that question. Oh, one
more. Okay.

DR. NELSON: I was going to comment that what could possibly be more
valuable is after the labeling change has been agreed to, have a
publication that explains the evidence base for that labeling change.

DR. GALSON: Good. Okay. I think we will move to the next mike question.

MS. OMINOFF: Okay. I am June Ominoff from the directorate at
GlaxoSmithKline, and I am also the chair of the PhRMA/FDA working group
on safety evaluation.

And I am actually up here to respond to Dr. Faich\'s comments, and
concerns about data mining. We thank him for his comments, because they
do highlight some important caveats about the tool. However, I want to
make two points.

The first is that if we adapt Min Chen-Julie Beitz\'s, the working
group\'s definition, of a signal which is, quote, apparent excess of
adverse events associated with the use of a product, then what could be
more valuable then a disproportionate analysis or data mining tool that
objectively applies to statistical methods to identify these excesses.

My second point is that he addresses examples of false positives, or
what he calls false positives, and I wanted to say that no one who uses
this took responsibly would ever, ever draw conclusions without
reviewing the medical case information using standard pharmacovigilance
practices.

So in my view, the examples that Dr. Faich has presented as false
positives really seem to reflect an intermediate stage in the evaluation
of a signal. Thank you.

DR. GALSON: Thanks a lot.

DR. GOLDSMITH: I would like to address the same issue with regard to
data mining. My name is David Goldsmith, and I am a consultant, but I am
representing myself. And the issues that I would like to raise are
actually on both sides, and then follow it up with a question.

First is the issue of the false positives, and I would like to present a
rather simple scenario that gives an example of a false positive. You
have Drug A, which is a recently developed drug, and it is effective. It
is so effective that it works in a hundred percent of the patients who
are treated.

It has a very clean profile and indeed after being marketed for 6 months
there are a hundred-thousand patients exposed, and there are 10 serious
adverse events that are reported, nine of which represent migraine
headache.

The other drugs in the class that are used to treat the disease are not
terribly effective. They are only about effective in 50 percent of the
cases, and they are very dirty drugs. And they actually have lots and
lots of serious adverse events, and they are, for example, in that same
6 month period with that same sales ratio 500 cases of migraine
headache, and 500 cases of other serious adverse events.

And using a proportional reporting ratio, we come up with a proportional
reporting ratio of 17, which far exceeds the three that one would
normally be seeing, all right?

And therefore would potentially represent the signal. Given that people
would very likely look at the denominator effect, which is clearly
evident in this, and one would assume that the signal is probably an
erroneous signal.

But at the same time when does one turn off the system in order to stop
looking for a signal because this was a false signal. That is my
question.

The second point is that there are false negatives associated with data
mining, especially as it relates to the utilization of MedDRA. I
presented a paper at the annual DIA meeting last year which looked at
two specific conditions.

One was member proliferative glomera nephritis, a rather rare
occurrence, but is associated with heroin necropsy. The other was TTP,
again a rather rare entity, but is associated with some of the
anti-platelet agents.

In neither of those cases could you use either the MedDRA term itself at
the low level, preferred term, high level, high level group term, or at
the SOC level, to make sure that you had all of the cases and eliminated
appropriate cases that did not belong there when you had appropriate
case definitions.

So one really needs to be very, very careful about the data mining
considering, one, how has the terminology been put in, and two, what do
you do with false positives, and how do you turn off your risk
management program when a false signal is identified.

DR. GALSON: Who would like to address that?

DR. SZARFMAN: Can I say something? May I \-- I brought a computer, a
laptop, and we can look at the results that data mining show for the
specific examples, and then let\'s see how data mining performed,
because we need to look at the data.

DR. GALSON: So you are going to do that in the back room?

DR. SZARFMAN: Yes.

DR. GALSON: Wow. Full-service FDA here.

DR. GOLDMAN: I would like to make a comment. Steve Goldman.

DR. GALSON: Yes.

DR. GOLDMAN: We have been hearing very good points of view obviously
from Dr. Goldsmith, and Dr. Szarfman, and also Dr. Faich. I am going to
do the same cautionary note that I am going to say about MedRA.

This is a technique. This is a tool. It does not replace the greatest
tool of all, the human brain. It does not take away what we have learned
in 50 years of pharmacovigilance and how we look at each individual
case.

And I think if I may point out that if we can take down the level of the
heat and show a little light that this is another tool that we are using
to look for possible signals to use the dynamic database, such as the
AERS database, as to look for possibilities.

And I think that in that sense I think we have more of a consensus than
we have disagreement about how the actual application should be.

DR. GALSON: Dr. Faich, did you want to contribute something to that?

DR. FAICH: Yes.

DR. GOLDSMITH: Can I follow up?

DR. GALSON: Wait just one second.

DR. FAICH: Can I just say that I don\'t want to be misunderstood. I am
not opposed to data mining, and I think it is an interesting technique
and a research tool that is in development. So I think that it does have
utilitarian interest.

The second point that I would simply make, and I was trying to make it
by giving that Yogi Berra comment earlier, was that most signals I
believe will come from alert safety monitors either at the FDA or at
companies who are at the intake end of important reports.

And that the main triage tool is to look for a serious unlabeled. It
doesn\'t help very much for a serious labeled increased frequency, and I
think we all know that, and I believe that is one of the main utilities
that data mining can help us with.

I also would like to point out that Lou Lisana (phonetic) many, many
years ago said that the most important source of new information about
safety for drugs is the alert observant clinician who sees something out
of kilter.

And I think the same thing is true for a safety monitor inside the FDA
or perhaps maybe more importantly inside a manufacturer who has launched
a new drug and now receives reports that he didn\'t expect to receive.

The issue of whether false signals will occur or will be lessened
because those who use this steel-edged axe are indeed sophisticated and
will go back and review the data is dependent upon whether they have the
data in-hand or not.

My fear is that decisions or alerts will be made based on only the
electronic data, and not a more complete clinical review of individual
cases once a preliminary finding has been found.

That is exactly what WHO has done, and that is exactly what happened in
the case of Claritin that I cited, and that is exactly what happens when
attorneys troll through the database and do crude case count, such as
prozac and suicide.

DR. GOLDSMITH: Just a follow-up to Steve\'s comment and also to
Gerry\'s. I think that it is very important that we keep our eye on the
ball and watch what is happening to the signal to noise ratio, and we
are increasing to a large extent the amount of noise.

Yes, we are increasing the signals, but we are increasing the noise to
signal ratio so much that we won\'t be able to find those signals
terribly effectively.

And the same thing is true, for example, of the data that Steve
presented. Yes, there was a 17-fold increase in reporting following the
Rhode Island program, and there was a 9-fold increase in serious
reporting, but 17 versus 9, that seems that there was a lot more of a
bigger increase in the non-serious.

DR. GOLDMAN: I will respond to that. There was also 31 previously
unsuspected unlabeled serious events that were reported. I will stand
with having to make the choice between getting more possibility of drops
into a system if it were to enable me to get things that I didn\'t
previously know about, and that is my philosophy in terms of that.

DR. GALSON: Okay. Let\'s move on. Next, Dr. Szarfman.

DR. SZARFMAN: Well, I will try to answer some of the things that were
said here. First of all, data mining is a tool. It is not the only tool.
There are different tools. Data mining is a tool, but we have

\-- I don\'t think that there is a chance again that we are having this
meeting at the National Transportation and Safety Board.

The human brain is very important, but we have and we not always
remember to fasten our seatbelt and we need a light to let us remember
that we need to do so; and to airplanes that are maybe too close, the
human brain, of course, you know, of course \-- you know, we might
compare that with other things that we need to be alerted.

And this is essentially what we are trying to do. Data mining is not
data dredging. We are applying one statistical method to the whole
database, and there is not a problem of multiple logs, because using
this method, you can do multiple comparisons.

And to include all of the contents that you need to make the assessment.
We don\'t have backgrounds, background information for specific
diseases. And it will take a long time to get that part down.

If we can analyze every single drug used to treat that indication, and
then it will keep us informed as to which level of signals you have for
a certain indication.

We have also the human brain, and we have in the database over 50
quadrillion combinations for drugs. This means a 5 with 16 trailing
zeros. I don\'t think that the human brain can efficiently keep up with
these.

You know, if you don\'t have the gold standard for these, and what the
profile of every document, and we don\'t. You know, that\'s why we need
systematic approaches.

DR. GALSON: Okay. Can we move on to the next. Thanks, Dr. Szarfman. I
want to just use one of the questions that came in on a card. We have
heard the call to capture safety data electronically.

How can this happen with the industry pushing back on Part 2 of HIPAA,
and other security issues are implied in this question as well. Do one
of the members of the working group want to address that?

MS. CIAMPA: Well, first of all, I guess\--

DR. GALSON: If you could identify yourself, please.

MS. CIAMPA: I\'m sorry. I am Aileen Ciampa and I am with the Office of
Regulatory Policy in CDER. First of all, I would just note that with
regard to Part 11, the commentor might be aware of this, but FDA did
just recently issue a guidance document regarding our approach to Part
11.

And in trying to make sure that it is consistent with the adoption of
new technologies, and that is an ongoing process. And more broad based,
I guess I would just note that with regard \-- and I am not a HIPAA
expert, and so there might be people here who can speak better to that.

But I would just note that both patient privacy concerns and concerns
for data integrity and record integrity have always been present in the
collection of safety data and the transmission of safety data.

And so although I think the movement to the electronic environment might
have implications for new regulatory schemes, and might pose additional
concerns, that the fact that those concerns, and particularly the
patient privacy and data record integrity, should not prevent us from
moving and adopting new technologies.

DR. GALSON: Okay. Anybody else want to address that issue?

DR. VASHISHTHA: A new issue.

DR. GALSON: Any other questions on the security and those related issues
before we move on to something else. Okay.

DR. VASHISHTHA: Just a question for some of the members of the working
group possibly. One is that there might have been already done some
evaluation of the algorithms and causality assessments used in Europe or
globally, and secondly, whether if data mining is only one of the tools,
then defining a signal as an apparent access of adverse events in
association with a product sort of biases us towards more of a data
mining approach.

What about redefining a signal as an apparent risk associated with the
products? Any comments on that from the working group members?

DR. GALSON: Dr. Chen.

DR. CHEN: We are aware of many publications and many individual\'s
proposals for the criteria of causality assessment, and that we have not
found a consistent way that can help us as a standard or a guideline to
use currently.

So we are exploring all these possibilities and getting inputs from you
all. I think that your suggestion was wonderful, to look at the European
ways, and if that is something that most people think is applicable to
us, we certainly will consider it.

And the second question was about the signal difference?

DR. VASHISHTHA: (Off microphone) on the first question, also about using
the existing database to perhaps evaluate the appropriateness of some
algorithms with respect to that.

DR. CHEN: I think that we are all exploring all possible tools helping
us out to identify the signals, and human brains are limited in some
ways, but I think that in the process that we like to prioritize
different kinds of signals that we like to see as a work-up first.

So I think that is something that is only probably reasonable for such a
big volume of information coming into the agency every day. So our
priorities usually are the more serious ones, like on liver failure, and
severe skin reactions, and such as all of the events mentioned in the
medically significant events, and we pay more attention to that.

And data mining, as we said before, is an adjunct to that to help us out
to identify signals that human eyes cannot detect right away. As far as
the definition, I think we are open to suggestions on access to a
certain degree, and nobody knows when for a new drug that just came on
the market with new populations, and that we have never seen before,
maybe one case of a serious event is worth looking for and looking into
a further investigation.

But for other drugs, you probably need a lot of events to support the
hypothesis that the drug really caused that. So I think this all needs
to be considered in here on how much is access and what is the threshold
for a signal detection.

DR. GOLDMAN: There is a very famous law called Sutton\'s Law; you go
where the money is, and with the always expedited reports under the new
proposed rule, with the designated events which triages errors, we are
talking about scarce resources, scarce time, and yet public safety.

And the idea that we look for the most serious events \-- and again the
concept of risk, well, that entails benefit risk. When you are looking
at signals, I think we should be looking at \-- my philosophy is are the
actual events themselves are the ones that are the bad actors, the ones
that we know about.

The ones that tend to be lethal, in particular, and then tie that on in
terms of that. I gather, Min, that is the philosophy?

DR. CHEN: Yes, I agree.

DR. GALSON: Okay. Let\'s move to the question over here at the mike.

MR. HARVEY: Yes, in a complex area where we are trying to move the ball
forward in a very, very unknown way, I think it is often \-- I\'m sorry,
Bill Harvey from Life Free Technology.

It is very often worthwhile to keep things as simple as possible. And
the confusion between large simple safety studies and Phase IV studies
has been very difficult for me to comprehend, and especially in the
context of a statement that was made on Monday from Dr. Paul Stolley,
who said that 13 percent of Phase IV studies have been completed.

And implying that fully 87 percent of those studies have not been
completed, and given the expertise in this room either from PhRMA, from
the FDA, or from others, I think that is worth discussion.

DR. GALSON: Do you want to form it into a question?

MR. HARVEY: What is going on?

DR. GALSON: Who are you asking?

MR. FENISHELL: May I speak to that? This is Bob Fenishell, now a
consultant, but at FDA for a long time. We looked into this in the
division when those data were being gathered, and so we went back and
looked at all of the Phase IV commitments that we had gathered from
sponsors, and then looked at indeed what fraction of them had been
completed, and certainly the figure of 13 percent is consistent with our
experience.

It was not a large fraction, and then we said, gee, how do we feel about
this, and just looking at the raw numbers one must indeed ask as the
questioner asked what is going on.

But in fact you start looking at one of them after another, and you
know, that was a pretty stupid thing to ask for. And that other thing,
you know, that was a pretty good thing to ask for. That would be really
interesting to have.

But we now realize that it is just impossible to collect that data, or
you know, that was probably a pretty good idea at the time, but it has
been superseded by events.

We no longer believe that that is a significant problem with that class
of drugs, and so it perhaps was appropriate, but it is no longer to push
the sponsor to find out whether it is happening.

And so at the end of it, and of course there was self-serving and one
may have some skepticism about the freedom from bias of our process, but
we thought that he had not done badly.

DR. GALSON: Victor.

DR. RACZAKOWSKI: I am Victor Raczkowski from the Office of Drug Safety
in the Center for Drugs. The only thing I will add is that Phase IV
studies are not always safety studies. Many of them are pharmacokinetics
studies, and there might be chemistry manufacturing types of studies,
and so just keep that in mind as you talk about Phase IV studies.

DR. NELSON: I might also be able to shed some light on that 13 percent
figure, because that comes from an internal regulatory research report
that Dr. James Grumlish and I did in the early \'90s. I have two things
to say about it.

One is that it was on a time frame \-- I think the data was pulled from
the late 1980s, and into early 1990, and so that data are real, and it
did look just at the safety Phase IV studies, and not all the other
kinds of Phase IV studies.

And that is a very valid point, Victor, because actually most of the
Phase IV studies are biopharmaceutical, and long term cancer studies,
and all kinds of other things.

But that study was submitted to the agency and was actually submitted
down to Congress prior to the FDAMA Act being developed. It also served
as the initiator for the Phase IV tracking system that you have at the
agency, as well as the stipulations in FDAMA about putting the Phase IV
requirements for approval in the approvable letter and monitoring those
after.

And so it did have an impact. I think there has been some remedy at the
agency for that deficiency, and I would love to see that study redone to
see if the rates have gone up in the last decade.

DR. GALSON: Just one note on Phase IV studies, is that we are about to
activate in the agency a new website which is going to allow people to
search those studies directly; and without spending a lot of detail on
this, I think all the comments that I have heard are fairly or are valid
and a lot more as well.

The 13 percent number I think is really sort of deceptive, a deceptively
simple description of what is actually going on there, which is quite
complex. But we will be making more information public about what is
actually going on with those, and the follow-up on those studies, soon.

Dr. Stephenson, I think we will take this last question and then if we
could hold all additional questions, we have got a lot more time in the
afternoon than the morning for this type of discussion. We will have
time for all the questions.

DR. STEPHENSON: Okay. My comment, and I guess perhaps a question for
FDA, has actually been touched on slightly two questions ago. And that
relates to signals, and the description of what a signal is in the
concept paper, I think it needs to be flushed out more, especially in
just listening to this discussion, everyone seemed to use the word
signal in very different ways.

And I have heard possible signal, potential signal, and signal, used
sort of synonymously sometimes. I think the description of a signal is
good, but it is a broad description that I think encompasses several
degrees of strength of a signal.

Many of us are used to having a very low threshold for calling something
a signal, and which would then trigger you to take further action in
terms of pulling data together and better assessing the situation.

But more times than not those sort of signals end up being non-issues.
And so I raise this because when you read further on in the document,
there is a section that refers to reporting the signals to FDA, and the
expectation that if there is a signal that the FDA would expect the
company to submit a fairly comprehensive report, which would include
pre-clinical findings, and a discussion of potential risk management
plans, if that is appropriate.

And even further in the document there is some language to suggest that
risk, management plans may be a way to mitigate a signal, which sort of
brings you to the point of a signal being used almost synonymously with
risk. I wonder if Dr. Chen, if you can comment on that.

DR. CHEN: I think the points are well taken. We have not addressed the
difference between signals and risk and certainly we are still at a
level with the signals on what does that mean when we identify from the
database.

And then what do we do with the signals, and what we are hoping is that
the sponsor can help us out to fill the gap of when do you find that the
signals are useful to become a risk, a real risk, and that something has
to be done with that risk.

And how big is the risk, and again it depends on how big is the signal.
So hopefully this afternoon some of the studies and some of the analyses
of the information will help to bring the gap a little bit closer.

However, I think we need a lot of help from you all experts out there to
help us out to do that. So putting in writing what you think about it
and how we do that, I am really open.

DR. GALSON: Dr. Braun.

DR. BRAUN: Miles Braun, CBER, FDA. That last point was well taken. I
think that to take a step back from the discussion today, this is a
concept paper and the idea was to lay out different ideas and get
discussion.

You have homed in on what is a good point, and I think that having
reviewed the literature somewhat in this area, I can say that there is
not an agreed upon definition outside this room.

And that the one that was laid out of an excess of beyond the expected
in the reports certainly raises the issue of what one would expect. Some
have taken exception to the expected values that are generated through
automated data mining approaches.

And there is certainly or that is certainly a well-taken point. So this
needs to be flushed out further. It is in my view, it is a key issue,
and I will throw out an additional thought, is that the actual actions,
or actually two thoughts.

One is that actually because we have different views of signals, the
signal can be conceived of to some extent, and I am not trying to be
subjective about this; but in the eye of the beholder.

Let me give you an example. The physician who takes the trouble to fill
out a MedWatch form because a patient that was being taken care of had a
terrible adverse event to that physician, they are signaling to the FDA.

And to somebody that sits at the FDA and sees 300,000 reports a year,
this particular event may not seem to be a signal. And there are very
many other people involved in the process.

So I think that we need to think about who actually is the beholder, and
the reason that I think that is important is that each person has a
different action that he or she can take.

That physician\'s action is to notify the FDA. As an FDA person, we have
certain actions, and our agency can take certain actions, and similarly
at a company, you have different repertoire of actions that are
available to you.

And I think that it will be helpful if we can blend in some of this, and
as you said, if we can go into this in a little more depth, and we are
only laying out the concepts.

And to try to construct this signal idea in a way that will be helpful
in terms of actions that we all can take.

DR. STEPHENSON: Perhaps just one other point, and I appreciate the
responses. I agree with what you are saying. I think one of the things
that I was reacting to in the concept paper is that there was a
discussion of reporting, which is really not a concept.

And I think that if you start talking about the expectation that
companies should report signals to the FDA, you are going to get
barraged with things that you probably would not consider signals, just
because companies are going to want to be in compliance. So that was one
of the things that I was reacting to.

DR. GALSON: At the table, Dr. Faich.

DR. FAICH: Yes, I was just going to make a comment and an observation,
and it links to some of the discussion yesterday of Level 1, Level 2,
and Level 3, and that the levels analogy takes on a life of their own.

And I would like Anne Trontell\'s comments of let\'s call them Georgia,
Betty, and Sam, or something. We had a similar experience at one point
long ago that is worthwhile sharing, and that is that we started talking
about alerts driven out of the adverse drug reaction reporting system.

And the term alert is a potentially evocative one, and that was the
problem, and we learned that, and we changed the terminology to monitor
to adverse reactions. And then of course we were then accused of
bringing these from the outer reaches of the solar system and Mars.

But nonetheless, I mean, the point of that was to get away from a term
that could be misinterpreted, and to go to a more neutral term, and I
wondered whether thinking about not using the term signal, but using
some other term, might be considered.

DR. BRAUN: But not Georgia, or Betty. Those are taken.

DR. FAICH: Miles, to follow up on your comment asking about causality, I
have not got the exact reference about the authors, but I know that
there was a look at the French causality assessment method, versus
physicians reporting.

There was a tremendous rush to connect between the relative weight that
the reporting physician placed on particularly dechallenge, as opposed
to the algorithm system, and very much pointing to what you said, Miles,
as to what the treating doc, or the health professional would place, as
opposed to a straightforward algorithm, and that is one of the reasons
again that we don\'t advocate for that, because the weighting is
different.

DR. GALSON: Is this a follow-up?

MR. FENISHELL: Yes, it is. Yes, it is.

DR. GALSON: Okay.

MR. FENISHELL: I think what is the problem here \--

DR. GALSON: Please identify yourself.

MR. FENISHELL: Oh, I am still Bob Fenishell. I think the problem here is
what in computer terms would be called a missing level of indirection,
and what I mean by that is that what you have defined as the notion of
signal is that signal is an apparent excess of adverse events.

And the focus has been continually on this idea of apparent. Well, is it
real, and can we verify it by data mining, and can we verify it by this
or that. But if we verify it, well, then it is adverse events.

Now, the fact is that is not what in my experience most of the Phase IV
stuff of the post-marketing stuff that I used to see coming across my
desk at the agency consisted of.

It consisted of things that were not even adverse events. Prolonged QT
is not an adverse event. A doubling of hepatocellular enzyme levels is
not an adverse event. The patient doesn\'t suffer from any or either of
those things.

And here are indeed drugs that cause colossal changes in the
hepatocellular enzymes, or of QT \-- not very many, but there are some
\-- without ever causing adverse events.

The thing is that you are looking for an apparent excess in one
phenomenon, and then entirely independent \-- you know, once you decide,
yes, there really is an excess amount, and it is not apparent.

Then you have to decide, well, is that a phenomenon that in fact is
associated with a phenomenon that we care about, or is that just an
anomaly.

If you look at most big trials, you will see an excess of something. You
will see that, oh, yes, the people getting active treatment had slightly
higher bilirubin than the people who didn\'t, let\'s say.

And you say, oh, well, that is something that makes you think about some
liver disaster. Well, in the CAPRI trial, the Clopidogrel patients had
slightly higher bilirubin, and it was something in the third decimal
place.

But with 19,000 patients, you can show that. And that was just absolute
garbage. I mean, it didn\'t mean anything. I am sure that it is true,
but it just doesn\'t mean anything. If you say that a signal is \-- and
I am not sure what the word is, but a signal is something that catches
your eye, and it may be real.

And now, first of all, it is appropriate to see if it is real, and
secondly, it is appropriate to see if there was any reason to worry
about it. Those are two separate processes, and I think that a lot of
the discussion has been concerned about the onus of, well, yes, it
really is real, and there is a second step here.

It really is real, but perhaps the answer is so what.

DR. GALSON: Thank you. Does anybody want to comment on that? That is
sort of an open-ended point. All right. We are going to break for lunch
now, but before we break for lunch, I want to let folks know that if you
didn\'t have a chance to ask the question that you wanted to ask, we
will have plenty of time this afternoon.

I wanted to leave a question on the table particularly for the PhRMA
folks so that they have something to do during lunch. They asked us what
criteria should be used to determine whether a pharmacovigilance plan
describing firm efforts above and beyond post-marketing spontaneous
reporting is warranted.

And my lunchtime assignment for them is to provide us an answer to that
question, and then we will try to comment on that. And that is where we
will start the discussion. Thank you.

(Whereupon, at 11:40 a.m., the public workshop was recessed.)

A-F-T-E-R-N-O-O-N S-E-S-S-I-O-N

(1:12 p.m.)

DR. GALSON: We are going to start out this afternoon with a number of
presentations, first from the FDA, and then members of the public, and
then we will move into the discussion, and question and answer periods.

So our first presentation today is from Dr. Judy Staffa from the Food
and Drug Administration. She is an epidemiology team leader in our
Office of Drug Safety. Thanks, Judy.

DR. STAFFA: Good afternoon. I have the privilege of getting up here on
the third day of a three day meeting on Friday afternoon, at one
o\'clock, right after lunch, and giving FDA\'s last presentation on this
concept paper.

So I feel very privileged to do that, but I also have been told that I
have the authority if I need to, to burst into song at any time, and
that way if I see everybody kind of nodding off, I may do that.

And so since I am a really bad singer, I will try to keep my remarks
brief so that we can get to the input that we are really here looking
for today. But if you can try and stay with me, that would be great,
too.

Okay. What I am going to try to do today is give you an overview of the
section of the concept paper that really tries to focus on what we are
looking for from sponsors about observational data that we receive,
whether it is in the form of study results or protocols, that go beyond
the case reports that we talked about this morning.

Pharmacoepidemiologic studies obviously can be done at any time, but
typically there are two times when these studies are usually recommended
or most appropriate.

The first time is typically right after a product is marketed, or at the
time of marketing, and the purpose of those studies is to further
characterize potential safety signals that may have been identified
during the controlled trials.

This is typically important, particularly of interest in patients that
may be chronically exposed once the drug is marketed, or product is
marketed I should say, or in patients with co-morbid conditions or
concomitant drug therapies.

Another time when pharmacoepidemiologic studies might be recommended
would be after a safety signal is identified once the drug has been on
the market. And as we talked about this morning, many times a signal is
identified first from case reports, and then that signal, we try to put
that signal into an appropriate context using drug use data and
sometimes calculating reporting rates as you have heard about this
morning.

We all recognize that reporting rates are very preliminary, crude tools,
with which to describe and try to characterize the risk, but we really
need to move to the next step to pharmacoepidemiologic studies in order
to calculate an incidence rate.

Just very briefly, and I know that this is overkill for this crowd, but
very briefly there are several pharmacoepidemiologic study designs that
are commonly used in pharmacoepidemiology.

The first is a cohort study, in which patients are identified based on
exposure to a medical product, and then they are followed over time to
estimate the incidents of various events.

These studies, or the main advantage of this kind of study is that an
incidence rate can be generated for various events.

A case control study is a design in which patients are identified based
on the presence or absence of a particular disease outcome, and then
their history is examined for the use of different medical products.

Case control studies are very useful because they can help us identify
risk factors, in addition to the relative risk for an event occurring.
Another design, called nested case control, is a hybrid of the two
designs, in which a case control study is nested within a larger cohort
of medical product users.

One of the main advantages of this design is that it allows you to both
calculate an incidence rate, but also to then examine and explore risk
factors.

And then there are other hybrid designs, and as the field progresses
more and more designs are tried out, and some of them end up being very
appropriate for use in pharmacoepidemiology. We encourage sponsors or
researchers doing these kinds of studies to consider what design might
be best for answering the questions at hand.

In 1996, the International Society for Pharmacoepidemiology published a
document called \"[Guidelines for Good Epidemiology Practices for Drug,
Device, and Vaccine Research]{.underline}.\" And this document can be
found or can be linked to through the ISP website.

We find this document very helpful and very useful, and would encourage
people to consult this document when undertaking a pharmacoepidemiologic
study.

For the purposes of our concept paper, we selected just the minimum
requirements that we would like to see when protocols are submitted, or
studies are submitted that are just worth going over today very briefly.

First, we would like to see a clearly specified study objectives, and
these objectives should be tied in with the analysis so that it can be
easily seen how the study can achieve and answer the study questions
defined.

There should be a critical review of the literature that can take
advantage of earlier experience with either the drugs or products of
interest, and as well the outcomes of interest.

And we would like to see detailed research methods. That includes a
detailed description of the study population, including comparator
groups that have been identified as being relevant, as well as
operational definitions of both exposure and outcome that will be used.

We would like to see a full description of data sources that are going
to be used. We try very hard in the Office of Drug Safety to get out to
meetings and to see what kinds of data sources are available, because
that is always a very evolving field.

But we often run into protocols where there are \-- into data sources
that are unfamiliar to us, and we welcome detailed descriptions of those
data and the streams that generate them.

We would like to see projected sample size calculations, and the
associated statistical power. As you know, in the world of rare events,
power becomes a very important issue when determining whether to go
forward with the study.

And finally we would like to see detailed methods for how data will be
collected, and how it will be managed, and how it will be analyzed. In
pharmacoepidemiology, we use a lot of automated data, whether it is
administrative claims data or computerized patient record data, but
these present interesting challenges for some of the studies that we do.

And what we have listed here are just some factors as sponsors or
researchers are choosing the appropriate database to study the question
of interest, and we encourage you to openly discuss these in the
protocols, in terms of how you came to the choice of the database where
you have decided to conduct your study.

Some of these factors include the demographics of the patient
population. Clearly, if you are studying an issue in children, or in
women, you would want to select a data resource that captures a lot of
information on that subpopulation.

The turnover rate or mobility of the population in and out of the
database is critical, particularly for studies that are looking at
cumulative risk. As you know, many American health care is very
fragmented.

People have a great deal of mobility, and since many of our data sources
are based on health insurance plans, we are limited to the length of
follow-up that we can actually capture people, which is as long as they
stay in the system.

Plan coverage of study medications is another crucial element to look
at. You need to understand what both the uptake of a product is, and I
know that was mentioned this morning, but also whether or not a plan
will cover particular products when you are looking to study them.

There are many plans that will not cover products they consider to be
lifestyle drugs, such as weight loss products, or products used for
erectile dysfunction, and it is an important consideration to
understand.

We would like to have detailed information on the size of the population
available for study, and in automated data sources that is a particular
issue, because rather than simply get a first take which says that, gee,
I have 10,000 patients that had at least one prescription for this drug
in the database, we all know that by the time that we get done apply
different enrollment and eligibility criteria, that 10,000 number may be
quite substantially lower.

And any estimation that you can give of that quantity would be very
helpful in trying to get a realistic sense of how large the population
will be.

It is important to understand whether the outcomes of interest are
actually available in the database, and one very good example of this is
death.

Death would seem to be a very straightforward outcome to study, but in
an administrative claims database, unless death happens in a health care
facility where care is being provided, it may not be captured.

A death may just be the absence of continued claims, and so it is
important to consider whether the outcomes that you want to capture are
actually available in the data.

It is also important to understand whether the outcome of interest is
actually coded, or there is an specific ISD-9 code since many systems,
at least administrative claims systems, use this coding system, and it
is important to know if the code that you are looking for actually does
exist, whether it exists in a code with about nine other diseases as
well or to understand whether it is even captured in a specific code.

And finally we feel that it is very important to understand whether or
not you can access medical records through the data source that you are
using. Unlike some of our colleagues in pharmacoeconomics and health
services research, we often find the need to validate the diagnostic
data in the claims with medical record data to truly understand whether
the event that we think has happened has actually happened, and has
happened at the time at which we believe it has occurred.

Now, with new regulations, it can be quite challenging to gain access to
medical records, but there is now about 20 to 30 years of research using
these kinds of databases, and many authors have been publishing when
they are able to validate claims.

And it is important to go to the literature and understand if the
outcome that you are looking at has already been studied and validated
in a system like the one that you are planning to use, and that would be
useful information for us all to know.

Moving on to another source of observational data, I am going to talk a
little bit about registries. Now, we have defined a registry for the
purposes of the concept paper as a systematic collection of defined
events or product exposures in a defined patient population for a
defined period of time.

Registries are typically prospective in nature, collecting data on
patients that are identified either as having a particular disease, or
as receiving a particular treatment.

And typically they are used to collect information that may not be able
to be collected within standard systems. Like good pharmacoepidemiologic
study protocols, good registry protocols should include clear objectives
of what is to be gained, as well as a literature review.

Now, the detailed research methods really should focus quite a bit on
patient recruitment and follow-up, because that is clearly of interest
to understand whether the patients in the registry will truly be
representative of patients out there.

And if they are not, to understand in which ways they are not. Projected
sample size really is an important issue, particularly with regard to
attribution and loss to follow-up, and any estimations to that effect
can be very crucial to how useful the registry might be.

Methods for data collection management and analysis is important also.
As was pointed out this morning, there are often not comparator groups
for registries, and it is important to understand if there is a way to
develop or to anticipate a comparator group, or what background rates
may be relevant for that particular registry.

If possible, we would love to see the data collection forms that have
been designed to collect data on the registry, and help to be able to
contribute to those.

And also validation of findings, particularly if the findings are
patient reported, and whether there is a plan to validate those with
health care providers.

Another strategy that is used to collect observational data are surveys.
Surveys are not really \-- surveys are not studies that could actually
determine an incidence rate, or a cause and effect, but a survey can be
very helpful to provide supplemental information with regard to a drug
safety issue.

You can use surveys to assess knowledge of both patients and providers
about labeled events to better understand medical product use; to assess
compliance with risk management programs as was discussed yesterday; or
in the world of medication errors, surveys are often used to try to
survey pharmacists to understand more about the potential for
proprietary trade name confusion before a product is launched.

And again good survey protocols are very much like other study
protocols. Detailed objectives and methods are very helpful to have in
the protocol or in the finished report, with particular focus on where
the patient or provider sample came from.

And to understand the pharmacist, or patients, or providers that were
sampled, what was the sampling universe, so that we can better
understand whose views they represent within the profession.

Projected sample size and methods again, and again if there are survey
instruments that can be provided, then the reviewers at FDA could do a
much better job of determining the validity of the findings.

So what do we do once we have all this observational data? Well, what we
would like to do is rather than have things come in kind of piecemeal,
because we all know that there is no one piece of observational data
that is really going to solve an issue, or really give us an idea of
exactly the magnitude of a risk.

What we would like to see is a synthesis of the information that is
available; the case reports, the drug use data, the pharmacoepi studies,
the registries, the publications, all of the data that are available
about that signal that can help us to understand what in a total picture
might be going on.

We would encourage sponsors to provide their own assessment of the risk
benefit profile based upon that synthesis. And that synthesis should be
providing the limits and the strengths and limitations of the different
study designs and data sources used.

We would also encourage sponsors to propose steps to further investigate
if they feel that further study is needed. And finally this is an
opportunity to propose risk management programs if they appear to be
appropriate given the signal.

And I would also in response to one of the questions this morning, to me
if a signal turns out to not be a signal because of other observational
data, I would say that could be part of the synthesis as well, and that
that was really what the sponsor believes is the take home message.

Then at the FDA, we can do our own review of the synthesis, and we will
be looking at probably \-- I am hoping the same things that you are
looking at, which is that we will be looking at the magnitude of the
signal and its precision; trying to see what are the consistency of
these findings across the available data sources given all of their
limitations and more of the things that we know about them.

To be looking at the biological plausibility of the event, and the
seriousness of the event, but also the degree of benefit that the
product offers; the availability of other therapies, and in the event
that things aren\'t to the point where a risk management program is
proposed, the ability of that program to realistically be able to
mitigate risk.

And now this is just a summary of the three questions that I guess we
have felt were relevant, and I know that some folks have already
presented their thoughts on those, and we are very appreciative.

But for the afternoon conversation, we are very interested in hearing
more about what folks think about registries as surveillance tools, and
when they would be useful, and when they might not be.

Secondly, we are very interested in hearing your thoughts about active
surveillance strategies, and when you feel that they might be useful to
identify events that are not reported to our passive surveillance
system.

And finally what circumstances do you feel that additional
pharmacoepidemiologic studies might be useful, in addition to the
circumstances that we have described. Thank you.

DR. GALSON: Thank you, Dr. Staffa, and now it is time for our
presentations from the public, and I am going to shift the order of the
presentations a little bit.

First will be Dr. Stemhagen, and then John Ferguson, then Dr.
Stephenson, and then Dr. Kahn.

So if Dr. Stemhagen could come up, please.

DR. STEMHAGEN: Thank you very much. I am Annette Stemhagen, and I am
Vice President of Strategic Development for Covance Perry Approval
Services, and today I am speaking on behalf of the International Society
of Pharmacoepidemiology or ISPE.

As you have heard, if you have been here the last few days, ISPE is a
non-profit, international professional membership organization decided
to promoting the science of applying epidemiologic approaches to the
study and value of therapeutics.

The society provides an international forum for sharing knowledge and
scientific approaches to foster the science of pharmacoepidemiology. We
have over 700 members representing more than 45 countries.

Our members work in academic institutions, in government agencies, and
in industry, from for profit and for not for profit organizations. The
specific backgrounds of our membership is quite varied, with
epidemiologists, biostatisticians, those involved in medicine, nursing,
pharmacology, pharmacy, law, health economics, and journalism.

And I would like to thank the FDA for allowing us the opportunity to
provide comments on this concept paper number three. The comments that I
am going to provide are based on feedback from senior members of ISPE,
and we will also be providing written comments with a broader
representation of our membership as those comments are collected and
collated over the next week.

I would first like to say that we are very pleased in this concept paper
with the recognition of the role of epidemiology and the value of the
use of observational data and tools for the understanding of risk
assessment and risk evaluation, and assessments, excuse me.

It is not often that epidemiology and its value is quite as recognized
as in this document. So thank you very much, and we are very pleased,
and we look forward to working with you on that.

I would like to begin with several areas of the concept paper, where we
either have comments or some questions for the agency. I\'m sorry, I am
behind in my slides. This is our website.

The first is a qualitative definition of risk. From the epidemiologic
standpoint, we recommend providing a quantitative definition of risk in
the guidance document.

Risk is a measure of the frequency based on an enumerator and dominator,
and if you have ever been around epidemiologists, we certainly live and
die by enumerators and dominators. So we are always going to have a
comment like this.

The enumerator in risk consists of the number of people who develop an
event, and the denominator consists of the number of people exposed per
unit of time.

This can also be thought of as an incidence rate over a particular time
period. The next thing that we would like to comment on is expansion of
the type of appropriate studies for pharmacoepidemiology assessments.

We would suggest that the epidemiologic assessment of safety end points
is important and should be further emphasized in this concept paper.
Also very important are studies of the natural history of disease, as
well as evaluation of drug use patterns and patient characterization
studies.

And that these studies should be done proactively, and they will
certainly assist the sponsor in terms of gathering data should a risk
management program be required. So I think as Dr. Jones talked about on
speaking of concept paper number one, really pharmacoepidemiology needs
to extend back as you have acknowledged into the development phase, as
well as the post-marketing.

And we would like to urge that sponsors consider looking at natural
history studies and patient characterization studies so that you really
can gather background information that is going to be critical as we go
forward with risk management.

This serves to expand our understanding of safety end points, and also
helps put the spontaneous reports in context as well. Natural history of
disease and safety study among users of available therapeutics before
your own product is on the market should be initiated during early
development.

Drug use in patient characterization studies should be initiated during
the development stage of a new drug based on the therapeutic class, and
then continue after launch to further understand drug use in diverse
populations and across various countries.

Similarly, evaluation of end points among products in the therapeutic
class and within diverse populations are important, and I think we have
already acknowledged that as well during the discussions the past few
days.

Targeted safety studies should be initiated shortly post-launch based on
the specifications of the development program and the pharmacovigilance
plan. The next comment, I am actually going to deviate from my prepared
script.

This has caused quite a controversy, in terms of the definition of
registry, and as the three days have evolved, there are a number of ISPE
members here, and we have been discussing a lot the discussion of
registries.

I think we have here, because we couldn\'t even agree among ourselves
exactly what a registry was, with a lot of contention between the
definition as presented by the FDA, a systematic collection of defined
events or product exposure, in a defined population for a defined time
period.

That is somewhat self-limiting. It does describe, for instance, a birth
defects registry, which would be a collection of events, but it doesn\'t
necessarily describe all of the kinds of registries which must be
considered more like an observational cohort.

So I think here, again just in evidence from the discussions that the
audience has been having among themselves, that we need a better
nomenclature I think for registries.

It is not totally clear exactly what they are, and based on my
experience when we are talking about registries in other countries, many
countries really shy away from actually even using the term registry.

And it makes it even more muddied as to what they are talking about. So
I think that we need to work together to get a better definition on
that. I am also going to talk a little bit about definitions for
pharmacoepidemiologic studies.

We recommend that in Section 4(a) about pharmacoepidemiologic studies
that they be classified as pharmacoepidemiologic safety studies. Just
for clarification purposes, pharmacoepidemiologic studies are not
limited to safety studies as I am sure that everyone knows, but include
all observational studies which describe the use and effects of
pharmaceuticals in a real world setting.

We also recommend that in Section 4(a), where it is a definition of
pharmacoepidemiologic studies, that this be expanded to include
post-marketed data collection, in addition to the already mentioned
existing automated databases.

In several sections of the concept paper, pharmacoepidemiologic studies
seem to refer only to those studies conducted in automated databases,
and that is probably not the intent, but it is certainly could be
construed that way.

Things like the large simple studies which were discussed on day one in
terms of premarketing, can also be pharmacoepidemiologic or
observational studies.

So really the definition needs to be expanded to include prospective
data collection as well. In terms of Section 5 regarding safety signals,
there is a list on lines 254 to 258 of five examples of signals. It is
very useful, and it probably covers most of the options or types of
safety signals.

However, we recommend expanding that list to include a category of other
risk factors, or risk factors associated with the development of the
adverse event of interest.

Safety signals regarding specific subgroups of patients, and that might
be either age, race, genetics, specific co-morbidities, or ways in which
the drug is used \-- for example, dose escalation may also be important.

So there are really other categories that could be included in there. We
would also like to ask for a clarification of line number 319, point
number 8, which says that analyses of the potential for an excess of
adverse events given the disease being treated, such as might be
observed in advanced cancer or immunocompromised patients.

Our question is whether that is intended to address analyses to identify
the background risk of adverse events in the disease being treated,
which is actually then point number 9.

We weren\'t really clear what the distinction between those two are. And
then looking at all of those points together, we would like to make the
point that many of those are probably not appropriate for analyses using
spontaneous adverse event reporting data.

But much more apply to analyses based on other types of
pharmacoepidemiologic data, either prospectively collected or through a
large automated database.

The next point that we would like to talk about is Line Number 352
actually, the relative risk or odds ratio. The point mentions that
relative risks or odds ratios from pharmacoepidemiologic studies can be
used to evaluate causality.

Although risk ratios or odds ratios are very frequently reported as an
outcome frequency in pharmacoepidemiologic studies, we would like to be
on record as noting that risk differences, and not ratios, are best for
evaluating the frequency with which a drug causes or contributes to an
event.

So in addition to an estimate of relative risk, we would suggest that
risk differences or discussions of incidents rates in particular be
included in that section.

In the end the frequency with which an event occurs because of a drug,
along with other characteristics of the event, must be weighed against
the benefit of the drug.

And I think this point again was discussed several times over the last
few days, but in addition a risk management program needs to consider
how the risk associated with the intervention, which is designed to
reduce the risk of a particular event \-- for example, the potential
reduction of benefit associated with the use of the therapeutic as a
result of the risk management intervention.

So we have to remember again that balance of risk and benefit, and that
if we are putting a risk intervention in place, and we have to be sure
what the impact on the benefit is.

If we restrict things as an example too much, then the benefit may be
reduced with patients not able to get access to the drug. The next is a
discussion of reporting of safety signals. In Line 374, Paragraph (d),
and I think this is what Dr. Stephenson had pointed out earlier as well,
in the reporting of safety signals, there really needs to be a
definition of when is a signal really a signal.

A lot of what is discussed may be our own analysis to evaluate whether
it is a signal or a false positive. And the way that this paragraph is
described, it looks like hypothesis generating, testing, prospectives of
risk and benefit, and risk management, all need to be done in a package
and sent to the FDA.

When in fact many of these things, or the first part of this, will be
done many times by a pharmaceutical sponsor before the FDA needs to be
notified, and we are trying to rule out some things.

And in addition there are sequential \-- there is a sequential order to
this. You may not need to get to all of these steps again if you have
ruled something out. So I think we are asking for a little bit more
clarification on that point as well.

In terms of surveys, I thought that this was very interesting that this
was included. You don\'t often see again some discussion of surveys. We
would like to comment on the inclusion of surveys as a risk assessment
or risk management evaluation tool.

Many of the examples provided in the document are actually
cross-sectional designs. We would just like to make note that
longitudinal cohort studies to evaluate changes in risk over time can
also be surveys, and so they don\'t need to be limited to
cross-sectional surveys.

And in talking about these kinds of surveys, or some of the types of
surveys that are listed as examples, in addition to epidemiologic
expertise, which of course we believe is very important in many of these
analyses, it is going to be important to include members of other
academic disciplines, such as social scientists, who bring expertise in
areas such as knowledge, attitudes, and behavior surveys.

I think that this is an area where we as epidemiologists probably are
not as strong, or many people that are not psychosocial epidemiologists
are not as strong.

And within pharmaceutical companies, or within the FDA, I think we are
building this expertise. So, just an acknowledgement that we really have
a number of disciplines that will be important in trying to evaluate
risk intervention programs.

And I guess just one comment on what Dr. Staffa had said, that surveys
are really not used to assess cause and effect, but actually I think
there may be instances, especially if we are talking about a
longitudinal survey, where you could look at cause and effect.

You might use some of this survey methodology if you have a risk
intervention, and you want to evaluate its effect, you may do surveys in
the beginning, and you may do surveys again, and you really can look at
cause and effect.

So there is really a broader application I think to surveys than the
document might suggest. One other comment really before I talk about
analysis, which is really our last point, is that many people already
even at this concept stage are taking this document quite to heart, in
terms of very literally assuming that the study designs discussed in
here are the study designs that you must do, I think it is not always as
straightforward as I think everybody knows, in terms of the
circumstances in trying to decide what type of study design, a
pharmacoepidemiologic study design, is important.

That the guidance document is not prescriptive, and so many people that
are not pharmacoepidemiologists are taking this very literally as, okay,
this is the issue and this is the design we are going to do.

If there can be some discussion that there are a multitude of designs
that might be appropriate for a situation, I think that would be very
helpful. Again, it is a guidance document and should be interpreted that
way, but I am not sure that it always is.

Our final point in terms of ISPE is that we believe that there is still
a great deal of work to be done in the development and refinement of
analytic methods for evaluating safety signals, as well as methods for
risk assessment and risk management evaluation.

ISPE is committed to working with the FDA and to others, such as the
CERTS, to further the knowledge of the methodological and statistical
techniques required. ISPE is firmly committed to providing an unbiased
scientific forum for the views of all parties with an interest in the
science of risk management.

And we are deeply committed to advancing this science. We would like to
welcome the opportunity to work with the agency in this area as we did
in actually developing the guidelines for good epidemiology practices in
1996, and the data privacy document in 1997, both of which are
referenced in this concept paper.

And as I said, we will engage our full membership in the feedback
process on this concept paper as well. Our next annual conference will
be focused on risk management, and anyone who has been here for the last
couple of days has already heard this.

There is several workshops and sessions being planned jointly with the
FDA. So as my colleagues over the past two days have done as well, I
would also like to extend an invitation to everyone to join us for the
first International Conference on Therapeutic Risk Management, which is
held in conjunction with the 19th International Conference on
Pharmacoepidemiology.

It will be held in Philadelphia, August 21st through the 24th of this
year, and so we would like you all to come. I would also just like to
add one more thing.

I am the membership chair for ISPE, and so hopefully I have tantalized
you with all the wonderful things and great thoughts that ISPE members
have. And if anybody would like to become a member, please see me.

DR. GALSON: Thank you very much, Dr. Stemhagen. Our next speaker is John
Ferguson. I should have said Dr. Ferguson, excuse me.

DR. FERGUSON: My name is John Ferguson, and I am the Vice President of
Drug Safety at Biogen, and on behalf of the Biotechnology Industry
Organization, I thank the FDA for the opportunity to comment on concept
paper number three, risk assessment of observational data, good
pharmacovigilance practices in pharmacoepidemiologic assessment.

I will begin my presentation with general comments, followed by comments
in selected elements of the concept paper, and in so doing, I will
address questions posed by the agency, and at the end of the document, I
will close by reiterating key points.

BIO congratulates the FDA on concept paper number three. This important
initiative continues to move toward the increasing use of evidence-based
approaches to evaluating selected important safety questions.

BIO agrees that observational data can be very useful for confirming
signals. However, work is needed to maximize that utility. The level of
evidence derived from this data varied significantly with the type of
data collection, control of confounding, and the magnitude of the
observed effect size among other things.

Clarification of the agency\'s view of how level of evidence for causal
association varies with the design would be most helpful. Similarly, we
would welcome specification of the way in which regulatory decision
making might be conditioned on the level of evidence and the strength of
evidence.

While I believe that the need for pharmacovigilance plans will vary, and
should be determined on a case-by-case basis, explicit criteria that
could serve as the basis for decisions about pharmacovigilance plans
would help to ensure their consistent and effective use.

Small companies will require time to effectively implement the full
range of proposed activities, since some techniques require special
expertise that is not widely available or easily accessible to them.

Moving now to specific comments on selected elements of the concept
paper, the paper refers to the agency\'s expectations when a signal is
identified, criteria that delineate the basis for deciding that a signal
has been identified, contrasted with those that confirm a signal would
be very helpful.

And in this we reiterate the comments of colleagues in other
organizations made earlier. While it is not the intent of this paper to
explain the nature and strength of safety findings to regulatory
decision making, it seems logical to do so.

For example, what impacts to signal identification, as opposed to
confirmation have on labeling. High quality case reports are the
cornerstone of good pharmacovigilance practice.

The major limitations to high quality case reports are the lack of
reporter incentive to complete the considerable amount of work that is
required to obtain complete information on safety issues and
confidentiality, which I will address in a moment.

I want to stress that incomplete information will remain a problem, and
so discussions concerning the rational use of incomplete information for
purposes of signaling in regulatory decision making would be very
useful.

There is no doubt that confidentiality issues impact our ability to
obtain complete information on safety issues. It is noteworthy that
confidentiality initiatives are progressing in parallel to this risk
assessment initiative.

Ideally over time the developers of the risk assessment initiative can
provide guidance to those working on confidentiality initiatives to
ensure that confidentiality regulations support risk assessment needs.

In particular, guidance which is clear regarding which activities, which
of the proposed activities do not constitute surveillance activities
would be most useful.

The utility of registries and surveys for evaluating signals depends on
design, and I want to reiterate a point that was made earlier that there
seems to be a lot of confusion in terms of the definition of registry.

It ranges in our experience from designs that are purely exploratory and
that may be able to identify signals, but can do little to confirm them.
And on the other extreme to studies that almost approximate large simple
trials.

The FDA can do a lot to reduce confusion in this regard by providing
more detailed definitions of registries or registry variance in surveys;
and delineating a clear approach to the consistent implementation of
this design.

In this regard the FDA has considerable experience with registries and
could increase the utility of the guidance document by way of examples
of situations where registries have proven to be very useful for
confirming signals and where they have failed in this regard.

The use of automated databases is a complex subject as has already been
recognized. BIO suggests that the concept paper include discussions with
relative merits and limitations of automated databases for purposes of
evaluating signals.

We would also welcome specific guidance regarding the validation of data
derived from automated databases. Background rates are useful despite
limitations.

But since there are limitations, clarifications for guidelines for use
in interpretation would be most helpful, and in particular with regard
to the need for and value of standardization.

And the basis for requiring alternatives when background rates may not
be optimal are available, but may be useful for addressing safety issues
in a given context.

There are clearly pros and cons that have been discussed regarding
reporting rates. BIO feels that reporting rates are often useful as
crude tools for signal evaluation. However, reporting rates can be very
difficult to interpret.

As such, we would welcome a discussion of the assumptions and basic
requirements for generating the reporting rates, the limits to their
utility, and the approach to their consistent interpretation conditioned
on under-reporting.

In contrast, product comparisons based on reporting rates are very
limited. They require strong assumptions that are often questionable.
Regulatory decisions based on comparisons of product reporting rates
should not be made without a clear understanding of the surveillance
models used for each compound, as well as the evaluation of the
procedures used to estimate treatment and exposure.

Such comparisons should be avoided when products are used for different
indications. In general, causality assessments are unreliable, but there
are clearly situations where causality can be attributed in individual
cases and excluded in individual cases.

Therefore, there is a need to define the role of causality assessment in
signaling. For example, how should causality be used in constructing
reporting rates and sensitivity analyses that go to those reporting
rates.

The FDA\'s concept of linking the degree of causality to the quality of
evidence is a very useful concept that warrants further development, and
goes to the point that I made at the opening of my presentation related
to the level of evidence and how it should impact regulatory decision
making.

The strengths and limitations of data mining are not well understood at
this point in time. There are strong underlying assumptions, and there
is a need for additional developmental work to define the exact role of
data mining in regulatory decision making.

In particular at this point in time, BIO feels that data mining may be
capable of identifying signals, and maybe more appropriately identifying
alerts as described by Gerry Faich, but it is not clear how useful data
mining will be in terms of confirming signals on safety information.

In summary, observational data can be useful for signal confirmation,
and represents an important move towards evidence-based assessment of
safety issues. Additional work is needed to understand the contribution
of different types of observational data to levels of evidence for
causality.

We recommend efforts to link the level of evidence to specific
regulatory actions, and explicit criteria for pharmacovigilance plan
requirements will increase their effectiveness and consistency of their
use, as small companies need time to acquire the expertise needed for
effective implementation of the full range of the proposals.

In closing, we thank the FDA for the opportunity to comment on concept
paper number three, and we again applaud the agency\'s efforts to date
in this important endeavor, and we look forward to working with the
various partners to continue the development of these key concepts.
Thank you.

DR. GALSON: Thank you very much, Dr. Ferguson. Our next speaker is Dr.
Wendy Stephenson.

DR. STEPHENSON: Good afternoon. I am Wendy Stephenson, and I am a senior
vice president for Global Safety Surveillance and Epidemiology Labeling
and Health Outcomes for Wyeth.

I have worked within the industry in the area of safety for about 15
years, and I have been a member of the CIOMs working group on adverse
drug reaction reporting for just about as long.

This afternoon, I am here representing PhRMA. I would first like to
commend the FDA and the various committees for their excellent work
drafting the three concept papers that have been the subject of these
very interesting three days.

And to thank the FDA for allowing PhRMA to participate and to publicly
comment at this meeting. I will be addressing the FDA\'s questions 4, 5,
and 6, following Linda having addressed the first three questions.

But will end with some additional questions and points to consider for
the FDA that PhRMA hopes will generate some additional discussion. Under
what circumstances would a registry be useful as a surveillance tool and
when would it cease to be useful. While there are clear circumstances
where registries represent an important surveillance tool \-- for
example, for the use of pregnancy registries \-- in most situations
there are likely to be better alternatives.

For most products and in most circumstances, it is probably not going to
be useful to establish a registry. Registries may be useful when the
information you need to ensure safe use of a product cannot be obtained
from another mechanism.

And the need for the information justifies the extensive resource
commitment necessary to set up, operate, and comply with the registry.
Ideally there would be a concomitant comparison group, and if not, then
at the very least a comparable historical control, or well established
background incidence of the event in question.

With these criteria in mind, PhRMA supports the concept of pregnancy
registries under appropriate circumstances, and agrees that there may be
other similarly special circumstances where a registry may be
appropriate.

One example that comes to my mind is the Myelotarg VOD registry, which
was established after a signal suggested the possibility of an unusual
liver toxicity in patients with AML.

VOD is veno-occlusive disease and it is an unusual liver toxicity. That
registry, with a well defined outcome of interests, in a well defined
population, is expected to provide useful information on this potential
risk that would not otherwise be available through other sources.

Where a registry is deemed appropriate to achieve a safety surveillance
goal, there should be specific objectives and a well defined outcome of
interests. The size of the patient population, as well as the duration
of follow-up, should also be well defined, and not open-ended.

Industry responsive registries tend to be drug specific registries.
However, it may be advantageous to establish a disease specific registry
with collaboration across companies and other parties as appropriate.

If separate registries are established for similar compounds, there
should at least be some standardization so that comparable levels of
reporting and evaluation may be accomplished.

A well tested alternative to registries is the use of large automated
databases to serve as a source of appropriate cohorts. Databases such as
Kaiser Permanente, Saskatchewan in Canada, GPRD in the U.K., are
powerful tools used by industry to sponsor research without assuming the
cause of initial data collection, which registries entail.

In addition, large automated databases capture all patients in a defined
population; whereas, registries usually only capture those patients who
are voluntarily enrolled in a study.

Therefore, selection bias of an unknown magnitude and origin may be a
particular concern for registries. PhRMA encourages collaborative review
by industry and FDA of large linked databases for the evaluation of
safety signals.

Under what circumstances would active surveillance strategies prove
useful to identify as yet unreported events.

Active surveillance strategies, which often include the concept of
registries, may be useful when information cannot be obtained from more
readily available tools that are less resource intensive and less
obtrusive.

One approach to active surveillance involves the use of specially
trained sentinel sites generally affiliated with academic centers to
stimulate the reporting of events that might not otherwise have been
spontaneously reported.

Such strategies may be subject to selection bias, and are likely to
involve relatively small populations, and so it may not be useful for
detecting rare events.

When successfully employed, however, they may provide more in-depth
information and a more accurate estimate of the reporting rate, and may
be more likely to provide information that will help with making a
causality assessment.

Another approach to active surveillance is prescription event
monitoring, such as the green card system that has been in place in the
U.K. for many years. This approach differs from most others, in that
events are collected, regardless of the suspicion of drug relationship.

If conducted with a systematic consistent approach across a wide variety
of products, the resulting database can be used to make comparisons
across products.

And because the denominator is known actual incidence rates can be
calculated. Care must be taken, however, in the interpretation of
comparisons between drugs that may be prescribed under different
circumstances that lead to confounding and bias.

It is not clear that prescription event monitoring has ever lead to the
identification of previously undetected signals. It can, however,
sometimes be useful for confirming signals that have been previously
detected through spontaneous reports.

Other initiatives which probably should be considered to encourage the
identification and reporting of adverse events were mentioned earlier by
Dr. Goldman, and that is the implementation of educational programs for
health care professionals.

Perhaps providing CME credits for reporting complete and high quality
reports to either the agency or the company might be considered as an
incentive.

And the last question, under what circumstances would additional
pharmacoepidemiological studies be useful. Pharmacoepidemiological
studies are useful to test hypotheses generated by a signal from the
spontaneous reporting system, and to further evaluate the safety profile
of a product for which a potential risk has been identified prior to
approval.

However, as with all data sources we have been discussing today, no
study will ever be definitive. Rather, it will represent just one more
piece of the puzzle. Pharmacoepidemiology studies may have false
positive, as well as false negative, results due to bias and
confounding.

Fortunately, utilization of some of the better automated databases
allows for control of confounding to a large extent. Perhaps the
greatest value of pharmacoepidemiologic studies is their usefulness both
pre-and-post approval in identifying background incident rates for
adverse events of interest.

In addition, they may be useful to identify high risk groups or risk
factors for a particular adverse event. The same databases may be used
to conduct drug utilization studies which can provide useful information
on physician prescribing and monitoring behavior, characteristics of
patients who are treated with the drug of interest, as well as drug
usage patterns, and the impact of those usage patterns on outcome
related to both benefit and risk.

The same types of studies can thus be used as a mechanism for measuring
and evaluating the effectiveness of risk management interventions, at
least in the population covered by the database.

In summary, PhRMA supports the development and use of observational
studies of information beyond spontaneous reports. PhRMA agrees with FDA
that pharmacoepidemiologic studies are useful to further evaluate the
safety profile of a product for which a potential risk has been
identified through the spontaneous reporting system or other source.

All of the modalities available have limitations, and no one approach
can be expected to provide a definitive answer. Conclusions regarding
actual risk will largely depend upon the consistency of findings, and
the strength of evidence after taking into account information from as
many sources as possible and within reason.

And I end with some additional \-- and I am afraid to ask these
questions, but hopefully these will be more difficult to pass back to
us.

DR. GALSON: Try me.

DR. STEPHENSON: Since both the FDA and industry conduct
pharmacoepidemiologic studies, it would be helpful to hear FDA\'s
thoughts on when a study should be conducted by the company, versus FDA,
or both.

And, secondly, what does FDA view as its obligations to a company when
the FDA conducts a pharmacoepidemiologic study on that company\'s drug.
Thank you for your time.

DR. GALSON: Thank you very much. Our next speaker is Dr. Sidney Kahn.

DR. KAHN: Thank you. Good afternoon. I am Sidney Kahn, President of
Pharmacovigilance and Risk Management, Incorporated, an independent
organization providing consulting and support services to the
pharmaceutical industry, and its supporting organizations.

And I am very appreciative of being allowed to present my views in front
of this august gathering today, and I would also like to thank the FDA
for providing such provocative and insightful documents, as well as
allowing the public to comment on their content and hopefully their
eventual improvement into the eventual guidance as the product is
developed.

I would like to start with what I consider to be an omission from the
current documents, which is the role of therapeutic or Phase IV studies
in risk assessment. I started talking about this yesterday afternoon,
and so this is the right slide and the right presentation at this point.

The May 1999 reports of the task force on risk management had a fairly
extensive discussion on the use of utility or the potential utility of
Phase IV studies in risk assessments.

And I was quite surprised then to find that the current concept paper
makes almost \-- I think it makes almost no mention of it at all, or if
it does, it is purely in passing.

And this is odd to me because it strikes me that this is a wonderful
mechanism and tool for identifying or characterizing, or both, either
new risks or established risks that have been identified during clinical
trials.

And this is a mechanism that is already in place, and this is not
something that would have to be added on as a new or large burden. The
additional burden of uniform data collection and data aggregation is
real, but it is nowhere near as large as some of the other interventions
that have been proposed.

So using it in the vernacular, I see a substantial bang for the buck in
doing this type of approach, and it would also I believe help us to move
forward with some additional characterization of new signals that could
then be evaluated further in other subsequent larger databases.

And I would like to suggest that that may be included in the final
guidance as an area that could be used as a significant risk assessment
tool in the marketed product that we know.

I am going to address several specific points in the document, and some
I will and some I won\'t touch, as several have been dealt with much
better than I could by others.

But if I could talk about an issue of case follow-up and to a lesser
extent expedited reporting, which is not part of the guidance. In
relation to a different regulation that the FDA has proposed, which are
related to format and content of the adverse reaction sections of the
drug safety labeling, under the new proposal only adverse reactions,
whatever they may be, will be included in the label.

And events that occur in the study population at a comparable frequency
or without drug therapy, will not be included in that label proposal.
Now, for those of us who have worked for any length of time in
spontaneous post-marketing reports, I think we all know that the profile
of the events that get reported post-marketing tends to be quite smaller
to that which was observed in the clinical trials in the population.

And if we omit from the labeling any description or listing of those
common events which may be serious in many populations, such as
diabetics or cancer patients, or people with congestive heart failure,
or a variety of other conditions, if those are omitted from the
labeling, they will become under current regulations unexpected or
unlisted if they are in the company core data sheet.

And unexpected events have specific regulatory requirements which are
quite burdensome, especially if there are very many of them. The issue
of aggressive follow-up, we have heard talked about.

And in addition to the burden on health care providers and the
difficulties that the new HIPAA legislation creates for them, all this
would do would require the sponsors or the manufacturers of drugs that
have high background rates of significant adverse events to follow up
the natural history of the disease, which I don\'t believe was the
intent.

So I see this as an area where there is some unintended consequence of
two intrinsically logical approaches that when taken together give you
something that you really had not intended at the end of the day.

So I would ask the agency please to take that into consideration, and
just to help them along. I am putting up here some issues or methods
that I consider could be used to obviate that problem, because this
would be a real problem.

One is that we could continue to include the totality of the adverse
events, including those that occur with comparable frequencies in the
active control groups in the label, as we do now in the table of adverse
reactions.

We could create an additional section of the label that contained
adverse events that the agency and the sponsor would consider expected
for the purposes of regulatory reporting and follow-up.

Or there is a possibility, and I don\'t know how this would drive with
the lawyers, but that is something that we could explore, of omitting
these comments from the prescribing information because it is not clear
that the physicians necessarily need to know how many patients with
congestive cardiac failure have been hospitalized with pulmonary edema.

But rather to agree between the sponsor and the agency at approval, at
the time of ongoing periodic reporting, for example, what common events
could be considered expected, and therefore not subject to aggressive
follow-up, or expedited reporting.

The FDA asked a question in this concept paper on population background
rates. We have heard a lot about this and clearly knowledge of this is
very important, and is very valuable.

What I would submit though is that the FDA has a unique opportunity to
find such data in great detail from its own databases, especially for
those of controlled populations in their archives in NDAs and BLAs that
the pharmaceutical and biologic industries have presented to the agency.

And there is an initiative that I just heard about recently from Randy
Levin, called JANUS which is under way in CDER, which is apparently
allowing electronic integration of such data, and perhaps our FDA
colleagues could comment on that in due course.

And I would certainly urge sponsors to allow their proprietary data to
be used in an appropriately automized way to help the FDA develop these
assessments of background rates, because these data are probably even
more robust than the pharmacoepidemiologic data for background rate
calculations.

We have heard a lot about under-reporting over the years, and I would
like to make some caveats and points about the myth as we heard this
morning on under-reporting.

First of all, high reporting rates for events that have a high
background rate to the population I think have to be considered
uninterpretable. So you can\'t say anything about them in my opinion,
and here we have an issue where again the labeling proposals might also
have an effective, because whether you include or exclude an event from
the label, will have some impact \-- I don\'t know how big or in what
direction \-- on the reporting by health care professionals.

And that would be something that should be evaluated. It would be very
helpful here to have some input from the medical professional
organizations as we heard about in relation to, say, the Rhode Island
study results, to determine how physicians would see such a change.

The issue of the much publicized 10 percent reporting rate which has
been touted as reflecting the tip of the iceberg, I am not sure that
many people believe that, and we have heard examples of why that may or
may not be so earlier.

But I want to quote you an actual example from personal experience. The
FDA sent my company at the time a letter in which they assessed an
incidence rate of a very severe, life-threatening, or fatal adverse
event at approximately 20 times the population background rate for the
population exposed to the company product, and based entirely on
spontaneous reports.

The company went out and did two extensive studies. We also went to the
PM database as Dr. Stephenson mentioned, and a variety of other places,
and we have been unable to confer anything like the number of cases in
the epidemiology studies.

We are paying an independent expert opinion from a clinician with great
expertise in the area who assured us that in his experience at least 80
percent of such cases would be reported by practicing physicians.

And rather than under-reporting and getting to the over-reporting
concept, we received multiple reports of the same case from various
reporters, and so further supporting the idea that the threshold for
reporting such cases is relatively low.

And to cap this, and I know that there are studies that have suggested
that publicity can alter the reporting weight following publications in
the medical literature in the U.S. and other countries, the addition of
a black box warning to the label, and a dear doctor letter, and there
was no increase in the reporting rate, which remained constant
throughout (inaudible).

And so I realize that this is an anecdotal experience, but I think it is
illustrative, and as a result, I would like to propose the following
hypothesis. That the more severe the event and the outcome, and the more
obviously it appears to be related to the drug, that is to say that the
less commonly it occurs in the treated population, and the less commonly
it has in alternative explanation, the more closely the reporting will
approach the true incidence rate.

That hypothesis is potentially invaluable, and if the FDA can access in
its various databases for a variety of products which sponsors do not
necessarily have access to, and compare instances of known adverse
reactions of varying degrees of severity \-- cough with ACE inhibitors,
versus liver failure with troglitazone, for example, with the
corresponding numbers of reports and time adjusted in errors, and it
would be very easy to get true estimates of the reporting rates.

The FDA has asked for comments on how to improve spontaneous report
quality. Not on the slide, but a thought that has occurred to me on
numerous occasions is that in my estimation probably the majority of
supposed SADRs received by pharmaceutical manufacturers come into
medical information in the form of I had a patient taking your drug
among 5, 7, or 10 other drugs, who has developed Event X. Is this known
for your drug.

And sometimes it is and sometimes it isn\'t. Every one of those is
captured and characterized as a suspected adverse drug reaction, and is
given precisely the same weights as a report from a health professional
saying that I had a perfectly healthy patient who had a minor condition.

I put her on your drug and so this terrible thing happened, and your
drug did it. I see those two scenarios as being somewhat different, and
I believe it could be helpful to alter the categories of reporting in
such a way as to differentiate between these sort of incidental reports
where a physician is looking to confirm or refute a differential
diagnosis in relation to looking for a known drug explanation, as
opposed to reporting a previously unknown adverse action.

DR. GALSON: Excuse me, Dr. Kahn. You have got a little bit less than a
minute left.

DR. KAHN: Okay. Well, I will complete the rest of my presentation
publicly in terms of written comments to the document. The last thing
that I want to switch to is to add a few comments or a couple of
comments to data mining.

We have heard a great deal about it, and the biggest problem that I see
with it is the absence of any pre-specified hypothesis. And therefore
because data mining can pick up signals that no one knows what to do
with, it is my opinion that this should probably be reserved for
regulators who do not have any necessary regulatory obligation to act on
this, as opposed to pharmaceutical manufacturers who could be placed in
a very awkward position, both from a regulatory and a legal perspective
if they undergo data mining when they do not have access to the complete
data set that the regulators have.

Pretty much everything else I had to say has in fact been said by other
speakers, and so I will be happy to conclude there, and I thank you very
much.

DR. GALSON: I thank you very much. Let\'s just have a quick show of
hands whether folks want to take a break or just go on through
conclusion, and if you want to just go on until we finish, raise your
hand.

(A show of hands.)

DR. GALSON: Okay. Good. Let\'s move now to discussion and Q&A section,
and let\'s go back to the question that we left hanging that I mentioned
right before lunch.

I don\'t know who from PhRMA wants to address that. Is it possible to
put the question back up, or you can just restate it, whichever is
easier.

DR. STEPHENSON: I am going to give you the short answer, and Linda is
going to give you the longer answer. The shorter answer is \-- well,
read the question first.

MS. HOSTELLEY: Okay. The question is what criteria should be used to
determine whether a pharmacovigilance plan describing pharmacovigilance
efforts above and beyond post-marketing spontaneous reporting is
warranted.

Okay. And the short answer is that on a case by case basis. And the more
elaborate answer is that essentially we would encourage the agency to
consider the criteria that have been put forward in CIOMS-III actually,
specifically strength of evidence, significance of medical saliency, the
seriousness of the event.

That is, you know, does it fulfill the serious criteria, or is it a
non-serious type of situation; the predictability, and the
reversibility, as well as the validity of methods available to be able
to answer the question within reason.

We would encourage also open dialogue between the FDA, and that would be
the Office of Drug Safety Review Division, and the sponsor, to really
reach a consensus as far as the best approach in involving this.

DR. GALSON: Okay. Thanks very much. Let\'s just stick on that for a
minute and see whether anyone else wants to make a comment on those
responses. Anybody from the agency want to say anything?

We will certainly consider that question and I think in general we agree
that there has to be a case-by-case assessment. It is very difficult and
it may be possible in certain classes of drugs to set out specific
criteria if you have a lot of background information.

But we can\'t or we won\'t be able to leave at least some aspect of a
case-by-case analysis. Any other comments on that? FDA folks? Go ahead.
I want to remind folks as well that if you want to fill out cards and
send them up, we are happy to take more cards.

MS. PEREZ-GUTTHANN: Susana Perez-Gutthann, Pharmaceia. It is regarding
this question of the pharmacovigilance plan, and I think one of the
useful parts of the discussions that we have had these last couple of
days is that there has been a lot of clarification in terms of what we
mean when we say a word, and the same word doesn\'t mean the same to
everybody.

And I think that something like this is happening to the
pharmacovigilance plan, where we are discussing when to initiate a
pharmacovigilance plan because in the definition it is implied that it
is a reaction to something happening to the drug.

And in fact I might add that probably most companies or most sponsors at
the time of launch have a pretty clear idea of what the background is in
terms of risk profile, and have formalized some kind of document in
which they say this is the area that we should monitor more closer or
not.

And whether the activities are going to be monitoring some kind of
reports, or going beyond and doing some pharmacoepidemiologic, et
cetera. And as I mentioned, if we compare it a little bit with the
perspective that the European agencies are taking, and this is referring
to the document that I was mentioning yesterday, the summary of the
heads of agency reports, they are taking a little bit of a broader
pharmacovigilance plan, and that these are general activities that one
would take.

And the criteria that they use for initiating this or for recommending
it, the pharmacovigilance plan, rather than based on an event or safety
issue related to the drug, is based on the type of drug or the
medications.

And if you allow me since we were talking about criteria, they mentioned
that they would be particularly relevant to new chemical entities on
biotech derived products, or from additional products, and significant
changes in established products, such as new forms introduced to a
population, establish products introduced to a new population, of
significant new indications, and finally establish products when
reclassified from prescription only to non-prescription availability.

I am not saying that we should be adopting this criteria, but this is
kind of a different take on when do we initiate the pharmacovigilance
plan.

DR. KAHN: May I continue the discussion on pharmacovigilance plans?

DR. GALSON: Sure.

DR. KHAN: This is Sidney Khan. In my last position in industry, I had
the opportunity to both develop and review a variety of what we called
pharmacovigilance plans, but which were primarily focusing on the Phase
III stage of development, because the goal was to try and focus
attention on events that could possibly be indicators of problems down
the road.

And there was a carryover as Linda Hostelley I think said earlier, and
we all understand that drug safety is a continuum, and therefore, you
can\'t start pharmacovigilance plans at the time of launch. You really
have to know up front where you are going to look, and you have to have
high sensitivity to what I would call harbingers of future events.

Bob Temple yesterday mentioned several times the elevation of the minor
transaminase and bilirubin as potential indicator of a pathotoxicity,
which would be very nice to see established by a formal means.

But pharmacokinetics should be focusing on safety professionals in the
companies and in the agencies on what to look for in drugs, and which
can be identified from the preclinical data, and from pharmacokinetics,
and pharmacodynamics, et cetera, throughout the development and life
cycle of the product.

DR. BEITZ: I think what I would just like to say is that by giving this
activity a name, like pharmacovigilance plan, and it gives us the
opportunity to actually talk about what we all want to be doing to
monitor the product.

And I am not so sure that the folks at the FDA always know exactly what
industry might be doing, and vice versa, and I think if we had some
clarify on what is appropriate or what is feasible, then we are way
ahead of where we are now.

DR. GALSON: Any other comments from folks, from the FDA, or any
questions from the microphone, folks at the microphone?

DR. BEITZ: This is not about pharmacovigilance plan, but this was
actually related to our active surveillance question. I didn\'t hear a
whole lot of comment about that, and it seemed as though what I was
hearing was somewhat of a lukewarm endorsement of this kind of activity.

And I was hoping that there might be some more discussion if possible.
One of the things that we sometimes toy with is perhaps mining
information from liver transplant centers, for example, to look for
potential cases of liver failure, or acute liver injury.

Does that sort of activity make sense to you all, and do you have any
other possible examples; surveying emergency departments for adverse
events coming in to such places, such settings? Any comment at all?

DR. STEPHENSON: I think that example makes a lot of sense. In fact, I
know that when we were dealing with the issue with Bromfenach, that was
one of the places that we went to, to see if there were any additional
cases of liver failure.

It was a new transplant registry, I believe, or it was a liver failure
registry, and that was very helpful.

DR. KAHN: In relation to the liver transplant registry, it doesn\'t
cover all the transplant centers. It is only a subset, and it is very
useful. But I think if one were to take the data from that database as
published and presented by Dr. Lee out of the University of Texas at the
Hepatoxicity Workshop, I think we would certainly have to take a certain
amount of it off the market.

DR. VASHISHTHA: This is a situation in which I think an active survey
might make sense.

DR. GALSON: If you would introduce yourself, please.

DR. VASHISHTHA: I am Anshu Vashishtha from Watson. Probably in
Stevens-Johnson situations, because again they are typically drug
induced reactions and it might be fruitful to an active surveillance
there. That is one situation that I think comes to mind.

We might for overdose consider poison control centers. I think it was
mentioned earlier about not integrating the poison control and drug
information centers, and they might be helpful to get (inaudible) in
those situations.

DR. GALSON: Let me just \-- well, any comments on that comment? Yes.

DR. STEPHENSON: Just one comment. I think that maybe some of us
misunderstood what was being discussed, in terms of registries, as part
of a pharmacovigilance plan.

My comments were sort of focused on should you develop a registry for a
particular product. I think that there are certain already existing
types of registries, like liver transplants, or Stevens-Johnson, that
obviously are very useful, but cut across all products.

So you are basically starting with the reaction and looking to see
whether there are any signals for any particular product. So I don\'t
know if that was also what was intended in the document, and maybe that
could be clarified a bit.

DR. GALSON: Let me read one of the questions that came in on a card this
morning. Dr. Hostelley stated that efforts should be directed towards
limiting false positives of data mining.

The question is are similar efforts under way to limit false negatives,
and next, further, how would one decide that a particular signal was a
false positive. Is a controlled trial necessary in that case. Did you
have any response to that?

DR. STEPHENSON: I looked to members of the collaborative group, either
Dr. Szarfman, or Dr. Almanoff, to perhaps respond.

DR. SZARFMAN: Dr. Almanoff left, unfortunately, and I think that \-- oh,
I thought you left. First of all, we need to start the program a little
bit. I think that the (inaudible) that we can do multiple comparisons
and we can have multiple controls, and (inaudible) that having controls
of what is going on with other drugs used to treat the same disease, so
that you understand where you are.

And it is a way of finding a (inaudible) for the event across other
drugs that are being used (inaudible), and move away from the
(inaudible) comparisons, and we need to look at the big picture and it
is not two dimensions. It is multi-dimensional. But we are going to do
the testing together.

DR. ALMANOFF: The working group actually has a number of initiatives,
and we will be studying a number of factors that affect performance, and
what is the best database, and what are the best stratification schemes,
and what are best practices for this tool.

A couple of points. I wanted to reiterate what I said this morning about
the fact that when I hear the term false positive, I get a little bit
nutty, because I just view this as a positive signal as \-- you know, as
a \-- I\'m sorry, a positive score in a data mining run.

I just view that as an intermediate step, and I view it as an alert that
I should pull the cases and use my clinical judgment, and actually pull
cases of related events, and maybe even look at related drugs.

And then use all of that information and bring it together as an
intelligent clinician to make a decision. So again I think that the risk
of pulling these things out is very low.

And in terms of false negatives, maybe Anna has some thoughts on that,
but \-- well, go ahead.

DR. SZARFMAN: The false positives. There were examples, for example,
this morning about drugs, prozac, related to suicide. Well, suicide
events that are suicidal ideation, et cetera, it is an indication where
prozac is being used.

The identification of adverse events, the data mining finds them because
they are called adverse events. If we are looking at frequency counts,
we will find them also.

And this is not bad, and the data miming is behaving, and if we call
them adverse events, or are defined as adverse events, then the
interpretation is important. You drill down \-- for example, with
pemoline, I have a new signal for multiple sclerosis, and I drill down
and this is a new thing now.

The multiple sclerosis is an indication, an off-label indication, and
where it is being used (inaudible) patients with multiple sclerosis. You
drill down in the media until you realize what is going on.

Then maybe we need a protocol on indications or indications in a
different field is solvable. The negative signals. Okay. This is very
interesting. We are working with the same numerator.

When people are analyzing reporting rates, we are working from the same
database, then (inaudible) would agree that something somebody can call
a signal from data mining after looking and doing multiple comparisons,
and we need to move and when we are comparing drugs, we need to look at
the overlapping (inaudible) limits, then we are now using the tools to
be able to do it systematically.

But when you know that when you are looking at the ones that are bad,
the drug in question, and you are looking at the ones that are the same
as the drug in question, and you look at the ones that are below the
drug in question, and you see where \-- because you can do multiple
comparisons, that when we are looking at the (inaudible) output, and you
have multiple comparisons, you know (inaudible) to do multiple
comparisons when they are trying to find a (inaudible), then you can do
these sorts of things.

And if we are working with the same evidence, and we are working with
the same numerator data, and essentially with the same denominator, then
these things are not at issue.

You know, what is the background that we need to use, and if we are
using external sources of data that the location of the event in
question was found very differently \-- and maybe it is a paper, a
published paper that assumes the expected value of acute liver failure
across the board is one in a million.

But maybe for a specific disease it is different, and this is where data
mining can help you within the same database, because if I know
something is about, I touch the data, and I know that things can go
wrong, and if you are using data from different sources, and even the
management of the data is different, and it can make a difference.

That\'s why we stratify by time, because it is different, like in 1968,
and this is now, and this was the problem with the false positives and
data mining thought that they were not stratified at all.

And then they came to me and they showed me something else, and I knew
beforehand that you are stratifying or having false positives.

DR. ALMANOFF: Just three more quick comments. First of all, in terms of
false negatives, another thought that occurred to me was that I think
that we cannot rely always on this tool to find every single signal of
importance, and as somebody said earlier, one or two cases that are very
compelling might be critical.

So I think that for the way that we think it is good to look for signals
is to use this to increase frequency, but if you see a case or two of
Stevens-Johnson Syndrome, or interstitial nephritis, or something that
is clearly drug related, you know, that is the way that you deal with
the question of false negatives.

If it is medically important and likely to be drug related, then you can
cover that by simply using your medical knowledge. No method is perfect.
Two quick things about false positives.

One thing Anna has alluded to is the fact that there can be confounding
by age and gender, and so internal stratification of the data really
does help eliminate false positives, and we can talk about that off-line
if folks are interested.

The other thing that we have discovered as a source of \"false
positives: is the phenomena that we described as an innocent bystander
phenomenon, and the way that you deal with that is that you might have
two drugs that are used a lot together, and you can actually subset
patients who are on one drug, or just the other drug, and on the
combination.

And as you actually go through those exercises, you can often comb out
what is going on. So I hope that is helpful.

DR. YONDRIN: Sam Yondrin, Millennium. This is just a different slant on
data mining. I know that when we talk about data mining currently the
focus is on market put out data, but data mining has also been applied
to clinical trial data, and primarily here, and which is where I have an
interest, is how do we apply data mining methods, or at least develop
these methods, such that we could probably identify predictive
algorithms using statistical packages.

And I think that this is where I think identifying or developing
clinical \-- you know, predictors, probably is within reach compared to
pharmacogenomics, which is where my colleagues talk a lot about.

But I think also in parallel that there may be a utility for clinical
predictive algorithms. What do I mean? So if you were able to identify a
patient, say, that was a male, and of a certain age group, and received
the same concomitant medication, and had the lab parameter value, then
they would have a certain probability for an adverse outcome.

So I would just like to understand if there is any efforts in this
direction, because in terms of data quality, we have a better change of
achieving data quality in pre-marketing.

DR. ALMANOFF: Yes, this is an important area, especially because of risk
management and actually the working group has a subteam that Christy
Chaung-Stein is going to head up to explore ways that one can look at
NDA data.

There obviously have been a lot of challenges in sort of pooling lots of
NDAs, and getting access to that type of information, but the view has
been that any sort of \-- and as Sidney says, harbingers that can be
seen as smoking guns in clinical trials could be useful certainly for
setting up pharmacovigilance plans.

I know to date that Christy has done a little bit of work with this, and
I know in one large NDA that she looked at actually didn\'t find
anything, which I guess is a good thing. I don\'t remember what drug it
was and how that turned out.

But the other question though is that when we start digging into these
NDAs, and how does that impact on the sort of more frequented safety
profile, and how do you sort of reconcile the two, and will that become
maybe a concern as companies move forward.

Do they maybe feel that a more exploratory analysis is not something
that they want to get involved with. So we will have to see how that
goes. Thanks.

MS. HOSTELLEY: I wanted to address something that has to do with
quality, and also has to do with the nature of some of the events that
go into adverse or post-marketing adverse event databases, and in which
we make the assumption that they are unsolicited and that they are
therefore potentially related.

The example of the MedWatch form sent in by the physician is something
that we should always assume a presumption of a relationship. In certain
areas, particularly in oncology transplant medicine, and certain enzyme
replacement therapies, the sponsor has extensive long term contact with
groups or representatives in medical information, and has extensive long
term contact with the treating physicians.

And under those circumstances you collect information. An example, for
example, in a transplant situation, everything went fine. You are
looking good, and there was a wound dehiscence, and he had to go back to
the OR.

So that wound dehiscence goes in under those circumstances as a
spontaneous report. They really can\'t be viewed as unsolicited reports.
There is really no way of distinguishing that in the data that are
submitted.

In fact, often the situation, and I think Sidney sort of alluded to,
where a physician will adamantly deny that they submitted an adverse
event report, and you place on them report forms, and send it to them
and so on, when they don\'t believe that this is what happened.

But because of compliance concerns, that is what you do. Now, as far as
I can see, in each of these there is a field that you can use to reflect
that, and in our database we do preserve it. But I don\'t see how that
comes in to the information that the FDA has, except perhaps as a
comment in the narrative, which is essentially lost.

So in thinking about data quality issues, I wonder if there is any way
that we can deal with that kind of information. Thank you.

DR. GALSON: Did anybody want to comment on that?

DR. BEITZ: That is certainly an issue that we have dealt with at CBER,
and I don\'t know whether there is any kind of a field or a statement in
the narrative. We are aware of certain companies that do have certain
programs, where there is a lot of active calling and talking to centers
and hospitals where certain treatment is delivered.

And we understand that under those circumstances that they are going to
call and say \-- and this is sort of maybe a combination of finding out
really what is going on and also to connect it to marketing to some
extent.

But if there is all this discussion, you know, they will say how are
things going, and somebody will mention something that they might not
have written in their report, and then it has to be put into the
database.

And so we are aware when we look at certain products that we understand
that there is going to be an elevated rate of reporting because of these
programs.

But I don\'t know that there is any systematic way that we have of
knowing across the board for which products there are such programs.
Miles, do you want to \--

DR. BRAUN: Well, I think there is a box on the MedWatch form that
relates to consumer reports, and so that is one potential way that those
could be subsetted out, but that is definitely an issue, and sometimes
companies actually have at least in biologics had \-- they had a policy
of submitting such reports, and then they realized that for patient
support programs that according to the guidance they are not necessarily
obligated.

I am talking about patient support programs now to support reports to
the FDA, unless they believe that there is a reasonable association,
causal association with a product.

And so you can have even within one product over time a change in the
reporting for that product, and you could certainly imagine that there
is a whole bunch, a large number of reports, that would have come into
the agency in the early period, where everything was sent in.

And then when they realized that there was a guidance that could be
applied, and they did not have to send those in, that there would be a
different \-- that many of those earlier reports would no longer be sent
in.

And then if you stepped back and said, well, if you are using one of
these data mining tools that really look at proportional rates, you
basically could change the proportions radically even though it is the
same product, and really with the same risk profile, artifactually
change those proportions, because a big segment of the reports is no
longer sent in as part of the database.

So I like to look at data mining as kind of a chainsaw. It is a good
tool that can do important work, but it can be dangerous if not used by
a trained operator.

DR. GALSON: Is there follow-up to that?

DR. GOLDSMITH: If I can add a point to that question, and Wendy, I am
sure that you probably remember during our discussions of V2A, when we
were actually together. There was a discussion of what you do with a
clinical trial patient who has completed his clinical trial data.

And indeed after the patient has completed his clinical trial data, you
go back and you touch base with the investigator, and the investigator
says, oh, by the way, this patient had 2 years after the trial was over,
and that was considered not to be a spontaneous report in E2A, and it
would be subject to analysis causality according to the rules for
clinical trial data.

DR. SZARFMAN: Of course, you can do a specific report for a (inaudible)
coming from a single source. Then you can separate the other options
that at a grade level you can look at the reports and photographs
submitted by the applicant, and the reports for the same drugs omitted
by other sources. There are ways of understanding what is going on.

You know, we have many possibilities, and there are many tools, and a
computerized environment can let you do things that before was
impossible.

DR. GALSON: Min, did you have a comment or anything that you wanted to
add to that; that or the previous point?

DR. CHEN: On the previous point, yes. I think that without going into
any specifics about that report, I just want to say, yes, almost or a
lot of reports from the spontaneous resource.

We know that they are solicited, and they are solicited either from
studies or for disease management purposes. But I think that spontaneous
reports is really \-- that is the beauty of it. That it really
accommodates all kinds of reports.

A lot of information flow in, but not necessarily that you can use every
report for every purpose that you are looking into investigating. So you
have to be careful how you use the information.

Although you should allow a specific field to capture some information,
we know that will help us to identify. These reports may not be directly
related to the drug. But we know also that there is always a kind of
safety net somewhere, either within a company or in the FDA.

When we look at certain issues with a certain drug, we will look at the
reports, the case reports, and then look at the narrative. If it is not
in the E2B field, we will look at the narrative and then decide whether
there is a reasonable possibility that the drug has something to do with
the event and make your best judgment to use that report as part of your
case series analysis.

So I am not that worried about the source of the information, and really
the quality of the reports is the key here, and we are not going to get
all kinds of reports with all kinds of efforts to gather the information
to help us for every instance of the suspicious.

But I think that the SADR rule, the proposed rule, dictates that really
for those serious outcome events that the company should make more of an
effort to append additional information to help us out on whether there
are recognized events so that we have the information to analyze them,
or for those recognized events with serious outcomes and increased
frequency situations.

And that we can use the quality information that you obtain from the
initial reporter to help to understand more about this recognized event.
So there is a priority issue when you look at the whole database.

It is not that we don\'t encourage everybody to gather all the
information for every case, but we have got to put our priorities there
on how to use the information, and then we know how to gather the
information for our own use and for everybody\'s use.

So that is probably the most important message that I feel about from
the reporting system. And data mining has all its uses, and we have
evaluated so many situations in the past, and it is very useful.

But the FDA database is so big, and it has 30,000 \-- I\'m sorry, almost
3 million records, and from day one, 1969, all the way to 1997, it was
coded in COSARS, and there were only four terms to be coded for each
report.

And we know that each case report can go over 40 or 50 events. So the
limits are already there to capture quality of events on those older
reports. And from 1997 on, they are unlimited events coded MedRA in
those reports and with E2B element specifications.

So there is a whole horizon of new things going on since 1997, and we
have this whole data mining technique applying to the whole thing. And
we have to be careful what we are getting into the system, and what we
are getting out of the system.

And when we validate it, we have to think about all these caveats, all
these validation, too, on whether it is there to help us out to
understand the signals.

So I am really happy to see today that we had so much discussion about
data mining, because we are sometimes in a dilemma saying how do we use
these scores in our data lives, but we can\'t really ignore them, and we
are trying our best to understand it, and see how we can best use it.

DR. SZARFMAN: We stratify by year, and by age group, and by gender, and
by report source. Then we are taking care of that problem, and
(inaudible) device, the whole scheme, and I think that we need to look
at the big picture, and I am willing to help as much as I can with
interpretation and implementation.

DR. GALSON: Victor.

DR. RACZKOWSKI: The discussion about causality assessment makes me
wonder whether this might be a time to think about a shift in paradigm.
Someone had suggested a possibility that instead of classifying a drug
as an adverse events as possibly, probably, or unlikely related to drug,
rather modifying the MedWatch form along the lines of do you think this
drug \-- what do you think this adverse event is due to.

Do you think it might be due to the underlying disease, drug A, drug B,
drug C. Might it be related to background incidence, and perhaps
capturing that sort of information on a MedWatch form would be more
informative than these attempts to causal associations for individual
adverse events.

DR. NELSON: I have two comments regarding spontaneous reports.

DR. GALSON: Could you identify yourself.

DR. NELSON: Oh, Bob Nelson, consultant. I have two comments. There was
discussion this morning about how to best collect quality data for
specific organ cells, and I really need to put a reference on the
record.

There is an excellent book that most of you know about that Christian
Beneshou (phonetic) wrote about a dozen years ago, and I would have to
give you the exact title, but it is something like the detection and
diagnosis of adverse reactions.

In there, he has good logic strings of how to think through liver
toxicity, bone marrow dyscrasia, et cetera. And there are data
collection instruments that have been validated, and these are very,
very strong tools to help you guide collection of the correct kind of
data elements to help you determine whether or not the drug was likely
to cause that reaction.

It was a very powerful tool, and that if used correctly, will trump the
idea of using an algorithm later on to determine whether it is possible
or probable, because you will have such strong data.

The second point about spontaneous reports regards the \-- well, I just
wanted to put that reference on record, because it is a very great text,
and I think that most people know it.

The other one about reports that are not really spontaneous. There is of
course a need to be a company comment field, and the CIOMS comment. It
is good pharmacovigilance practice for a company to state their opinion
on that case.

This was a non-solicited or a solicited report that we don\'t believe
had anything to do with the drug, period on the bottom of the report.
You may in fact submit it because you feel that you need to submit it
because of regulatory compliance, but you are also \-- it is incumbent
on you with good practices to state your opinion on the value of that
case.

And as Min Chen just said, when they do case series analysis, they just
throw those out. So those are my comments. Thank you.

DR. DANA: It is on a different topic, and it is when you get ready to
talk about registries. So go ahead.

DR. GALSON: Okay.

DR. KAHN: I was going to come back to Dr. Raczkowski\'s comments on the
issue of causality assessment and also just Bob Nelson\'s. I am not
absolutely sure that I know what value causality assessment on
individual cases actually adds at this point, because causality is like
pornography. You know it when you see it, but it is very hard to define.

And typically the active pharmacovigilance professional will have
(inaudible) and say I don\'t like the look of that, and they will go and
explore for similar cases.

And at the case series is a far more useful tool for traveling to assess
causality than individual cases, and of course in the United States
currently we do have the FDA disclaimer for legal protection, and if we
were being asked as companies to add causality assessments to individual
cases, there would probably have to be some type of legal relief that
goes with that. So if I could just add that to the causality discussion.

DR. YODRIN: Sam Yodrin, Millennium. Just a follow-up on the issue of
causality. I think to really have clarity as to the role of causality
assessments with ICSRs, we need to step back and ask ourselves what is
the question that we are trying to answer.

Is it firstly does the drug cause this ADR, or does the drug cause the
ADR in this patient. And they are two different questions. And the
initial question that we are faced with is the broader question as to
whether the drug causes the ADR.

And the answer to that question is not from individual case causality
assessments, and that\'s why when you have regulators like Germany or
Japan asking for company causalities, it is misguided, because in the
process of signaling, the first question is does the drug cause the ADR,
and that requires a population approach.

And actually Judy Staffa at the FDA has communicated this very well in
several DIA sessions, and even on the FDA website. There are nice slides
there that clearly communicate this concept for anyone who doesn\'t
understand it.

So individual case causality really has no place in terms of signaling.
Now the case here is that one actually develops and that is where
ultimately we need to address that question as to whether the drug
causes the ADR.

Now, what I haven\'t heard so far today is that a lot of the time we
actually achieve that concept of threshold, and not based on some very
specific or direct causality assessment.

But from things like the clinical pattern. You know, either from time
dependency, or time to onset, or clinical features of the case. And a
lot of the time I find that is where the signal really lies, and this is
where the case disposition is very important, how you achieve that case
series that you now say that this is the analysis data set.

We have heard during this workshop how several companies exclude
relevant individual cases from that analysis data set simply because
they look at risk factors involving patients, and they interpret that as
alternative etiological factors.

And then there is also the misuse of risk factors which are confounders,
but all these patients who have risk factors, there is a pattern there.
The question is there is any sub-population or subgroup at risk for the
event due to the drug.

And so it is primarily \-- I think the issue is one of pattern
recognition, rather than some sort of algorithm that tells you that,
yes, there is a causality threshold there.

I am not dismissing that, you know, in totality, but a lot of the time
we recognize signals simply based on pattern recognition.

DR. FENISHELL: Bob Fenishell, Washington. We have heard that this has to
be determined on a case-by-case basis, and I think that most people have
come to roll their eyes at that, because it seems to be
non-communicative.

Nevertheless, in this case I think that is quite applicable and I guess
the analogy to make is do we ask for causality on the efficacy side.
When we were receiving hypertensive trials, did we ask the sponsors,
well, let\'s look at this patient.

Do you think that the blood pressure change was attributable to the drug
in this patient. What about this other patient. So we never ask that. It
seems senseless, in part because the variance \-- or what we were trying
to derive from a large population of patients was a result which could
not be derived from one in the period of observation that we had.

And in a different trial design it could have been. There are trials to
show the efficacy of antihypertensives, for example, in single patients.
It can be done.

But that is not the way that we did it, and that is not the way that
most people do it in hypertension. Now my understanding, and now I am
going to areas where my clinical experience is much less.

But my understanding was that in the anti-infective area, even within
the agency, even in areas of efficacy, individual patients were being
followed in a sense that we never followed individual patients in
anti-hypertensive trials.

That an application for an anti-infective product is really a mass of
little micro applications, one of which might be does this drug really
work for serratia infections of the central nervous system where there
is no localized abscess.

Well, there are probably about five people who have ever had that
disease in the history of the world, and the sponsors managed to get
three of them into the trial.

And so the real question is for each of those patients, gee, did that
patient really have that. Is this confounded by other things that were
happening with the patient at the same time because he was very sick,
and he was getting six other treatments and so forth and so on.

And so there was patient by patient causality consideration, which was
necessary. I think it is going to depend on what the ADR is, and what
the population is, and what the intrinsic variance is, whether one is
able to approach the question as a variance reduction through sample
size operation, or whether one is groping as one often is in safety
matters with individual cases that must be individually massaged.

So there is no simple answer and I hate to say it, but we are back to
case by case.

DR. SZARFMAN: Except for a famous case of interaction between
(inaudible) that was a single patient, and there was a blood level that
was too high, and the patient had no risk factors. It was a young woman
that received the two medications at once.

And this brings up that I was a little disappointed that being a
clinical pathologist that the comment of (inaudible) that we couldn\'t
use blood level information, because I thought that this could be a very
objective way of \-- we have done it for toxicity for certain drugs.

And I don\'t think that developing blood tests where you can measure
blood levels will be really expensive.

DR. GALSON: Thank you. Are there any other comments from the audience?

DR. DANA: Can we switch gears to registries?

DR. GALSON: Absolutely.

DR. DANA: I am Joanna Dana in Regulatory Affairs for Amlin
Pharmaceuticals. Having been in this industry for a number of years,
probably more than most of you around here, I kept seeing patient
registries, patient registries, meetings on patient registries.

I was involved with Accutane and some of the early issues with that, and
as I started to go to meetings to get a better appreciation of what was
meant by this patient registry comment, two statements came out that
were somewhat problematic.

One, 65 percent of all recently approved drugs now have registries; and
the second one is that 35 percent are FDA mandated. Now, I don\'t know
if this is a true statement or not, and it came from a meeting that was
entitled, \"[Patient Registries]{.underline}.\"

And it may have its own sort of internal self-fulfilling purposes for
those that spoke at that meeting. But the issue in listening to the
various presentations there was that the definition of patient registry
there was quite different than what I see in this document.

And to take the spectra of definitions that are out there for what
compiles anything from what used to be called in a seating study to one
that is a very specific pregnancy registry, and for one that is
therapeutic utilization and call it all the same thing, is problematic.

So I think clarity of that definition within this document would be
extremely useful.

DR. GALSON: Thank you for that comment. Any other follow-ups on that
registry issue or other registry issues? Okay. If not, I am going to
turn the mike over to Julie Beitz for some closing comments.

DR. STEPHENSON: Before you do that, we had a question for the FDA.

DR. GALSON: Okay. Oh, the second question, your second question.

DR. STEPHENSON: Yes.

DR. GALSON: Why don\'t you read it again, unless you want to put it up
on the screen.

DR. STEPHENSON: It had to do with \-- since both FDA and industry tend
to use pharmacoepidemiology data studies, the question was, it would be
helpful to hear FDA\'s thoughts on when a study should be conducted by
the company, versus the FDA, or both.

And then the more specific question is that if the FDA is going to be
conducting a pharmacoepidemiology study on a company\'s drug, are there
any obligations to include the company or communication, or collaborate,
with the company. That was the question.

DR. GALSON: Who wants to go after that one?

DR. RACZKOWSKI: I will go ahead and start. Victor Raczkowski. The
paradigm in general at the Food and Drug Administration is that
typically sponsors perform studies on their products.

And my sense is that what we often times require in those cases,
however, is that we have some sort of \-- and particularly on the
pre-approval side, is access to the data to be able to validate the
results of studies, and efficacy studies in particular.

And my initial reaction to seeing that question is that it seems likely
that in the post-marketing arena as well that generally these Phase IV
type studies on safety issues would be conducted by sponsors.

But I do think that raises the issue of then how do we go and look at
the data, and how do we validate the patient population, and what are
the tools that we need at the FDA if we are not doing the studies
ourselves in order to be able to give us some level of comfort that the
result is as it is stated.

I would also say, however, that this is \-- you know, we are trying to
be somewhat innovative in this area, and we are making great attempts to
see what types of resources and databases that we might be able to
utilize, such as claims databases, in order to answer particular public
health questions.

And Judy can talk about that more specifically, but I think that there
has not been a lot of attention to this necessarily in the past, and so
we are trying to be very proactive in that respect, in terms of trying
to find out how we might be able to use databases that were not
particularly designed to answer public health questions.

But that also requires a certain level of validation, such as access to
medical records, and that sort of thing.

MS. HOSTELLEY: I think the question behind the question was sort of the
awareness that the agency is moving into today\'s world into
consideration of doing pharmacoepidemiologic studies like through
various academic centers, such as through the CERTS organizations, et
cetera.

And so I think that the driver for the question was that apart from
perhaps seeing a listing on the web that this is a study being
undertaken through a grant provided by the agency, PhRMA was asking the
question is it routine to sponsor that at least that you are doing that
since it does involve the sponsor\'s drug.

So that \-- I mean, it is helpful to have a dialogue with the sponsor in
order to ensure that the sponsor is not sitting there thinking about
doing the same type of study. So that we can figure out a way to better
collaborate and more effectively utilize resources.

DR. GALSON: Did you want to add something to that?

DR. RACZKOWSKI: I just wanted to say that I think that is a fair
comment.

DR. BEITZ: What I actually was going to say was that at least some of
our resources that we have through cooperative agreements, for example,
are fairly limited in scope, and so I think the kinds of studies that we
might design to do in using the funds there, versus the kinds of studies
you might be able to conduct are quite different.

But there is no doubt that some dialogue might be useful.

DR. BRAUN: I just wanted to mention from the observational studies, that
if you look certainly in the cancer arena and to some extent also in
cardiovascular disease, it is well known that the studies don\'t agree
all the time.

In fact, there is a large amount of disagreement among findings of
independent studies, and I think that is an important consideration when
we talk about the numbers of studies in this area.

So if there is an important question one could look at it as actually a
strength that there are two separate groups looking at it independently.
And so I am not sure \-- you know, there is a concept of \-- some might
say that it is not cost effective to duplicate efforts.

In my view, it is not duplication, and in safety systems redundancy
sometimes is actually a good thing to have. So I see some merit in
actually doing that. So independently and actually what might seem in a
duplicative way. So that was my point.

DR. STAFFA: I think that Miles makes a good point about the \-- there is
not \-- it is not always bad to have two independent studies going on of
the same issue at the same time, because the differences in those
findings can be instructive.

But I also think that it is an intriguing idea to have more dialogue and
collaboration. I know as a reviewer of protocols that come in that we
would like to see some of our comments and issues be incorporated into
the protocols, and I am sure that the same is true, vice-versa.

And I think that we would welcome insights. One of our frustrations and
one of the reasons that we are trying to build more capability to do
studies on our own is that we sometimes find that in collaborative
efforts, when we can\'t touch the data, we can\'t always do the designs
or ask the questions in a way that we would really like to see them
asked. And so sometimes that is really what is driving us to do that.

DR. GALSON: Let me just make one comment on that myself, which is that
clearly we don\'t have an agency policy on this question right now, and
I do think that it would be very valid for us to cover this in the
guidance document, because it clearly is something that deserves
clarity, in terms of know where we stand on this question.

So we will definitely take that into consideration. Are there any other
follow-ups on that issue? Any additional issues?

MS. BORSTU: Yes, maybe one. I will say that I think \--

DR. GALSON: Just identify yourself.

MS. BORSTU: Maria Borstu (phonetic), Organom International USA. My
question pertains to the very far end of the third concept paper, I
think, but it links into this. We were talking about multiple parties
studying the same issue, and consistency of results.

But what we know from history usually there isn\'t consistency in
results, especially if we have more than just the FDA or the government,
and the PhRMA industry is studying the issue as soon as you have a \--
how do you call that \-- academia, government, industry, studying and
sometimes even duplicate partners.

Are there any thoughts or have there been any thoughts during the
discussions about guidance on interpretation of the findings from the
different sources from the different studies done by the different
parties using different sources.

I was thinking that there would be a need somewhere along the line for
norms and criteria as to put some weighting on the various studies that
we all are going to do. I was just curious whether that has been on the
table already.

DR. GALSON: Judy.

DR. STAFFA: Actually, I think that is an excellent point and I think
that is something that we are planning to tackle when we get to the
guidance stage.

Clearly we are going to have to at least lay out some criteria that
tries to speak to that issue, because it happens routinely.

DR. UNGER: Ellis Unger from CBER. This is a miscellaneous comment. At
the end of the day, irrespective of how a safety signal is generated,
whether it is data mining or what have you, we have to apply judgment.

And the quality of the judgment is inextricably tied to the quality of
the data, and at the end of the day the FDA reviewer has maybe 15 forms
on his desk or her desk.

And the quality is often marginal, and sitting here I keep thinking
about the potential of the internet for collecting the ADRs. If there
were an interactive system developed, whereby what is the patient\'s
problem, and check box, liver problem, and then it went through an
algorithm and took it all the way out to did the patient have a biopsy,
yes, no, don\'t know, et cetera, obviously it would take time to
develop.

But eventually we could get some very good data I would think from a
system like that, and I will just put that idea out there.

DR. KAHN: Can I answer that to some degree? Without naming names, which
is not appropriate here, in fact such a system has been developed by an
independent private vendor and I am not sure to what extent it has been
tested and validated.

But I have actually seen a demonstration of exactly such a system.

DR. GALSON: Okay. Thank you. Anything else? All right. I would like to
turn the mike over then to Dr. Julie Beitz.

DR. BEITZ: As you have heard today, there are many challenges to be
faced in carrying out good risk assessment based on observational data
sources. Some of the more thornier questions that came up today revolved
around how can the quality of adverse event reports be improved, and
what conclusions can be drawn from observational data sources given the
inherent limitations of these data.

And what more can be done to enhance our understanding of safety signals
once they are identified. On behalf of the members of Group III, the
CDER/CBER Pharmacovigilance Working Group, I would like to thank all of
the attendees today for their well considered comments and questions.

We plan to consider all the comments that we received today, as well as
those submitted to the docket when our prepare our draft guidance
document.

I also would like to thank all the members of the steering committee and
of the working group itself for their scientific input on the concept
paper thus far.

But I would also like to thank Lee Lemley and Diane Erlich for
organizing a very successful public workshop. And now I turn it over to
Dr. Galson to close.

DR. GALSON: Thank you very much, Julie, and I want to just very quickly
thank a number of people in particular today, the working group on
pharmacoepidemiologic. You have done a wonderful job. The document was
excellent, and we had good discussion today.

Also the members of the executive oversight committee who participated
today and throughout the week. Specifically, I want to thank the project
managers for each of the three working groups. These products and this
meeting wouldn\'t have gone so well if it was not for your hard work.

And to all the people sitting up here today, and all the audience for
your participation, thanks very much. Please submit written comments if
there are things that you feel haven\'t been covered.

The docket is going to be open and we need those comments for us to
produce high quality draft guidance documents over the next few months.
Thanks again.

(Whereupon, at 3:26 p.m., the meeting was concluded.)