+-----------------------------------------------------------------------+
| > ![](media/image1.png){width="1.070138888888889in"                   |
| > height="0.5548611111111111in"}**www.lilly.com**                     |
+-----------------------------------------------------------------------+

Eli Lilly and Company

Lilly Corporate Center

Indianapolis, IN 46285

U.S.A.

+-----------------------------------------------------------------------+
| > Phone 317 276 2000                                                  |
+-----------------------------------------------------------------------+

Dockets Management Branch

Food and Drug Administration

5630 Fishers Lane, Room 1061

Rockville, Maryland 20852

Comments to: Docket No. 02N-0528, CDER 200311.

Risk Management: Public Workshop

68 [Federal Register]{.underline}: Pgs 11120-11121 

**From: Eli Lilly and Company**

Eli Lilly and Company (Lilly) appreciates the opportunity to offer the
following comments to FDA Docket No. 02 N-0528, CDER 200311, Risk
Management: Public Workshop. Lilly compliments the FDA on the quality of
the concepts presented in the three papers and on their inclusive
approach to the development of final guidance around safety-related risk
management. Lilly agrees with and supports the comments submitted by the
Pharmaceutical Research and Manufacturers of America. We will not
duplicate their line-specific comments and are submitting only comments
that are not included in their response. Our comments consist of general
comments on the topic, followed by general and specific comments for
each of the concept papers discussed during the three-day public
workshop sponsored by the FDA on April 9-11, 2003.

**General Comments on Risk Management**

The decision to use pharmaceutical products is driven not just by the
consideration of their safety, but also by the consideration of their
potential benefits. In addition, the decision to use prescription
pharmaceutical products is made not only at the societal level by
regulatory agencies but also at the individual level by prescribers and
patients. Any approaches to risk management, therefore, should take into
account both the right of individuals to participate in this decision
making process and the diversity among individuals in how they will
value a product's benefits and risks when making their decision. These
considerations require that we use the best science available when
developing the information to be used in decision-making, and also that
we maintain flexibility in how we practice risk management across
different drugs and even across different patient groups for a single
drug. The emerging science of risk management will be well served if the
final guidance effectively balances the definition of clear criteria
with the needed flexibility in their application.

In the current debate around risk management the focus has been on drug
safety and minimizing the potential for harm to patients, both
individually and collectively. This approach has presumed that the
universal preference is to avoid all potential for injury, even at the
expense of forgoing a potential benefit. This ethical position is rooted
in the first principle of Medicine: "Primum Non Nocere". This approach
also reflects the historical practice of medicine in which physicians'
choices of therapies were limited in both number and potential
effectiveness. However, in the last 50 years medical science, and
especially pharmaceutical science, has made tremendous progress and
dramatically expanded the physician's armamentarium in both number and
potential effectiveness, but with some accompanying potential for harm.
With this marked increase in the potential to provide clinically
significant benefits to patients has come the need to be more adept at
assessing the balance between the potential for benefit and the
potential for harm, from the perspective of both society (the target
population) and the individual patient. Regulatory agencies have dealt
mainly with the former and practicing physicians with the latter. The
pharmaceutical sponsors have participated in both debates and provided
data to be used in making both types of product-specific decisions.

Concurrent with the scientific advances of the last fifty years there
has been a prominent increase in the self-declaration, and societal
perception, of the empowerment and activism of individual patients with
regard to the direction of their medical care. Patients are seeking more
and more to be active, rather than passive, participants in their own
care, including making decisions concerning the nature of their care.
Practicing physicians have long known about the individual differences
between patients and have tailored their selection of therapies
accordingly; this is the "art" of medicine. Now patients are in a
position to directly express their individual valuations of the
potential benefits of a given therapy and their willingness to accept
accompanying potential for harm. This helps the individual physician
practice the art of medicine one patient at a time, but complicates the
societal task of determining the limits for acceptable risks by tacitly
nullifying any approach that does not take into account the broad
diversity of individual valuations. Therefore, any approach to
pharmaceutical risk management that is driven solely by the desire to
reduce harm to some absolute minimum will be unacceptable to society at
large since it will ignore the desire of individuals to be involved in
decision-making for their own care, and also ignore the broad diversity
of their value judgments.

An acceptable approach to risk management needs to:

-   Respect the right of individuals to participate in decision-making
    concerning their own medical care,

-   Recognize that while individuals should consider potential for harm,
    potential for benefit, and potential cost when making decisions,
    actual behavior has very complex origins and is often driven by
    avoidance of negative consequences,

-   Account for the broad diversity in how individuals value each of
    these variables when making their individual decisions and
    understand that there cannot be a one-size-fits-all solution to risk
    management.

With this understanding it becomes apparent that risk management is all
about facilitating individual value judgments. In this regard it has two
primary roles:

-   Setting societal boundaries for acceptable potential for harm (upper
    boundary) and potential for benefit (lower boundary) for any given
    therapeutic intervention

-   Assuring that individuals have access to all of the information
    needed to make fully informed decisions.

In a capitalist economy the marketplace will reflect the collective
judgment of the individuals concerning the appropriate valuation for the
cost of the therapeutic intervention.

Each of the primary roles of risk management has several aspects.
Setting societal boundaries includes:

-   Detecting potential risks and benefits,

-   Evaluating the probability of the risks and benefits, including
    identification of any factors that may influence the probability for
    a specific individual,

-   Considering worst case/best case scenarios to determine the greatest
    potential for harm that would be tolerable to a reasonable minority
    of the target population and the least potential for benefit that
    would be attractive to a reasonable minority of the target
    population.

Assuring access to information involves:

-   Determining the information needs of prescribers and the target
    population,

-   Collecting, analyzing, and interpreting the data needed to meet the
    information needs of prescribers and the target population,

-   Packaging and distributing the information so that it is readily
    accessible and understandable by prescribers and the target
    population (while also realizing that information alone does not
    necessarily impact behavior). Professional labeling and patient
    package inserts should be the primary method of communicating this
    information.

Each of these components contains considerable challenges for
regulators, pharmaceutical sponsors, practicing physicians, and other
members of the healthcare delivery system. Our current environment
includes many components that are each actively working to address one
of these aspects of risk management. What is missing is the level of
coordination needed to assure that each individual prescriber and
patient is able to make a fully informed value judgment concerning their
acceptable balance of potential for benefit and potential for harm, and
their acceptable limit for the cost of a therapy that fulfills their
needs.

The current debate would better serve patients by broadening the focus
from harm minimization to better facilitation of informed valuations.
This approach is still in keeping with the spirit of "Primum Non
Nocere", but now includes the additional concept that patients can be
harmed as much by denying them access to therapies with some potential
for benefit as by denying them access to therapies with some potential
for harm, and also the concept that patients as individuals have a right
to participate in the valuation of the appropriate balance between
potential for benefit and potential for harm.

In the context of a broader-focused debate, a risk management role for
regulators emerges that has three main components:

1.  Understanding and describing the societal boundaries for the
    benefit/risk balance for a drug candidate.

2.  Assuring that all interested parties have access to the information
    they need to make informed value judgments.

3.  Clarifying the circumstances in which setting societal boundaries
    and provision of information will be insufficient to protect public
    health and therefore warrant governmental intrusion into the
    individual's right to make their own value judgments (within the
    societal boundaries) concerning the benefit/risk balance of a drug
    by placing additional restrictions on access to medicines (and
    applying such guidance to specific drugs). This includes both
    setting the criteria to be used in making these judgments and
    applying them on a case-by-case basis in regulatory practice.

In short, the final risk management guidance should provide a clear
definition of what constitutes a \"reasonable\" risk management program,
including a description of how differing circumstances may influence
what is "reasonable" for different products. The final guidance should
also provide a clear definition of the roles and responsibilities for
all parties involved in the development, administration, interpretation
and on-going assessment of a reasonable risk management program. This
implies that the final guidance should carefully consider the burden
placed on each participant of the health care delivery system. In
addition, the final guidance should be written to illustrate how the
requirements of a reasonable risk management program will be compatible
with the Health Insurance Portability and Accountability Act (HIPAA).
This latter topic is discussed in more detail in the specific comments
for each concept paper.

Comments on "Pre-marketing Risk Assessment" Concept Paper

**General**

This paper states that its focus is on risk assessment during clinical
development, yet the discussion centers on the evaluation that occurs
immediately prior to a marketing authorization decision. Additional
guidance on making benefit/risk valuations during pre-clinical and
clinical development would be useful. Such guidance should be both
general, in terms of methods and decision-making principles, and
specific, in terms of ground rules for discussions between sponsors and
the agency. Such guidance would promote better utilization of resources
by sponsors and delivery of higher quality submissions by sponsors to
the FDA. Higher quality submissions would facilitate agency review.

It is well acknowledged that the pre-marketing database will never be
large enough to capture rare events. Therefore, it will be useful to
focus on what the database should convey, and with what reasonable
amount of assurance. To this objective, section III.A can be best served
if the rationale for the 1500 patients requirement can be provided, and
additional guidance given to determining the "correct" size of the
safety database in those circumstances in which the ICH E-1 guidance
suggests that a larger database may be needed. In addition, the final
guidance should describe conditions under which a smaller database would
be sufficient, along with the associated rationale.

Many exploratory analyses were proposed in the concept paper. We hope it
is understood that not all analyses are expected of all product
candidates. Depending on the situations, one type of analysis will be
more informative than another. The choice on what analyses to conduct
needs to be based on the need and sound scientific considerations. The
final guidance should address these issues.

Sample size becomes an issue when we perform numerous exploratory
analyses of the data. If a signal exists, many subgroup analyses might
not have adequate sample size to detect it. On the other hand, false
positive rate can be greatly inflated when we conduct many subgroup
analyses. It could easily become an unmanageable task for both sponsor
and reviewer to investigate every "positive" finding without regard for
its clinical plausibility or significance.

One additional question concerning safety data from pre-marketing
studies is how much, if any, statistical testing should be included in
an individual final study report. We don't expect the guidance document
to provide explicit rules. However, some general guidelines would be
very helpful.

It would also be helpful for the final guidance to reference other
existing (or pending) guidance(s) as appropriate, for example, the
guidance on QT assessment and assessing the need for clinical studies
based on pre-clinical information.

# *Specific*

-   Line 25: We are concerned that the word \"comprehensively\" may be
    interpreted by some as finding a definitive answer to every specific
    and conceivable safety concern before approval. Such an
    interpretation would not be in keeping with the general thrust of
    this concept paper. An alternative would be to change the sentence
    to something like \"reasonably describes its safety within the
    limits imposed by the size and scope of the clinical development
    plan\".

-   Lines 29-31: We agree that FDA and ICH guidances address
    pre-clinical safety assessments, and that we cannot ignore the
    linkage between human and animal data. However, some circumstances
    would benefit from additional guidance; specifically, if there are
    positive signals in animals but not in humans, what are the
    requirements for post-marketing risk management?

-   Lines 79-83: We don't believe there is a need for a fixed minimum
    size of the safety database for "serious or life-threatening
    conditions". For such conditions, the adequacy of the submission
    database depends largely on the efficacy attainable. If the benefit
    is very high, there will be a higher tolerance for risks (known and
    unknown). As a result, the database might be of a limited size since
    it does not take too many patients to demonstrate a high efficacy,
    yet still be able to adequately characterize the acceptable risks.
    On the other hand, if the efficacy is moderate or low, one will need
    more subjects to better characterize the acceptable risks.

> For drugs for less serious conditions, the minimum size of the safety
> database may be driven by the desire to rule out the occurrence of
> serious adverse events (expected or unexpected) at pre-specified rates
> that would lead to an overall negative benefit/risk assessment. In
> fact, an a priori assessment could be made with the FDA that would
> allow pre-specification of the degree of efficacy required to mitigate
> safety concerns of various magnitudes. Such an assessment could help
> guide the sponsor through clinical development. We suggest that the
> FDA consider including the possibility of such an assessment in their
> final guidance.

-   Lines 92-94: It might be hard for a drug with a narrow therapeutic
    margin to have a substantial number of patients exposed above the
    highest proposed doses. What is the definition of "substantial"
    number of patients? In addition, if the high dose in a multiple
    doses phase III study is picked to be the commercial dose, then the
    submission package will not have many patients exposed to doses
    higher than the one recommended in labeling.

-   Lines 116-121: The justification of the need for a larger safety
    database based on the criteria described here highlights the
    inadequacy of current methods for quantitating the benefit/risk
    valuation. Will FDA be involved in the active promotion of research
    to develop improved benefit/risk valuation methods?

-   Line 126: The needed sample size will depend very much on the
    "pre-specified increase" over the baseline morbidity or mortality.
    The required sample size can be very high if the increase is small.
    Specific guidance on acceptable pre-specified increases in different
    circumstances would be useful.

-   Line 134: The final guidance should clarify what is meant by a "very
    safe alternative", and discuss how the comparative efficacy of the
    new drug will also be balanced against the efficacy of the "very
    safe alternative", specifically addressing how both factors will be
    used to determine the necessary size of the safety database. The
    inclusion of this criterion raises the question of whether the FDA
    has changed the standard for approval to require that a new product
    be substantially better than existing products, rather than
    adequately demonstrating safety and efficacy.

-   Line 155: "...death in patients with Alzheimer's" is not a very good
    example. Most studies of Alzheimer's disease are in patients with
    mild-to-moderate Alzheimer's disease, and deaths are rare, rather
    than "occur frequently".

-   Lines 170-181: It is generally more efficient to explore the
    complete range of doses in Phase I and II trials. A complete
    exploration of the dose-response relationship in phase III would
    unnecessarily increase the exposure of patients to both
    sub-therapeutic and supra-toxic doses. In addition, in some
    therapeutic areas (e.g., ID) it would not be practical to do large
    Phase III, comparator-controlled, non-inferiority studies using
    multiple doses. Also, if a range of doses is explored in Phase III,
    a statistical adjustment for multiplicity will be needed, thereby
    requiring an even larger sample size, but possibly resulting in less
    data on the dose that is ultimately marketed. Different divisions
    within CDER seem to have different attitudes toward whether one
    needs to make such an adjustment and sponsors would benefit from
    consistent internal FDA alignment. Ultimately, the demonstration of
    a dose-response relationship for both efficacy and safety is
    desirable, but this relationship should be described by data
    obtained across all phases of drug development. New guidance should
    focus on clearly describing the limits of the dose-response
    exploration that is expected to occur (and can be reasonably
    achieved) in phase III.

> One alternative may be to consider flexible dosing studies for some
> drugs. Although flexible dosing does not allow for a formal comparison
> between doses, it does allow patients and their doctors to find the
> dose that works the best for the patients (when all things are
> considered). The latter is also the medical practice in some disease
> area.

-   Lines 186-208: We agree that clinical pharmacology studies cannot
    fully describe all possible risks related to interactions. However,
    well-designed clinical pharmacology studies, supplemented with
    observations from other clinical trials, can be sufficient to
    explore most anticipatable interactions. The examples given in this
    section *are* anticipatable for many drugs, and focused studies can
    be designed to explore a new drug's potential for the interaction,
    so that pivotal phase III clinical trials do not have to be more
    diverse (i.e., larger) in order to achieve this objective. In
    addition, testing these interactions in larger efficacy can be
    seriously biased if important factors (e.g., concomitant meds,
    dietary supplements) are not balanced at baseline. Therefore,
    incorporating the exploration of these interactions in such trials
    could markedly increase their complexity, or, at minimum, some
    adjustment for baseline covariates might be necessary.

-   Lines 235-236: Active-comparator-controlled trials will not provide
    an assessment of the background rate of an event, but rather an
    assessment of the rate in patients exposed to the study treatments.
    Placebo-controlled trials would provide an assessment of the
    background rate in the untreated patient population. Additional
    sources for such information include literature reports, census
    reports and broad based health surveys. Such sources should also be
    recognized.

-   Lines 279-298: Final guidance should provide criteria for when
    results from large simple safety studies (LSSS) would be expected to
    be included in a submission, and when such studies could be a phase
    IV commitment as a part of a risk management program. LSSS are
    generally not intended to develop the basic understanding of the
    drug's safety profile (this is done in the context of the other
    clinical trials) but rather to further define a known risk or
    explore an identified safety signal of a potential risk. Such
    studies could add significantly to the development time for new
    drugs, thus depriving patients of the potential benefits of new
    therapies. Careful consideration must be given to both the potential
    risks associated with the new drug and the potential risks
    associated with depriving patients of its benefits. It would also be
    helpful to discuss the design aspects of a LSSS. In general, is a
    LSSS a single-arm trial or a randomized trial?

-   Lines 300-309: Guidance concerning research using reserved
    blood/fluid/tissue samples should include a thorough discussion of
    the measures needed to meet the requirements of the recently
    implemented Health Insurance Portability and Accountability Act
    (HIPAA) regulations.

-   Line 350: Typo -- change "trasmissibility" to "transmissibility"

-   Lines 412-424: Please clarify whether all of these analyses deal
    with time to the onset of the first event occurrence. They are all
    important analyses. An additional analysis for #3 is to summarize
    event occurrence at distinct time intervals. In such an analysis,
    patients are included in the "numerator" if the event was present at
    any time during that time interval. This latter summary will help
    identify events that have high incidence rates early in the study,
    but tend to go away over time. Some adjustment will be necessary to
    take care of subjects who dropped out because of the events of
    interest.

-   Lines 436-437 and Line 449: The analysis of "...event rates based on
    cumulative dose" is similar to that looking at the hazard for the
    event over time if the dosing regimen is kept constant over time.
    Clarification is needed concerning the appropriate use of these
    multiple analyses.

-   Line 454: The number of patients in the border zones of dosing cut
    points will likely be too small in an individual study to permit
    statistically sound inferences to be made, and possibly even in the
    analysis of aggregate data.

-   Line 514-516: Pharmaceutical sponsors have employed various missing
    data techniques to handle dropouts in efficacy analysis. Some of
    these techniques also increase sensitivity for detecting true
    treatment effects. Therefore, the thinking and rationale behind
    these techniques are applicable to safety analysis. One possible
    analysis of continuous safety measures (e.g., ECG intervals, many
    laboratory variables, vital signs) is to use a repeated measures
    mixed model to examine treatment differences at not only the end of
    the trial, but also at intermediate time points. For categorical
    analyses of binary outcomes (e.g., adverse events, categorical labs,
    incidence of extreme lab or vital signs values), at a minimum one
    should summarize the average exposure (in a unit of time) for each
    of the treatment groups. Additionally, one should summarize the
    average number of measures (visits) for each treatment group. This
    can become especially important when dropout rates are very
    different between treatment groups. For example, if Drug 1 has twice
    as many lab measurements per patient than Drug 2 (because of many
    early dropouts for Drug 2), then one would expect more extreme lab
    or vital sign measures in Drug 1 than Drug 2 for reasons of number
    of measures alone. One conservative approach used by one company was
    to assign events to discontinued patients and see if the safety
    conclusions changed in a significant way in a comparative trial.
    Missing data in this case was used to set up boundaries (or
    best/worst case scenarios) for safety conclusions.

-   Lines 526-527: Can what is being proposed here truly be done within
    the MedDRA structure? Methodology research results should be
    available to the agency and industry before this proposal is
    implemented.

**Comments on "Risk Management Programs" Concept Paper**

***General***

As discussed in our initial general comments, risk management programs
should balance the role of government to protect the public health with
the rights of individuals to make decisions concerning their own medical
care. In the performance of this balancing it should be recognized that
using understanding or knowledge of the risks as the outcome of interest
for a risk management program, rather than reduction of adverse events,
may be optimal. This would acknowledge that risk management programs
should facilitate individual value judgments of potential risks and
benefits, rather than just reduce the incidence of adverse events, and
affirm that people have the right to make "unwise" decisions. In most
circumstances our goal should be to make sure people have all the
information they need to make a "wise" decision, regardless of the
decision that they actually make.

Even though risk management programs may have well-defined "objectives",
they cannot be compared to the objectives of a controlled clinical
trial, which have statistically rigorously defined null and alternative
hypotheses, with associated test statistics. The science of controlled
clinical trials has developed over decades of experience. The science
surrounding risk management programs is still in its infancy. The final
guidance should recognize that regulators, industry and academia are all
still learning about how to do risk management.

We support the initiative to have an internet site that contained the
package inserts (PI) for all FDA approved drugs, and the patient package
inserts (PPI) for all FDA approved drugs for which one was available.

FDA has jurisdiction over the development, approval, and distributions
of food and drugs. Recommendations specified in the PI help lead to
well-informed prescribing decisions. There are situations where drug
misuse was responsible for a high proportion of adverse drug reactions
and eventually led to a product's withdrawal. We agree that sponsors can
play a role in educating health care professionals and patients in the
appropriate use of our products. It would be useful to hear about how
the FDA will collaborate with other parts of the health care delivery
system to influence the safe and appropriate use of the products it
regulates.

Because the scope and details of a risk management program (RMP) need to
be worked out for each product individually, presumably requiring
interpretation of the final guidance, the discussion between the FDA and
a sponsor might take considerable time. Therefore, it is important that
the communication line between the agency and the sponsor is open and
fluid so as not to delay the regulatory approval of a pharmaceutical or
a biologic product. Clarification concerning the timing and nature of
such discussions between sponsors and the agency will be helpful. We
encourage the agency to entertain the consideration of risk management
as a topic for all meetings and discussions between sponsors and the
agency, with its actual inclusion being driven by the data available at
the time of the interaction. The final guidance should also describe the
dispute resolution process to be used if the sponsor and the FDA review
division cannot reach agreement on an appropriate risk management
program. We also encourage the agency to achieve alignment among the
different review divisions and with the Office of Drug Safety concerning
the processes for the development of risk management programs and
expectations for their utility.

The term "validation" needs to be better defined in the context of risk
management tools. The degree of validation and the amount of work
required for validation could be a major challenge at a time when a
significant amount of a sponsor's resource is devoted to the NDA/BLA/PLA
preparation and submission. It would be helpful for the FDA to make
available a listing of tools and interventions that it considers to be
validated already and the mechanism by which they were validated, and to
update such a listing as new tools become validated.

## Specific 

-   Line 26: The statement that a product is considered safe if it has a
    positive benefit/risk balance on a population and individual patient
    level fails to provide clarity on whether the FDA will be addressing
    both perspectives in its regulatory decision-making. If the agency
    is planning to include an assessment of benefit/risk balance for
    each individual patient, both studied and anticipated, this would
    represent a departure from the current agency approach that focuses
    on evaluating public health impact at the population level and would
    be an incursion into patient level medical care. Requiring sponsors
    to conduct a benefit/risk assessment for each individual patient
    would require the use of quantitative benefit/risk valuation
    methodologies that are not currently available. We suggest deleting
    the reference to individual patient level or clarifying the intent
    of the FDA in including it here.

-   Lines 28-29: Additional clarity is needed concerning the use of the
    word "minimizing". A "minimum" amount is open to interpretation with
    regard to actual quantitation. Different drugs and different
    diseases may also have different acceptable lower limits for
    achievable targets for efforts to decrease risks. While we agree
    with the concept being expressed, we encourage the agency to express
    its intent with greater precision. Defining "minimize" within the
    context of the final guidance, or more fully describing the methods
    to be used to determine the lower acceptable limit for specific
    risks for specific drugs or diseases, could accomplish this.

-   Line 51 and 63: "Minimize" and "decrease" are used in these two
    places, possibly interchangeably. As discussed immediately above,
    clarity of the agency's intent is needed here.

-   Line 63: Please clarify here whether tools that are designed for the
    purpose of assessing and estimating risks are to be considered to be
    in the scope of risk management programs, or if risk management
    programs will include only tools specifically intended to decrease
    product risk. Discussions in the third concept paper appear to
    indicate that the FDA considers them to be a part of the
    pharmacovigilance plan.

-   Lines 69-73: Defining goals as the achievement of critical
    processes, behaviors and human factors to increase safety raises the
    concern that pharmaceutical companies will be required to intrude
    intimately into managing patient care rather than producing drugs
    and providing guidance on their appropriate use. Managing individual
    patients should be the province of physicians who apply their
    disease, drug and patient knowledge to select the best management
    strategy that fits the unique circumstances of each patient. The
    following scenario is illustrative:

Despite awareness and understanding of a risk management program warning
against such a course of treatment, a physician consults with a patient
who is in the "high-risk group" for a particular adverse effect and
elects to treat the patient with a drug because he/she believed the
benefit/risk balance for the drug was positive for that particular
patient. If the adverse effect occurs with that patient would the
sponsor be liable because its risk management program failed to achieve
its goal of educating the physician? If the sponsor were to be deemed
liable in this situation then sponsors would be forced to police the
physician implementation of the risk management program. The unintended
consequence of such policing of physicians would be to discourage
physicians from reporting the occurrence of adverse events that were the
focus of risk management programs. This would also take away patients'
and physicians' ability to base their decisions on their own balancing
of known risks and benefits.

-   Lines 104-105: The concept paper states that a risk management
    program can be developed at any time during the product\'s life
    cycle. For a risk management program proposal based on data from
    late phase studies, considerable time is needed to fully develop and
    negotiate a proposal, with attendant risk that approval times could
    be lengthened, thereby depriving patients of the potential benefits
    they could obtain from the new product. Clarification is needed of
    the FDA's commitment to commence discussions early in the
    pre-submission period and move them along expeditiously.

-   Line 113-115: We agree with the concept expressed here as it applies
    to the potential or actual indicated population. However, this
    statement should not be interpreted to extend the role of sponsors
    to controlling the actual practice of medicine. In addition, it
    would be helpful to have more specific details concerning the
    criteria for deciding if a risk management program is needed, and
    what type of program would be most suitable.

-   Line 119-122: Researchers need to be encouraged to develop tools for
    benefit/risk assessment, and decision-makers need to adopt these
    tools once they are developed. It will be helpful if the agency can
    partner with researchers and provide input, including providing
    information through the use of case studies. However, care must be
    taken that such case studies do not include proprietary information
    that may be included in FDA-approved risk management programs. The
    FDA should clarify how it will balance public access to risk
    management program content with legitimate industry needs to keep
    proprietary information confidential.

-   Line 136-173: Similar to comments for line 69-73. Several of these
    examples require the drug sponsor, via the risk management tools, to
    intrude intimately into managing patient care.

-   Line 149-150: Enforcement of this provision, and documentation of
    such enforcement, will require resources that are currently not
    available in the FDA, industry or the medical practice setting. The
    FDA should provide guidance on how this requirement would be
    monitored and enforced.

-   Line 151-152: The final guidance should provide clarity concerning
    the specification of requirements, the ascertainment of physicians'
    meeting the requirements, and the documentation of both. The FDA
    should provide guidance on how this requirement would be monitored
    and enforced.

-   Line 154-160: Patient education programs that utilize
    product-specific educational materials delivered to or read by the
    patient after the product is prescribed or dispensed can be useful
    for warning patients before they actually take a medicine about
    contraindications and potential adverse effects, but they are of
    limited use in directing patients to communicate with their
    physician before a prescription is written. In some circumstances
    patient education programs will need to be focused on potential
    patients prior to their receiving a prescription for the sponsoring
    product. One obvious example of this a product for which
    direct-to-consumer advertising is done.

-   Line 169-170: Clarification is needed concerning the extent to which
    pharmacists would have to search to determine if a patient is likely
    to have condition A. This may be impossible in some pharmacies and
    in some patient circumstances (e.g., snowbirds, travelers, or those
    moving to new communities). One unintended consequence may be to
    encourage the use of nationwide pharmacy chains that can access the
    same individual's data from any of their pharmacies in the U.S.,
    thus prompting the business failure of smaller independent
    pharmacies.

-   Line 178-180: We support the use of an internet site to provide
    information on different types of risk management tools that are
    currently in use, provided that the legitimate proprietary
    information of sponsors is kept confidential.

-   Line 186-187: We propose that safety-related publications in the
    scientific literature be allowed to be included in generalized
    education and outreach to health professionals.

-   Line 211: The final guidance should include a discussion of the use
    of safety-related phase IV studies.

-   Line 241: We suggest that "of a risk management program" be added
    between "effectiveness" and "are", so this sentence reads "...of
    effectiveness of a risk management program are discussed....".

-   Line 243: Another section should be added to address how tools could
    be terminated or modified, either after having achieved their
    desired outcome or being proven not effective in a particular risk
    management program. For example, if a post-launch physician
    education program is shown to have established good prescribing
    habits among physicians then the program could be scaled back to a
    "maintenance phase", or if a survey shows that patients do not read
    the medication guide and it\'s making no impact on medication use,
    the specific intervention could be terminated.

-   Lines 247-251: It would be useful to expand on the rationale for the
    risk management program categorization, and the appropriate
    interpretation of the overall safety of a drug on the basis of its
    categorized level. The risk management program categorization is
    closely related to safety concerns, but a drug has to be evaluated,
    by all parties, based on benefit/risk ratio. The risk management
    program categorization of a compound solely on the basis of risks
    may interfere with patient care decisions since it may impede the
    careful consideration of specific potential risks and benefits in
    the context of an individual patient.

-   Lines 277-278: The final guidance should provide additional
    information on the nature of pre-testing of tools that is expected,
    including criteria for when pre-testing is, and is not, necessary,
    and descriptions of acceptable methods for pre-testing.
    Specifically, the final guidance should acknowledge that in many, or
    even most, cases, the tools will have been pre-tested with previous
    products that have employed the specific tool. The final guidance
    should also address the impact that the need for pre-testing might
    have on the timing of the final approval of a product. In addition,
    we suggest inserting \"whenever possible and needed\" after \"RMP\"
    in line 278 to indicate that pre-testing may not be possible or even
    needed in some situations.

-   Line 287-289: Additional clarification is needed concerning the
    periodicity of the evaluation of a risk management program and the
    criteria for increasing or decreasing the level of the risk
    management program.

-   Line 315-341: This section presumes that well-defined, validated
    metrics are available for both tools and programs. Additional
    information is needed concerning the criteria for "validating" a
    metric. This is most problematic for program outcome metrics. In
    addition, this section is exclusively geared towards a risk
    management program's ability to reduce adverse events. It would be
    useful to also address other objectives of a risk management
    program, for example, objectives to slow down market uptake and/or
    reduce the size of the high-risk population taking the medicine.
    Also, even if adverse events aren't "reduced" (assuming a baseline
    is known), that may not imply failure of a risk management program.
    It may simply imply that patients are willing to tolerate the risk
    of the adverse event in order to obtain the benefit of the
    treatment.

-   Line 348: Additional practical guidance is needed concerning
    criteria for determining if a "patient outcome" can be "practically
    or accurately measured" and if a "closely related measure" should be
    used. Specifically, each concept in quotations should be completely
    defined.

-   Line 368: In some circumstance is will be very difficult to obtain
    appropriate outcome metric data, or even appropriate surrogate
    outcome metric data. Consideration should be given to permitting the
    use of spontaneous adverse event data as a signal detector (an
    appropriate use of these data) to trigger the implementation of a
    more cumbersome approach to measuring the effectiveness of a risk
    management program.

-   Lines 389-394: We suggest this paragraph be put into a list, and
    instead of numbers (i.e., 1-4), use letters (i.e., A-D) to match the
    more detailed information that follows.

-   Lines 399-408: We suggest that this section of the guidance document
    include instructions that the background section of a risk
    management program document should contain a brief discussion of the
    benefits of the drug as well as the risks. This should be followed
    by a brief articulation of the benefit/risk balance in order to
    place the selection of the risk management program goals and
    objectives into complete perspective.

-   Lines 416-424: Similar comments as for Line 247-251, see above.

-   Lines 426-434: This section should also invite commentary on the
    risks associated with denying therapy to patients due to prescribing
    hurdles imposed by the risk management program. It would also be
    helpful if the final guidance included information on the criteria
    for requiring a product to develop and implement a risk management
    program as some point after initial market launch.

-   Line 465: Suggest addition of "of a risk management program" after
    "effectiveness".

-   Line 474-484: Similar to the comment concerning line 243 above, this
    section should include a discussion of the possible actions that
    might be taken following the evaluation of the risk management
    program, including both increasing program interventions, and
    decreasing or eliminating interventions. A decision tree on how and
    when to abandon non-performing tools, and to modify other tools
    based on their success, would be helpful. In addition, the risk
    management program will result in more reports by sponsors to
    regulatory agencies, especially when Europe and other countries or
    regions have sponsors implement risk management programs. We
    encourage the FDA to align their draft guidance with evolving ICH
    guidance on this topic so that sponsors do not have to submit
    different reports to different authorities under different report
    schedules.

# Comments on "Risk Assessment of Observational Data: Good Pharmacovigilance Practices and Pharmacoepidemiologic Assessment" Concept Paper

## General

We agree with the FDA's statement that despite the rigor of the product
development process it is not possible to detect all safety concerns
during clinical trials. We agree that pharmacovigilance activities
relate to the detection, assessment and understanding of adverse events.
We submit that while pharmacovigilance may be involved in the scientific
and data gathering activities related to the prevention of adverse
events and other product-related problems, these activities also involve
many other disciplines from the pharmaceutical and health care
industries. In addition, the implementation of measures to prevent
adverse events requires a broad based approach.

We believe that it is important for the final guidance to emphasize this
health care delivery system approach in order to more effectively enlist
all the necessary disciplines into the risk management process. The role
of these other disciplines should be appropriately described, including
their role in the pharmacovigilance activities as defined above. For
example, such descriptions could include:

-   Regulatory agencies have a responsibility to communicate their
    concerns effectively and early enough to permit the data needed to
    address their concerns to be incorporated into clinical development
    plans and risk management programs.

-   The clinical development component within a pharmaceutical company
    has a responsibility to anticipate safety-related issues, to be
    responsive to expressed regulatory agency concerns, and to
    incorporate the collection of the data needed to address
    safety-related issues into their clinical development plans.

-   The sales and marketing organization within a pharmaceutical company
    has a responsibility to incorporate information about the risks and
    benefits of their products into their communications with their
    customers in order to optimize the appropriate use of their
    products.

-   Health care professionals have a responsibility to be open to
    receive new information concerning the benefit/risk balance of a
    product.

-   Health care professionals have a responsibility to incorporate sound
    scientific information into their patient care decision-making.

-   Health care professionals have a responsibility to respond with
    complete information to queries concerning adverse events.

-   Patients have a responsibility to be open to receive new information
    concerning the benefit/risk balance of a product.

-   Patients have a responsibility to incorporate sound scientific
    information into their own decision-making concerning their health
    care.

-   Patients have a responsibility to communicate with their health care
    professional regarding any adverse events they experience.

Any approach to prevention of adverse events must recognize the diverse
roles that the participants in the health care system play, and enlist
the support of each of them, in order to achieve success.

## Specific

-   Line 28: We suggest replacing the word "minimizing" by
    "characterizing". A product's risk profile is a statement of its
    safety-related characteristics and cannot be minimized. Only the
    impact of a product's risks on the patient population has the
    potential to be decreased or minimized.

-   Lines 33-38: This discussion gives examples of possible safety
    signals, but does not provide a conceptual definition. We propose:
    "A safety signal is any data-driven observation that raises the
    possibility that a product might have a specific safety issue." This
    definition clarifies that a safety signal is an invitation for
    further evaluation, and is *not* the equivalent of an established
    risk.

-   Line 41-42: We submit that safety signals alone are insufficient for
    inclusion in a product's safety profile (see discussion immediately
    above). Only safety signals that have been evaluated and determined
    to be truly representative of a possible risk should be included in
    a product's safety profile.

-   Line 56: In addition to "ongoing evaluation of identified safety
    signals", pharmacovigilance can also identify new signals.

-   Line 60: Please clarify the nature of the clinical trials mentioned
    in this section. All clinical trials have a safety element since
    adverse event data are collected. Clinical trials that are a part of
    a pharmacovigilance plan should have a safety-related primary
    objective.

-   Line 67: The term "minimize" is used here in reference to a
    product's safety profile. Please refer to discussion of Lines 28-29
    of the "Risk Management Programs" concept paper (see above) and
    clarify the intended meaning.

-   Lines 68-70: We strongly support the concept of "a pharmacovigilance
    plan should focus on detecting new signals and/or evaluating already
    identified signals" and suggest that the final guidance documents be
    aligned with it (see general comments and Line 22 comments above).

-   Line 84-88: We support the statement that the intensity and method
    of case follow-up would be driven by the seriousness of the event
    reported, its origin, and other factors. The sponsor should be
    allowed some flexibility to focus resources on the most important
    issues.

-   Line 104: FDA and industry need to do a better job conducting
    benefit/risk assessment, since this is usually the most important
    consideration for product approval and use. It would be useful if
    spontaneous reports could also include an assessment of the benefit
    the patient experienced, not just the negative effects. The efficacy
    assessment could be based on standard endpoints for the underlying
    disease/condition, if such standard endpoints exist, otherwise based
    on more generic methods.

-   Lines 128 and 464: \"datamining\" should be divided into \"data
    mining\".

-   Line 135-137: We suggest the final guidance document explicitly
    recognize that the initial step in the evaluation of a safety signal
    is a thorough evaluation of all the information contained in the
    pertinent cases, and that further evaluation by studies, registries
    and surveys may be needed only to address questions that remain
    after such an initial evaluation.

-   Line 182-184: The recently implemented Health Insurance Portability
    and Accountability Act regulations may have an impact on the
    availability of medical records for validating diagnostic findings
    in claims databases. This impact needs to be explored and described
    in the final guidance.

-   Line 190-191: It would be helpful to supplement the definition of
    "registry" used in the final guidance document with a few
    illustrative examples.

-   Line 207-209: The recently implemented Health Insurance Portability
    and Accountability Act regulations may have an impact on the
    availability of medical records for validating registries. This
    impact needs to be explored and described in the final guidance.

-   Line 232: We suggest that the final guidance document provide
    information to assist sponsors in selecting the appropriate sample
    size for surveys.

-   Line 234-235: The recently implemented Health Insurance Portability
    and Accountability Act regulations may have an impact on the
    availability of medical records for validating survey findings. This
    impact needs to be explored and described in the final guidance.

-   Lines 237-246: As mentioned numerous times above, the full impact of
    the recently implemented Health Insurance Portability and
    Accountability Act regulations on the ability to do
    pharmacovigilance studies needs to be explored, considered and
    described in the final risk management guidance.

-   Lines 277-278: The final guidance should explicitly recognize that
    patient level prescription data is available currently for the
    United States but not for other countries, and clarify that the use
    of such denominator data to calculate reporting rates should be used
    only in conjunction with numerator data that is also only from the
    United States.

-   Line 347-349: The final guidance should acknowledge that for
    extremely rare events, e.g., osteosarcoma, assessing causality might
    be very difficult. Rigorous study designs, such as a case-control
    study, and long-term follow up may be needed to adequately assess
    causality. Uniform guidelines for such situations would be helpful,
    possibly in the form of a decision tree.

-   Line 358-372: The final guidance on safety signals related to
    medication errors should be aligned with the final regulations
    published to implement the related regulations in the recently
    published proposed rule on Safety Reporting Requirements for Human
    Drug And Biological Products (Docket No. 00N-1484, the "tome").

-   Line 376-381: The final guidance on reporting safety signals should
    be aligned with the final regulations published to implement the
    related regulations currently being proposed in the "tome".

-   Lines 399-409: The final guidance should contain more explicit
    information on how the agency will take each of the five factors
    (lines 403-409) into account when determining the need for a
    pharmacovigilance plan, and how each can influence the need for
    specific elements that may be a part of the plan.

-   Lines 428-444: The final guidance should contain more explicit
    information on how the agency will determine the need for each of
    the six listed elements of a pharmacovigilance plan.

-   Lines 446-450: The final guidance should clarify the circumstances
    in which the FDA would bring questions regarding safety signals,
    proposed pharmacovigilance plans, and findings arising from
    pharmacovigilance efforts before its Drug Safety and Risk Management
    Advisory Committee.

-   Line 462: An FDA campaign to educate health care professionals and
    patients of the value of providing information concerning adverse
    events would assist the sponsor's efforts to obtain this
    information. Currently, even vigorous, active queries are met
    frequently with minimal responses. This has been aggravated recently
    by perceptions of increased liability by health care professionals
    due to the recently implemented Health Insurance Portability and
    Accountability Act regulations.

-   Lines 464-466: Data mining is another approach to signal detection.
    The statistical techniques employed may increase the sensitivity for
    signal detection but do not increase the specificity. Signals
    detected by data mining techniques must still be thoroughly
    evaluated to determine if they represent a real risk associated with
    the product. Since the source data for data mining are spontaneous
    adverse event reports, underreporting makes interpretation of
    results very difficult. The FDA educational campaign mentioned
    immediately above would also help address this aspect.

-   Lines 468-469: The assessment of causality at the case level often
    suffers from the lack of reliable information. Measures described in
    the "tome" to promote collection of complete information may be
    helpful, but it is unrealistic to expect that all information will
    be collected for all cases, especially those reported by consumers
    with no corroboration by their primary physician. In addition, the
    only instances for which a determination of causality is of
    potential utility are for unexpected or unlisted events, particular
    those that are serious. Even for many of these events, a definitive
    answer to whether a product is associated with a particular adverse
    event will come from a pharmacoepidemiology study that evaluates
    this association at the population level (rather than the individual
    case level). Therefore, the assessment of causality for each
    individual case provides little utility and should be abandoned as a
    mandated routine practice. Instead, individual cases should be
    evaluated for causality only as part of a larger effort to evaluate
    an identified safety signal.

-   Lines 471-472: Registries (as defined in lines 190-191 of the
    concept paper) can take many forms. Their primary utility is to
    obtain precise estimates of the incidence rate of uncommon events,
    and to explore confounding and contributing factors to events of
    interest. Hence, they should be deployed when these data are needed
    to guide the appropriate use of a product and this information
    cannot be obtained in less costly or time consuming ways. They use
    of registries to quantitate the incidence rate of rare or very rare
    events is prohibitively costly and does not provide information that
    is any more useful than that provided by an active surveillance
    program. If a registry is used it should be governed by a
    well-design protocol that specifies both the questions to be
    addressed and the criteria by which it will be known that the
    questions have been answered. Registries should not be initiated, or
    maintained indefinitely, "just in case" they are needed.

-   Lines 474-475: Active surveillance strategies would be useful to
    identify those rare serious events that are not seen in limited
    clinical trials (e.g., \<5000 patient exposures). To be most useful
    the active surveillance program should also be targeted at events
    that can be anticipated, either on the basis of the known mechanism
    of action of the drug or because of heightened concern for any
    reason.

-   Lines 477-478: Pharmacoepidemiologic studies should be used to
    further explore safety signals that have been determined to be
    "true" signals by standard pharmacovigilance evaluation procedures,
    and for which additional information is needed to guide the
    appropriate use of the product involved. If a pharmacoepidemiologic
    study is used it should be governed by a well-design protocol that
    specifies both the questions to be addressed and the criteria by
    which it will be known that the questions have been answered. For
    example, if a rare serious event is identified as a "true" signal,
    it could lead to a pharmacoepidemiologic study to compare the
    apparent rate on patients receiving the drug to the general
    population with the same disease, identify confounding and
    contributing risk factors, and possibly identify potential
    ameliorating measures.

Paul Eisenberg, M.D.

Vice-President, Global Drug Safety

Eli Lilly and Company

#######