New Drug Approval: FDA's Consideration of Evidence from Certain Clinical Trials
Published: 2010-07-30T00:00:00

Fast Facts:

Highlights:
section_title: Why GAO Did This Study
paragraphs: Before approving a new drug, the Food and Drug Administration (FDA)--an agency of the Department of Health and Human Services (HHS)--assesses a drug's effectiveness. To do so, it examines information contained in a new drug application (NDA), including data from clinical trials in humans. Several types of trials may be used to gather this evidence. For example, superiority trials may show that a new drug is more effective than an active control--a drug known to be effective. Non-inferiority trials aim to demonstrate that the difference between the effectiveness of a new drug and an active control is small--small enough to show that the new drug is also effective. Drugs approved on this basis may provide important benefits, such as improved safety. Because non-inferiority trials are difficult to design and interpret, they have received attention within the research community and FDA. FDA has issued guidance on these trials. GAO was asked to examine FDA's use of non-inferiority trial evidence. This report (1) identifies NDAs for new molecular entities--potentially innovative new drugs not FDA-approved in any form--that included evidence from non-inferiority trials, (2) examines the characteristics of these trials, and (3) describes FDA's guidance on these trials. GAO reviewed NDAs submitted to FDA between fiscal year 2002 (the first full year that FDA documentation was available electronically) and fiscal year 2009 (the last full year of submissions), examined FDA's guidance, and interviewed agency officials.
section_title: What GAO Found
paragraphs: Evidence from non-inferiority trials was included in about one-quarter, or 43, of the 175 NDAs for new molecular entities that were submitted to FDA for review from fiscal years 2002 through 2009. Many of these applications were for antimicrobial drugs, such as those treating bacterial, viral, and fungal infections. As of December 31, 2009, FDA approved 18 of the 43 NDAs on the basis of evidence from non-inferiority trials. Of the remaining 25 NDAs, FDA approved 11 based on other evidence, such as proof that the new drug was more effective than a placebo (no treatment), and decided not to approve 14. The non-inferiority trials included in these NDAs varied with respect to their characteristics. FDA generally requires sponsors to provide evidence of a drug's effectiveness as shown in more than one trial. For the 18 NDAs that were approved based on evidence from non-inferiority trials, the number of non-inferiority trials used to provide primary support for approval ranged from one to four, with an average of 2 such trials per NDA. Half of these applications included non-inferiority trials that tested the effectiveness of the new drug against more than one active control. The non-inferiority margins--the maximum clinically acceptable extent to which the new drug can be less effective than the active control and still show evidence of an effect--ranged from 5 to 20 percent among trials that supported approval. Among the other 25, FDA identified nine NDAs that included poorly designed non-inferiority trials which did not provide primary evidence for approval. Some of these problems included an inappropriate selection of an active control and an improper calculation of a non-inferiority margin. FDA notified sponsors of its concerns with the poorly designed trials prior to the sponsors' submissions of all NDAs that included such trials. In March 2010 FDA issued draft guidance which focused solely on the use of non-inferiority trials. This guidance presents detailed and comprehensive recommendations on how non-inferiority trials may be used to provide evidence of a drug's effectiveness. For example, it provides advice on how to select an active control and how to set the non-inferiority margin, as well as how to interpret the trials. This guidance offers broad, generally applicable recommendations to supplement indication-specific guidance documents that FDA had previously issued. These indication-specific guidance documents include FDA's advice on many issues related to the development of drugs for particular indications, some of which are related to the use of non-inferiority trials. GAO's review of FDA's guidance showed that the agency has become more conservative in allowing evidence from non-inferiority trials to demonstrate a drug's effectiveness. First, FDA has limited the indications for which these trials may be used. Second, the agency has also become more rigorous in its review of evidence from non-inferiority trials. We sent a draft of this report to HHS for review. HHS provided us with technical comments, which we incorporated as appropriate.

section_title: Letter
paragraphs: Testing new drugs on human volunteers is an essential step in the drug  development process. These tests, known as clinical trials, are  instrumental in determining whether a drug is safe and effective. Their  purpose is to measure the effect of a new drug separately from other  influences, such as a spontaneous change in the course of a disease.  Before a new drug can be marketed in the United States, the drug’s  sponsor—typically a manufacturer—must submit a new drug application  (NDA) to the Food and Drug Administration (FDA) for approval. The  agency will only approve the NDA if it determines that the drug is safe and  effective for its intended use. To make this determination, FDA reviews  the results of clinical trials that sponsors submit as part of their NDAs.
Sponsors must provide evidence of a drug’s effectiveness based on  adequate and well-controlled trials. Several different types of trials may  be used to gather this evidence. Some clinical trials are designed to test  whether a new drug is more effective than a placebo or no treatment at all.  However, in some instances it would be unethical to offer a placebo or  withhold treatment because it would deprive human volunteers of an  available treatment known to prevent death, irreversible injury, or other  serious harm. In such instances, a clinical trial may be conducted to  measure the effect of a new drug compared to an active control—a drug  already known to be effective. For example, a superiority trial could be  conducted to show that a new drug is more effective than an active  control. Another type of clinical trial, a non-inferiority trial, is intended to  establish that the difference in the effectiveness of a new drug and an  active control is small. Non-inferiority trials that demonstrate a small  difference in the effectiveness of the two drugs may be able to support a  conclusion that the new drug is effective because the effectiveness of the  active control is known.
FDA considers the use of non-inferiority trials to be acceptable in certain  circumstances and has approved drugs on the basis of evidence from such  trials. Although non-inferiority trials may prove that new drugs are no  more effective than the active controls they are compared to, new drugs  approved on the basis of non-inferiority may provide patients with other  important benefits, such as improved safety, fewer drug-to-drug  interactions, convenience of administration, or a lower cost.
However, non-inferiority trials are more complicated to design and their  results are more difficult to interpret than other types of clinical trials. For  example, in interpreting the results of non-inferiority trials, sponsors not  only must examine the difference in the effectiveness of the two drugs as  measured in the non-inferiority trial, they must also assess whether the  active control proved to be as effective as expected. If the active control  did not demonstrate its expected effect in the trial, results showing the  similarity of the two drugs are meaningless. As a result of these and other  issues, non-inferiority trials have received attention within the research  community and FDA. To assist sponsors in appropriately using non- inferiority trials to establish a drug’s effectiveness, FDA has issued written  guidance conveying the agency’s understanding of non-inferiority trials  and how such trials may be used to support a drug’s approval.
You asked us to review FDA’s use of evidence from non-inferiority trials to  establish a drug’s effectiveness and support approval. In this report, we   (1) identify the type and status of drug applications submitted for FDA  review that included evidence from non-inferiority trials; (2) examine the  characteristics of non-inferiority trials FDA considered in making approval  decisions; and (3) describe FDA’s guidance for establishing a drug’s  effectiveness on the basis of non-inferiority trials.
To identify the type and status of drug applications submitted to FDA for  review that included evidence from non-inferiority trials, we limited our  scope to a subset of drugs. FDA officials told us that the majority of non- inferiority trials for drugs listed in NDAs were for new molecular  entities—potentially innovative drugs containing active ingredients that  have never been approved for marketing in the United States in any form.  We therefore examined FDA data on the 223 NDAs for new molecular  entities that were submitted to FDA during the last 8 years—fiscal years  2002 through 2009 (October 1, 2001, through September 30, 2009). Fiscal  year 2002 was the first full year that records documenting FDA’s review of  NDAs were available in electronic format, and so we did not include prior  years in our scope. Fiscal year 2009 was the last full year for which data  was available at the time we requested data from FDA and so we excluded  NDAs submitted to FDA after that period. We limited our scope to  prescription drugs intended to prevent or treat diseases or other medical  conditions. We excluded 18 NDAs for nonprescription drugs, drugs used to  aid in diagnosing diseases, or drugs aiding in the absorption of other  drugs. We further limited our scope to those NDAs for which FDA had  completed its review by December 31, 2009—excluding 30 NDAs that were  either withdrawn or pending review as of that date. These two limitations  excluded 48 of the 223 NDAs from our review and we conducted our  analysis on the remaining 175 NDAs for new molecular entities. As our  analysis was limited to a subset of all NDAs received by FDA, the results  of our review may not be generalizable to other types of applications.
To determine whether NDAs within our scope included evidence from  non-inferiority trials, we examined documents summarizing the results of  clinical trials that we obtained from FDA’s Web site (for approved NDAs)  and from agency officials (for NDAs not approved as of December 31,  2009). Specifically, we reviewed FDA’s statistical and medical reviews to  determine whether the application included a non-inferiority trial. For  those NDAs with at least one non-inferiority trial, we determined whether  FDA considered data from these trials as pivotal—that is, providing the  primary evidence of effectiveness to support the NDA. Because FDA  could not readily identify the NDAs that it had received that included  evidence from non-inferiority trials, we submitted the results of our  analysis to FDA. We asked agency officials to confirm that we had  correctly identified the NDAs that included evidence from non-inferiority  trials—including those that were approved primarily based on this  evidence, approved based on other primary evidence, or that had not been  approved as of December 31, 2009.
To examine the characteristics of non-inferiority trials FDA considered in  making approval decisions, we analyzed FDA’s statistical and medical  reviews for those drugs that included evidence from at least one non- inferiority trial in their NDAs. For those drugs FDA ultimately approved  based on evidence from non-inferiority trials, we gathered descriptive  information about the non-inferiority trials that FDA considered. In  particular, we determined the number of non-inferiority trials conducted  to support each application, identified the active controls used in these  trials, and reviewed whether and how each trial supported the conclusion  that the drug was effective. We also reviewed whether the characteristics  of the non-inferiority trials providing primary evidence for approval  revealed evidence of biocreep. Biocreep refers to a concern that  successive generations of drugs approved based on non-inferiority trials,  with the active control changing in each new generation, could lead to the  adoption of decreasingly effective drugs, culminating in the approval of  drugs that are no more effective than a placebo. We examined how FDA  assessed the effectiveness of the active controls used in non-inferiority  trials to ensure that comparisons to these drugs were appropriate— particularly where successive generations of non-inferiority trials  occurred. We also reviewed FDA’s correspondence with sponsors about  their non-inferiority trials to analyze the information FDA communicated  with sponsors before, during, and after their clinical trials. We gathered  information on the concerns FDA identified and communicated with  sponsors regarding their non-inferiority trials and determined when FDA  notified sponsors about these issues.
To describe FDA’s guidance for establishing drug effectiveness on the  basis of non-inferiority trials, we reviewed guidance FDA issued from  January 2002 through June 2010 that was related to the design of such  trials. From these published guidance documents we gathered FDA’s  recommendations on how to use non-inferiority trials to support the  effectiveness of new drugs. We interviewed FDA officials to obtain  contextual information regarding their issuance of this guidance. We also  interviewed experts, such as biostatisticians and physician specialists,  including those affiliated with the Infectious Diseases Society of America  and the American Society of Clinical Oncology, to obtain their  perspectives regarding the content and clarity of FDA’s guidance. We did  not assess the extent to which sponsors’ non-inferiority trials or FDA’s  approval decisions on applications that included evidence from these  types of trials comported with agency guidance.
We conducted this performance audit from July 2009 to July 2010 in  accordance with generally accepted government auditing standards. Those  standards require that we plan and perform the audit to obtain sufficient,  appropriate evidence to provide a reasonable basis for our findings and  conclusions based on our audit objectives. We believe that the evidence  obtained provides a reasonable basis for our findings and conclusions  based on our audit objectives.
subsections: section_title: Background
paragraphs: FDA, an agency within the Department of Health and Human Services  (HHS), is responsible for overseeing the safety and effectiveness of drugs  marketed within the United States. These responsibilities begin before a  product is brought to the market, and include reviewing drug sponsors’  proposals for conducting clinical trials, providing advice and publishing  guidance regarding these trials, as well as reviewing applications for new  drugs.
subsections: section_title: The Use of Non-Inferiority Trials in Obtaining Evidence of a Drug’s Effectiveness
paragraphs: Once a drug sponsor identifies a promising chemical compound it believes  to be capable of curing or treating diseases, the sponsor may decide to  conduct clinical trials on humans to gather the evidence necessary to  demonstrate to FDA that the drug is safe and effective for its intended use.  Before beginning clinical trials in the United States, a sponsor generally  must submit an investigational new drug application to FDA for review.  This application provides FDA with extensive information about the drug,  including safety and manufacturing information, and outlines the  sponsor’s plans for clinical trials, which gradually introduce new drugs to  increasingly larger numbers of patients. FDA assesses the information in  the application—which is later included as part of the NDA—to ensure  that the drug is reasonably safe to begin studying in humans.
Sponsors may use these clinical trials to gather evidence of a drug’s safety  and effectiveness. In general, FDA requires sponsors to submit the results  of more than one clinical trial demonstrating effectiveness in order to  provide substantial evidence that a drug is effective for the intended  indication and population. FDA has issued regulations and guidance that  provide industry with information to properly design, conduct, and  interpret these trials. For example, in 1985 FDA substantially revised its  regulations including the provision addressing the characteristics of  adequate and well-controlled trials and the types of controlled trials that  can be used to gather evidence of a new drug’s effectiveness.
Sponsors may use trials of varying designs to obtain evidence of a drug’s  effectiveness. One type of clinical trial is a non-inferiority trial. The  objective of a non-inferiority trial is to show that any difference in the  effectiveness of two drugs is small enough to allow a conclusion that the  new drug is also effective, but not substantially less effective than the  active control. To conduct a non-inferiority trial, sponsors must make  many decisions regarding how the trial will measure the new drug’s  effectiveness. For example, they must select the trial’s primary endpoint,  the principal measure used to determine a drug’s effectiveness. The  primary endpoint may be a clinical endpoint—a direct measure of how a  patient feels, functions, or survives—or, in some cases, a surrogate  endpoint—a laboratory measure or physical sign used as a substitute for a  clinical endpoint that reasonably predicts a clinical benefit. Sponsors  must also determine when to measure the trial’s endpoint—for example,  are patients cured within 7, 14, or 30 days after starting treatment—in  addition to determining the number and type of patients to be enrolled in  each trial.
Sponsors conducting non-inferiority trials must also make decisions to  account for the new drug’s comparison to the active control. Sponsors  must identify an available treatment for use as an active control in the non- inferiority trial. They must then use evidence of the active control’s  effectiveness as shown in prior clinical trials to estimate the effect that the  active control will have in the planned non-inferiority trial, adjusting for  any differences between the prior and planned trials. Using this estimate,  sponsors determine the trial’s non-inferiority margin—the maximum  clinically acceptable extent to which the new drug can be less effective  than the active control and still show evidence of an effect. FDA considers  the selection of a margin to be the single greatest challenge in designing,  conducting, and interpreting non-inferiority trials. Its calculation is not  only dependent on a string of other decisions related to the trial—for  example, the data collected on the active control’s effectiveness in other  trials—but also includes the application of clinical judgment to determine  the maximum amount of effectiveness that could be lost without having a  substantial impact on the drug’s effectiveness. If a non-inferiority margin is  incorrectly calculated and is set too large, a drug that is not effective may  appear to be effective; if the margin is too small, an effective drug may  appear to be ineffective.
In a non-inferiority trial, patients are randomly assigned to receive either  the new drug or an active control. After the trial, the sponsor identifies the  observed effect of each drug in the trial, and calculates the observed  difference in the drugs’ effectiveness. The actual difference in the drugs’  effectiveness in the entire population could be greater or less than what is  observed in the trial. For that reason, sponsors calculate a confidence  interval around the observed difference in effectiveness between the new  drug and active control drug. The confidence interval provides a range of  values for the difference in effectiveness within which the true difference  is likely to be found.
The confidence interval around the observed difference in effectiveness is  used to determine if the new drug is non-inferior to the active control. It is  compared to the non-inferiority margin—the maximum clinically  acceptable extent to which the new drug can be less effective than the  active control. If the confidence interval is within the non-inferiority  margin, and the sponsor provides adequate evidence that the active  control demonstrated its expected effect in the trial, the new drug may be  deemed non-inferior to the active control. A new drug can be non-inferior  to an active control even if the estimated difference in effectiveness and its  confidence interval lies entirely below zero, meaning that the active  control drug is more effective than the new drug, but by an irrelevant  amount. However, if the confidence interval shows that the effect of the  drug could be below the margin—even if the observed effect of the drug  was within the margin—the drug would not have shown an effect, and is  therefore considered inferior. In addition, if the confidence interval lies  entirely above zero—demonstrating that the new drug is more effective  than the active control—the drug can be considered superior. (See fig. 1.)
Since issuing regulations that address the elements of adequate and well- controlled trials, FDA has also periodically issued guidance documents to  provide updates on the agency’s current thinking on a range of topics.  These guidance documents encompass broad issues such as statistical  principles for use in clinical trials and how to select an appropriate  control, whereas others are more focused and serve to consolidate  relevant recommendations on the development of drugs treating a  particular indication.
In addition to disseminating guidance on non-inferiority trials, FDA  provides specific advice regarding the design of clinical trials at the  request of sponsors. For example, sponsors may ask FDA to review and  provide advice on a trial’s proposed active control, non-inferiority margin,  or endpoint before the given trial has begun. After the conclusion of their  clinical trials, sponsors may consult with FDA regarding the interpretation  of trial results or to discuss the information the agency would expect to  see submitted in an NDA. FDA’s advice and recommendations to sponsors  are considered advisory; sponsors are not required to implement any of  the agency’s suggestions.
If sponsors believe they have successfully demonstrated a new drug’s  safety and effectiveness, they may submit an NDA to FDA for review. The  NDA contains information about the safety and effectiveness of the drug  as demonstrated in clinical trials and other research, such as studies in  animals. Once the agency receives an NDA, the application is reviewed by  one of FDA’s medical review divisions, depending on the indication the  drug has been proposed to treat. If FDA determines that the drug is safe  and effective for its intended use—that its clinical benefits outweigh its  potential health risks—and that other requirements are met, it will  approve the application. After approving a new drug, FDA’s  responsibilities continue as it is charged with monitoring the safety,  effectiveness, and promotion of approved drugs. FDA executes these  responsibilities in the same manner regardless of whether drugs were  approved on the basis of evidence from non-inferiority trials.
section_title: Issues Unique to Non-Inferiority Trials
paragraphs: Non-inferiority trials present unique issues in measuring the effectiveness  of new drugs. For example, the use of these trials can raise uncertainties  about the true effectiveness of new drugs because non-inferiority trials  cannot measure this directly. Instead, these trials measure the  effectiveness of the new drug relative to the active control, and sponsors  must assess whether the active control can be considered to be as  effective in the non-inferiority trial as was expected based on past  experience. Using data from the non-inferiority trial and from prior trials  measuring the effectiveness of the active control, the effectiveness of the  new drug is estimated—but not ever fully known. In addition, non- inferiority trials are more prone to certain biases than superiority trials.  For example, if patients in a superiority trial do not take the new drug as  directed, this poor compliance will dilute the measured effectiveness of  the new drug, making it less likely that the trial will successfully  demonstrate superiority. In a non-inferiority trial, however, poor  compliance by patients taking the active control drug can have a different  effect. It can reduce the difference in the measured effectiveness between  the new drug and the active control, making the treatments appear more  similar than they might otherwise be. As such, poor compliance in a non- inferiority trial can increase the likelihood that an ineffective drug is  concluded to be effective.
The use of non-inferiority trials over time also raises concerns about the  potential for “biocreep” to occur. This term is used to describe the concern  that successive generations of drugs approved based on non-inferiority  trials, with the active control changing in each new generation, could lead  to the adoption of decreasingly effective drugs and ultimately to the  approval of drugs that are no more effective than a placebo. Non- inferiority trials that are poorly designed are especially prone to biocreep.  The selection of inappropriate active controls—that is, drugs that are not  known to be consistently effective, or drugs that were themselves  approved on the basis of non-inferiority trials—could lead to biocreep.
Even if successive generations of non-inferiority trials are conducted and  each trial is itself well-designed, biocreep may still occur because placebo  controls are not included in these trials. Non-inferiority trials are only able  to measure the effectiveness of the new drug relative to the active control,  not a placebo. As a result, the true effectiveness of any of the new drugs,  compared to a placebo, is not measured. Without this metric, it is  impossible to determine the extent to which the effectiveness of the new  drug is similar to that of a placebo and whether biocreep has occurred.
FDA has acknowledged some concerns over the uncertainties inherent in  non-inferiority trials and the potential these trials create for biocreep. For  example, FDA stated in a 1992 guidance document that, in order to avoid  biocreep, sponsors should consult with the agency regarding the active  controls they were considering for their trials. In other guidance  documents, FDA has also encouraged sponsors to consult with the agency  regarding their planned non-inferiority trials.
section_title: One-Quarter of NDAs Included Evidence from Non-Inferiority Trials and FDA Approved a Majority of These Applications
paragraphs: One-quarter of NDAs submitted to FDA for review from fiscal years 2002  through 2009 included evidence from non-inferiority trials, and many of  these applications were for antimicrobial drugs. FDA approved a majority  of the applications that included evidence from these trials.
subsections: section_title: One-Quarter of NDAs, Many for Antimicrobial Drugs, Included Evidence from Non-Inferiority Trials
paragraphs: Forty-three, or one-quarter, of the 175 NDAs we reviewed that were  submitted to FDA from fiscal years 2002 through 2009 included evidence  from at least one non-inferiority trial. The number of NDAs with evidence  from non-inferiority trials varied from year to year and generally declined  from fiscal years 2002 through 2009. On average, FDA received five NDAs  each year that included evidence from non-inferiority trials. (See fig. 2.)
About half of the 43 NDAs submitted with evidence from at least one non- inferiority trial—or 22—were for antimicrobial drugs, such as those that  treat bacterial, viral, or fungal infections. The remaining portion of NDAs  submitted with evidence from these trials represented a variety of drug  types. (See table 1.)
section_title: FDA Approved a Majority of NDAs That Included Evidence from Non-Inferiority Trials
paragraphs: FDA approved 29 of the 43 NDAs submitted for review from fiscal years  2002 through 2009 that included evidence from at least one non-inferiority  trial. Most NDAs—18 of the 29—were approved based on evidence from  pivotal non-inferiority trials. FDA approved the remaining 11 applications  based on other evidence, such as the superiority of the new drug  compared to a placebo or an active control. As of December 31, 2009, FDA  had decided not to approve 14 applications that included evidence from  dence from  non-inferiority trials. (See fig. 3.)  non-inferiority trials. (See fig. 3.)
Many NDAs including evidence from non-inferiority trials were for  antimicrobial drugs, and the majority of approvals based on this evidence  were also for these types of drugs. Two-thirds, or 12 of the 18, NDAs  approved on the basis of non-inferiority trials were for antimicrobial  drugs. The remaining one-third of NDAs approved on the basis of non- inferiority trials were for various other types of drugs, including those  treating diabetes and chemotherapy-induced nausea and vomiting. See  appendix I for a list of all 18 NDAs approved based on evidence from non- inferiority trials, including fiscal year of approval, drug type, and approved  indication.
section_title: Non-Inferiority Trial Characteristics Varied and Not All Such Trials Provided Primary Evidence for the Approval of NDAs
paragraphs: Characteristics varied among the non-inferiority trials providing primary  evidence to support FDA’s approval of 18 NDAs. Some other applications  also included non-inferiority trials that FDA identified as being poorly  designed; these trials did not provide primary evidence for approval.
subsections: section_title: Characteristics Varied Among the Non-Inferiority Trials That Provided Primary Evidence for Approval of 18 NDAs
paragraphs: Characteristics varied among the non-inferiority trials that provided  primary evidence for the approval of the 18 NDAs. FDA relied on primary  evidence from multiple pivotal non-inferiority trials to support the  approval of most of these applications. The number of pivotal non- inferiority trials used as primary evidence for these 18 NDAs ranged from  one to four, with an average of two pivotal non-inferiority trials supporting  the approval of each application. In addition to including evidence from  pivotal non-inferiority trials, five applications included evidence from  other types of pivotal trials; for example, trials demonstrating superiority  to a placebo or active control drug. Thirteen of the 18 applications  included only pivotal non-inferiority trials in their applications. Of these  applications, FDA approved four based on evidence from a single pivotal  non-inferiority trial.
Two-thirds, or 12, of the 18 NDAs included trials that measured drug  effectiveness using a surrogate, rather than a clinical, primary endpoint in  at least one of their pivotal trials. Although FDA generally prefers that  drug sponsors demonstrate the effectiveness of a new drug by showing its  impact on a clinical endpoint, in certain cases, it will consider a surrogate  endpoint if it determines it is a reasonable substitute. However, all experts  we interviewed who commented on this topic noted that the approval of  drugs on the basis of both non-inferiority trials and surrogate endpoints  increases uncertainty in the drugs’ true effectiveness.
Half of the 18 NDAs FDA approved on the basis of non-inferiority trials  tested the effectiveness of the new drug against more than one active  control. A majority of the active controls used in non-inferiority trials were  FDA-approved for the indication. However, three applications included  evidence from non-inferiority trials that used one active control that was  not FDA-approved for the indication. For example, in fiscal year 2003,  FDA approved Cubicin for the treatment of complicated skin and skin  structure infections on the basis of evidence from two pivotal non- inferiority trials that used a total of five different active control drugs.  While three of these active control drugs were FDA-approved to treat this  indication, two were not. In addition, some of the active controls used in  non-inferiority trials were themselves approved on the basis of evidence  from other trials that compared the drug to another active control.  However, FDA reviewed the selection of nearly all of the active controls  used in the pivotal non-inferiority trials that supported the approval of the  18 NDAs, and found the active controls appropriate for use in these trials.  FDA officials also told us that if a new drug was approved on the basis of  evidence from non-inferiority trials, the active control used in these trials  would most likely also be used in subsequent trials, except in cases where  the newer drug proved to be superior to the active control.
The margins used for most of the 18 NDAs approved on the basis of  evidence from non-inferiority trials ranged from 5 to 20 percent, with the  most commonly used margin being 10 percent. That is, for trials using a  10 percent non-inferiority margin, the new drug could be estimated to be  up to 10 percent less effective than the active control. However, the  observed difference in the effectiveness of the new drug and active  control, as measured in the clinical trials, would be less than 10 percent.
At the time of its review of the NDAs, FDA agreed with the non-inferiority  margins set for all of the pivotal trials submitted for the majority of drugs  approved on the basis of evidence from non-inferiority trials. All of the  pivotal trials submitted for these drugs—that is, those where FDA agreed  with the margin—demonstrated that the new drug was non-inferior to the  active control drug as measured on the primary endpoint, with one  exception. These trials showed that the confidence interval for the  difference in the drugs’ effectiveness was within the non-inferiority  margin.
FDA did not agree with the non-inferiority margins set for pivotal trials  submitted with three applications, though the agency approved these  drugs based on evidence from these trials. For two drugs, Exjade and  Reyataz, FDA stated that the proposed margins could not be used to  measure the drugs’ effectiveness. FDA conducted additional analyses of  data from pivotal trials submitted in these drugs’ applications which  showed that the drugs were superior to a placebo. For the third drug,  Noxafil, FDA did not agree with the sponsor’s proposed justification of the  margin for one trial, although this trial showed the difference in the drugs’  effectiveness to be less than the disputed margin.
FDA approved Exjade in fiscal year 2006 to treat chronic iron overload in  certain patients receiving blood transfusions. Exjade’s NDA included  evidence from one pivotal non-inferiority trial that had an objective of  showing that Exjade lowered iron levels to a similar extent as the active  control. Upon reviewing the application, FDA disagreed with the non- inferiority margin proposed for this trial. FDA analyzed data from the trial  which showed that Exjade was effective in lowering patients’ iron levels  despite ongoing blood transfusions (which typically result in increased  iron levels), particularly among those patients who began the trial with  very high iron levels. FDA approved Exjade on the basis of this evidence,  which showed that the drug would have been more effective than a  placebo. In addition, FDA officials noted that Exjade presented a valuable  alternative in the treatment of this indication.
FDA approved Reyataz in fiscal year 2003 for the treatment of human  immunodeficiency virus (HIV) infection. Reyataz’s NDA included evidence  from two pivotal non-inferiority trials, including one in patients that were  naïve to HIV treatment and one in patients that had experience receiving  HIV treatment. FDA agreed with the margin proposed for the trial  conducted in the treatment-naïve population, which was successful in  demonstrating that Reyataz was non-inferior to its active control.  However, FDA disagreed with the margin proposed for the trial conducted  in the treatment-experienced population. Agency officials analyzed data  from this trial which showed that Reyataz was effective in treatment- experienced patients, and this effect was greater than what would have  been expected with a placebo. FDA approved Reyataz to treat HIV  infection on the basis of this evidence, as well as other pivotal evidence of  effectiveness in the treatment-naïve population. In addition, FDA officials  noted that Reyataz presented an alternative to HIV-infected patients that  were not responding to available HIV treatments.
FDA approved Noxafil in fiscal year 2006 for the prevention of invasive  Aspergillus and Candida infections in certain patients on the basis of  evidence from two pivotal trials. In its review of this NDA, FDA noted that  the sponsor had not adequately explained the relevance of the proposed 15  percent non-inferiority margin. One of these trials demonstrated that  Noxafil was superior to its active control, and the other trial demonstrated  that the drug was at most three percent less effective than the active  control. FDA approved this drug on the basis of this evidence of  effectiveness.
Table 2 provides a summary of the characteristics of non-inferiority trials  for the 18 NDAs we identified as approved on the basis of evidence from  non-inferiority trials.
We found that FDA reviewed the characteristics of the non-inferiority  trials supporting the approval of the 18 NDAs to ensure that the drugs it  approved were more effective than a placebo. FDA’s review therefore  minimized the potential for biocreep. Similarly, our examination of the  trials’ characteristics also revealed no evidence of biocreep.
section_title: Some Non-Inferiority Trials Were Poorly Designed and Did Not Provide Primary Evidence for Approval
paragraphs: While non-inferiority trials provided primary evidence of effectiveness to  support the approval of 18 NDAs, other non-inferiority trials were poorly  designed and did not provide such evidence. Of the other 25 NDAs that  included evidence from non-inferiority trials, FDA identified 9 applications  that included poorly designed non-inferiority trials. These trials were  unable to accurately measure the new drugs’ effectiveness and did not  provide primary evidence for the approval of these drugs. Some of the  concerns FDA identified with sponsors’ non-inferiority trials were  inappropriate use of non-inferiority trials for the indication being treated, inappropriate selection of an active control, including cases where the  drug was not FDA-approved or the sponsor did not provide an adequate  justification, and improper calculation or justification of the non-inferiority margin.
FDA informed sponsors of its concerns with all of these applications’ non- inferiority trials prior to the sponsors’ submission of the NDAs.  Specifically, FDA notified the sponsors between 1 month and 94 months  before submission, with an average of about 30 months prior to  submission. With the exception of one application, FDA notified all  sponsors at least 6 months prior to submission.
For example, FDA advised one sponsor before the sponsor began its non- inferiority trials—24 months prior to submitting its NDA—that the agency  did not consider it appropriate to use non-inferiority trials to support the  approval of the drug for the indication being sought—treatment of  schizophrenia. FDA reiterated this position on another occasion prior to  the NDA submission. FDA did not consider the results of this trial to  provide primary evidence to support its approval decision. The agency  ultimately approved the drug based on evidence that the drug was superior  to placebo as demonstrated in several other trials.
In another case, a sponsor conducted the non-inferiority trial outside of  the U.S. and had not requested FDA’s input while planning or conducting  the trial. The sponsor requested a meeting with FDA to discuss its planned  NDA. During this meeting, which occurred 1 month before FDA received  the NDA, the agency learned of the sponsor’s non-inferiority trial and  communicated its concerns regarding the design of the trial. FDA did not  consider the results of this non-inferiority trial in its approval decision, but  ultimately approved the drug based on evidence of superiority to placebo  as demonstrated in another trial.
section_title: FDA Has Issued Detailed and Comprehensive Guidance on the Use of Non-Inferiority Trials to Establish the Effectiveness of New Drugs
paragraphs: In March 2010, FDA issued draft guidance on non-inferiority trials that  provides detailed recommendations on using these trials to provide  evidence of a new drug’s effectiveness. This March 2010 draft guidance  offers broader and more comprehensive information to supplement other  indication-specific guidance documents the agency previously issued.
subsections: section_title: FDA’s March 2010 Draft Guidance Provides Detailed Recommendations on the Use of Non-Inferiority Trials in Establishing the Effectiveness of New Drugs
paragraphs: In March 2010, FDA issued new draft guidance on non-inferiority trials  that provides detailed recommendations on how these trials may be used  to establish the effectiveness of new drugs. Although FDA had previously  issued guidance documents that included information regarding the use of  non-inferiority trials for certain indications, this March 2010 guidance is  the first focused solely on the use of non-inferiority trials. It explains the  key principles involved in using a non-inferiority trial to demonstrate the  effectiveness of a drug and provides detailed recommendations for such  trials, including how to select an active control and how to set the non- inferiority margin (that is, determining the maximum clinically acceptable  extent to which the new drug can be less effective than the active control),  among other things. The March 2010 guidance also explains why the  agency considers its recommendations appropriate, offers answers to  frequently asked questions, and lists detailed examples to illustrate some  common challenges in designing and interpreting non-inferiority trials.
FDA officials told us that they developed the March 2010 guidance on non- inferiority trials because it was clear to them that these trials were not  well understood. The concepts elaborated on in the March 2010 guidance  are not new, however. They have been part of FDA’s considerations since  at least 1985 when the agency substantially revised NDA regulations to  include a provision describing the characteristics of adequate and well- controlled trials. These concepts have also been addressed, in part, in  other agency guidance documents. However, FDA officials saw the need  for more detailed guidance as they noticed many errors, especially related  to the selection of a non-inferiority margin, in sponsors’ execution of these  trials. FDA officials also expect that the use of non-inferiority trials will  rise as more drugs become available to prevent death or serious illness  and the use of placebos may become unethical.
FDA’s March 2010 guidance explains when non-inferiority trials may be  used to establish a drug’s effectiveness. The guidance states that these  trials are generally used when an available treatment is known to provide  an important benefit—for example, the prevention of death or irreversible  harm. In these cases, it would be considered unethical to use a placebo in  a clinical trial. The guidance also states that non-inferiority trials may only  be used when they are capable of measuring the effect of the new drug in  the study—that is, when the active control is able to consistently  demonstrate its expected effect in the non-inferiority trial. FDA’s March  2010 guidance explains that non-inferiority trials may not be able to  demonstrate the effectiveness of drugs treating certain indications  because not all drugs have a consistent effect in treating these  indications. The guidance also offers suggestions for other types of trials  that may be useful in demonstrating a drug’s effectiveness in cases where  a non-inferiority trial is unable to provide evidence of effectiveness.
The March 2010 guidance provides detailed recommendations on how to  select an active control. For example, when more than one potential active  control exists, the guidance recommends that the most effective drug be  chosen as the active control. In addition, the frequently asked questions  section also clarifies that the active control does not need to be FDA- approved for the indication. However, FDA officials we interviewed stated  that active controls used in non-inferiority trials are usually FDA- approved. If the active control is not FDA-approved, FDA asks sponsors to  provide evidence of the active control’s effectiveness.
FDA’s March 2010 guidance also offers detailed advice on a range of other  topics related to the use and interpretation of non-inferiority trials. For  example, it suggests two methodologies that can be used to set the margin,  offers step-by-step instructions on how to use each of these approaches,  and addresses the role of clinical judgment in determining the margin. It  also explains how to adjust the margin to account for some of the  uncertainties related to non-inferiority trials, such as differences between  the planned non-inferiority trial and prior trials that measured the  effectiveness of the active control. The guidance offers advice on how to  determine the proper number and type of patients to enroll in the trial, and  how to select an endpoint. For example, the guidance states that the  endpoint should be “one for which there is a good basis for knowing the  effect of the active control.”
Most of the experts we interviewed who reviewed FDA’s March 2010  guidance told us that they thought the recommendations it included were  clear and detailed, and addressed the key principles involved in  conducting non-inferiority trials. Some experts noted that the guidance’s  frequently asked questions and examples were useful in illustrating the  key principles described in the document, and said that FDA’s  recommendations would help sponsors appropriately use these trials to  prove a drug’s effectiveness.
While experts we interviewed who reviewed FDA’s March 2010 guidance  noted that it addressed key principles, most identified additional technical  issues that they would have liked this guidance to have addressed. For  example, the March 2010 guidance does not address how the use of a  surrogate endpoint impacts the design and interpretation of a non- inferiority trial. FDA officials told us that the guidance applies to non- inferiority trials that use surrogate endpoints. However, some experts we  interviewed noted that such trials are difficult to design and interpret;  therefore, additional guidance on this topic may be helpful. Since the non- inferiority margin represents the maximum clinically acceptable extent to  which the new drug can be less effective than the active control, experts  told us that sponsors would need to translate the drug’s effect on a  surrogate endpoint into its expected effect on a clinical endpoint in order  to calculate the non-inferiority margin and interpret the trials’ results.  Some experts also noted that the guidance does not include enough  detailed instructions on how to estimate the effect of the active control in  the non-inferiority trial. Finally, some experts who reviewed FDA’s March  2010 guidance told us that they wished the guidance more emphatically  stated that non-inferiority trials should only be used as a last resort when  seeking drug approval.
section_title: FDA’s March 2010 Draft Guidance Provides Broader and More Comprehensive Recommendations to Supplement Previously Issued Indication-Specific Guidance Documents
paragraphs: FDA’s March 2010 draft guidance provides broader and more  comprehensive information about the use of non-inferiority trials,  supplementing other indication-specific guidance documents the agency  had already issued. The objective and content of these two types of  guidance documents differ. The March 2010 guidance offers  comprehensive information on one topic, non-inferiority trials, that may be  generally applied for all drugs using these trials. In contrast, FDA’s  indication-specific guidance documents present recommendations on  many topics—including trial design—for consideration in developing  drugs to treat a particular indication or set of indications. Some of these  indication-specific documents provide recommendations on how to use  non-inferiority trials for that particular indication; for example, by  suggesting a specific margin or a specific endpoint. However, unlike FDA’s  March 2010 guidance, not all indication-specific guidance documents  include information on all of the key principles involved in using a non- inferiority trial to establish a drug’s effectiveness. In addition, these  indication-specific guidance documents do not include the same level of  detail on the key principles that is in the March 2010 guidance. For  example, several of FDA’s indication-specific guidance documents state  that sponsors should justify their selection of non-inferiority margins in  their NDAs. However, in these documents FDA does not elaborate on the  methods sponsors could use to select or justify the margins. In contrast,  the March 2010 non-inferiority guidance provides detailed instructions on  how to calculate the margin.
FDA’s indication-specific guidance documents provide sponsors with  additional clarity on when non-inferiority trials may be used to establish  the effectiveness of drugs treating a particular indication. From January  2002 through June 2010, FDA issued 17 guidance documents that state the  agency’s position regarding the use of non-inferiority trials in  demonstrating the effectiveness of drugs treating certain indications. In  these indication-specific guidance documents, FDA stated that non- inferiority trials may be able to demonstrate the effectiveness of drugs  treating eight indications, including those for HIV, cancer, diabetes  mellitus, and certain severe infections. During the same period, FDA also  issued nine indication-specific guidance documents which state that non- inferiority trials may not be able to demonstrate the effectiveness of drugs  treating other indications—including some less severe infections such as  sinusitis and acute bacterial otitis media (ear infections)—because the  agency has been unable to identify available drugs that have a consistent  effect and could serve as active controls in non-inferiority trials. (See   table 3.) Appendix II identifies the guidance documents FDA has issued  with information on the use of non-inferiority trials from January 2002  through June 2010, including indication-specific documents as well as the  March 2010 draft guidance on non-inferiority trials.
Our review of FDA’s indication-specific guidance showed that the agency  has become more conservative in allowing evidence from non-inferiority  trials to demonstrate the effectiveness of new drugs. First, FDA has  revised its view regarding when non-inferiority trials may be used. Prior to  2007, for example, FDA had approved drugs treating several less severe  infections—including acute bacterial sinusitis, acute bacterial otitis media,  and acute bacterial exacerbations of chronic bronchitis—on the basis of  evidence from non-inferiority trials. Experts we interviewed noted that  these infections can often be resolved without treatment—and thus it is  difficult to estimate the effect that an active control drug would have in a  non-inferiority trial. In 2007 and 2008, FDA issued several guidance  documents stating that non-inferiority trials may not be able to  demonstrate the effectiveness of drugs treating these indications. Second,  FDA has become more rigorous in its review of evidence from non- inferiority trials. For example, prior to 2001, FDA’s guidance on the  development of anti-infective drugs had not advised sponsors to  scientifically calculate or justify their selected non-inferiority margins—a  step that FDA’s March 2010 guidance recommends.
section_title: Agency Comments
paragraphs: We provided a draft of this report to HHS for review. We received  technical comments from HHS, which we incorporated as appropriate.
As agreed with your offices, unless you publicly announce the contents of  this report earlier, we plan no further distribution until 30 days from the  report date. At that time, we will send copies to the Commissioner of FDA  and appropriate congressional committees. The report also will be  available at no charge on the GAO Web site at http://www.gao.gov.
If you or your staff have any questions about this report, please contact me  at (202) 512-7114 or crossem@gao.gov. Contact points for our Offices of  Congressional Relations and Public Affairs may be found on the last page  of this report. GAO staff who made major contributions to this report are  listed in appendix III.
section_title: Appendix I: New Drug Applications Approved on the Basis of Evidence from Non-Inferiority Trials
paragraphs: The Food and Drug Administration (FDA) approved 18 new drug  applications (NDA) that were submitted from fiscal year 2002 through  fiscal year 2009 on the basis of evidence from non-inferiority trials. The  majority of these were antimicrobial drugs, such as those that treat  bacterial, viral, and fungal infections. (See table 4.)
section_title: Appendix II: FDA Guidance Documents Issued with Information on the Use of Non- Inferiority Trials
paragraphs: From January 2002 through June 2010, FDA issued 17 indication-specific  guidance documents that included information about non-inferiority trials,  and one guidance document that included broad recommendations  regarding the use of non-inferiority trials.
section_title: Appendix III: GAO Contact and Staff Acknowledgments
subsections: section_title: GAO Contact
section_title: Acknowledgments
paragraphs: In addition to the contact named above, Geri Redican-Bigott, Assistant  Director; Kathleen Diamond; Carolyn Garvey; Cathy Hamann; Julian  Klazkin; Kaitlin McConnell; and Patricia Roy made key contributions to  this report.