Meta-Analyses of Randomized 
Controlled Clinical Trials to 
Evaluate the Safety of Human 
Drugs or Biological Products 
Guidance for Industry 
 
 
DRAFT GUIDANCE
 
This guidance document is being distributed for comment purposes only. 
Comments and suggestions regarding this draft document should be submitted within 90 days of 
publication in the Federal Register of the notice announcing the availability of the draft guidance. 
Submit electronic comments to https://www.regulations.gov. Submit written comments to the 
Dockets Management Staff (HFA-305), Food and Drug Administration, 5630 Fishers Lane, Rm. 
1061, Rockville, MD 20852. All comments should be identified with the docket number listed in 
the notice of availability that publishes in the Federal Register. 
 
For questions regarding this draft document contact (CDER) Scott Goldie at 301-796-2055 or 
(CBER) Office of Communication, Outreach, and Development, 800-835-4709 or 240-402-8010. 
 
 
 
 
U.S. Department of Health and Human Services 
Food and Drug Administration 
Center for Drug Evaluation and Research (CDER) 
Center for Biologics Evaluation and Research (CBER) 
 
 
November 2018 
Drug Safety 
 
 
   
Meta-Analyses of Randomized 
Controlled Clinical Trials to 
Evaluate the Safety of Human 
Drugs or Biological Products 
Guidance for Industry
 
 
 
Additional copies are available from: 
 
Office of Communications, Division of Drug Information 
Center for Drug Evaluation and Research 
Food and Drug Administration 
10001 New Hampshire Ave., Hillandale Bldg., 4th Floor 
Silver Spring, MD 20993-0002  
Phone: 855-543-3784 or 301-796-3400; Fax: 301-431-6353 
Email: druginfo@fda.hhs.gov  
https://www.fda.gov/Drugs/GuidanceComplianceRegulatoryInformation/Guidances/default.htm 
 
and/or 
 
Office of Communication, Outreach and Development 
Center for Biologics Evaluation and Research 
 Food and Drug Administration 
10903 New Hampshire Ave., Bldg. 71, Room 3128 
Silver Spring, MD 20993-0002 
Phone: 800-835-4709 or 240-402-8010 
Email: ocod@fda.hhs.gov 
https://www.fda.gov/BiologicsBloodVaccines/GuidanceComplianceRegulatoryInformation/Guidances/default.htm  
 
 
 
 
U.S. Department of Health and Human Services 
Food and Drug Administration 
Center for Drug Evaluation and Research (CDER) 
Center for Biologics Evaluation and Research (CBER) 
 
 
 
November 2018 
Drug Safety 
  Contains Nonbinding Recommendations 
Draft  Not for Implementation 
 
TABLE OF CONTENTS 
I.  INTRODUCTION............................................................................................................. 1 
II.  BACKGROUND AND SCOPE ....................................................................................... 1 
III.  THE QUALITY AND RELEVANCE OF CANDIDATE TRIALS ............................. 4 
A.  Basic Principles .............................................................................................................................. 4 
B.  Consistency with Standards for Adequate and Well-Controlled Trials ................................... 4 
C.  Outcome Definition and Ascertainment ...................................................................................... 5 
D.  Duration of Exposure and Length of Follow-Up ......................................................................... 7 
E.  Subject Populations ....................................................................................................................... 8 
F.  Dosing and Comparators .............................................................................................................. 8 
G.  Relevance to Current Medical Practice ....................................................................................... 8 
H.  Availability of Subject-Level Data................................................................................................ 9 
I.  Quality over Quantity .................................................................................................................... 9 
IV.  THE IMPORTANCE OF PRESPECIFICATION AND TRANSPARENCY .......... 10 
A.  Potential for Bias, Multiplicity, and Other Errors ................................................................... 10 
B.  Meta-Analysis Protocol ............................................................................................................... 11 
C.  Reporting Results from a Meta-Analysis ................................................................................... 12 
V.  STATISTICAL METHODOLOGY CONSIDERATIONS ........................................ 13 
A.  Overview ....................................................................................................................................... 13 
B.  Statistical Properties of Risk Estimates and Hypothesis Tests ................................................ 14 
C.  Heterogeneity ............................................................................................................................... 16 
D.  Sensitivity Analysis ...................................................................................................................... 17 
VI.  STRENGTH OF EVIDENCE AND REGULATORY DECISIONS ......................... 18 
A.  Critical Factors in Determining the Strength of Evidence ....................................................... 18 
B.  Hierarchy of Evidence for Decision-Making ............................................................................. 19 
VII.  EXAMPLES .................................................................................................................... 21 
A.  Example 1: Antidepressant Use and Suicidal Events in Adults ............................................... 21 
B.  Example 2: Tiotropium and Cardiovascular Events ................................................................ 24 
VIII.  REFERENCES ................................................................................................................ 25 
 
  Contains Nonbinding Recommendations 
Draft  Not for Implementation 
 
1  Meta-Analyses of Randomized Controlled Clinical Trials to 
2  Evaluate the Safety of Human Drugs or Biological Products 
3  Guidance for Industry1 
4  This draft guidance, when finalized, will represent the current thinking of the Food and Drug 
5  Administration (FDA or Agency) on this topic. It does not establish any rights for any person and is not 
6  binding on FDA or the public. You can use an alternative approach if it satisfies the requirements of the 
7  applicable statutes and regulations. To discuss an alternative approach, contact the FDA staff responsible 
8  for this guidance as listed on the title page.  
9  I.  INTRODUCTION  
10  This document provides guidance to applicants submitting investigational new drug applications 
11  (INDs), new drug applications (NDAs), biologics license applications (BLAs), or supplemental 
12  applications on the use of meta-analyses of randomized controlled clinical trials (RCTs) to 
13  evaluate the safety of human drugs or biological products within the framework of regulatory 
14  decision-making.2 This guidance is also intended for FDA reviewers and for third-party entities 
15  that prepare or evaluate meta-analyses assessing the safety of drug products. Specifically, this 
16  guidance describes the factors FDA intends to consider when evaluating the strength of evidence 
17  provided by a meta-analysis studying the safety of drugs. 
18   
19  In general, FDAs guidance documents do not establish legally enforceable responsibilities. 
20  Instead, guidances describe the Agencys current thinking on a topic and should be viewed only 
21  as recommendations, unless specific regulatory or statutory requirements are cited. The use of 
22  the word should in Agency guidances means that something is suggested or recommended, but 
23  not required.  
24  II.  BACKGROUND AND SCOPE  
25  Evaluating the safety of drug products, both before approval and after marketing, is a 
26  fundamental responsibility of the FDA. This evaluation often requires combining and 
27  summarizing information from multiple sources, and meta-analysis is a useful tool for this 
28  purpose. The term meta-analysis, as used in this document, refers to the combining of evidence 
29  from relevant studies using appropriate statistical methods to allow inference to be made to the 
30  population of interest. The most common reason for performing a meta-analysis is to provide an 
31  estimate of a treatment effect or measure of relative risk associated with an intervention and to 
32  quantify the uncertainty about the estimated effect or risk, when data from a single existing study 
33  are insufficient for this purpose, and the conduct of a new, large study would be impractical, take 
                                                 
1 This guidance has been prepared by the Office of Biostatistics in the Center for Drug Evaluation and Research, in 
cooperation with the Center for Biologics Evaluation and Research, at the Food and Drug Administration. 
 
2 For the purposes of this guidance, all references to drugs include both human drugs and biological products unless 
otherwise specified. 
  1  Contains Nonbinding Recommendations 
Draft  Not for Implementation 
 
34  too long, or be unethical. The term meta-analysis sometimes refers to the quantitative synthesis 
35  in a systematic review (Cochrane Handbook 2011) and the term systematic review refers to the 
36  broader effort, including defining objectives and selecting and evaluating studies, as well as 
37  synthesis. We use the term meta-analysis more broadly to include consideration of study 
38  selection as well as overall design issues such as prespecification and reporting.  
39  Unless a randomized controlled clinical trial is prospectively designed with a particular safety 
40  outcome as its primary endpoint and sized accordingly, the trial may not have sufficient sample 
41  size to detect important adverse consequences of drugs and to reliably evaluate whether there is 
42  increased risk of such events. This is because most serious drug-induced adverse events (1) are 
43  rare or (2) occur at only slightly increased frequency compared to background rates and are not 
44  obviously drug-related (e.g., cardiovascular events, cancers). Meta-analysis is most useful in the 
45  latter case, to detect and quantify an increased risk over the background rate of the safety event. 
46  For the former case, when events are rare and not expected to occur in the target population, 
47  meta-analyses may still be useful for improving the precision of the estimate of risk.  
48  Meta-analysis factors into FDAs evaluation of potential safety issues in a variety of ways: 
49    Meta-analyses may be conducted by sponsors and submitted to FDA as part of an 
50  IND, NDA, BLA or supplemental submission.  
51    FDA may ask a sponsor to conduct a prospective meta-analysis, as it has 
52  recommended for sponsors of new antidiabetic therapies to treat type 2 diabetes in the 
53  draft guidance for industry, Diabetes Mellitus  Evaluating Cardiovascular Risk in 
54  New Antidiabetic Therapies to Treat Type 2 Diabetes.3  
55    FDA may initiate its own meta-analysis in response to safety signals that FDA is 
56  aware of, using study data FDA has access to, but that may be unavailable to sponsors 
57  and other researchers. These meta-analyses typically have prospective protocols to 
58  address issues of bias and multiplicity, as discussed later in this document. 
59    FDA may evaluate a meta-analysis conducted by an external party that raises a safety 
60  concern about a marketed product.  
61  Because regulatory actions may stem from a meta-analysis, it is important that rigorous 
62  principles be applied to such studies.  In this guidance, the important principles underlying best 
63  practices for safety meta-analysis and the way that FDA intends to factor adherence to those 
64  principles into its decision-making are described. An overview of the most important principles 
65  presented in this guidance is as follows:  
66    Prespecification and transparency are recommended, as they enable a thorough 
67  evaluation of the meta-analysis.  
68    The criteria for selecting which trials to include should be determined prior to 
69  conducting the meta-analysis. The selection of the studies should not be based on the 
                                                 
3 We update guidances periodically. To make sure you have the most recent version of a guidance, check the FDA 
Drugs or Biologics guidance web pages at: 
https://www.fda.gov/Drugs/GuidanceComplianceRegulatoryInformation/Guidances/default.htm. 
https://www.fda.gov/BiologicsBloodVaccines/GuidanceComplianceRegulatoryInformation/Guidances/default.htm  
  2  Contains Nonbinding Recommendations 
Draft  Not for Implementation 
 
70  trial outcomes, but rather on trial quality and consistency of critical design elements, 
71  and should be executed by parties masked to the outcomes of the trials, whenever 
72  possible. 
73    The quality and relevance of the individual trials and the quality of the trial data are 
74  critical determinants of the quality of the meta-analysis itself. Outcome ascertainment 
75  and adequacy of exposure periods are two of the most important determinants of trial 
76  quality.  
77    Meta-analysis conducted to meet safety objectives often requires re-purposing trials 
78  that were originally designed to meet efficacy objectives. This can be challenging, 
79  particularly if subject-level data are not available. 
80    Meta-analysis based solely on published trials is particularly problematic because of 
81  the potential for bias and error, both known and unrecognized. 
82    Generally accepted principles of good statistical practice should be followed in 
83  selecting the statistical methods to be used for meta-analysis (but this guidance is not 
84  prescriptive as to the choice of method). 
85  This guidance applies to meta-analyses conducted in both pre-market and post-market settings. 
86  In the pre-market setting, the number and scope of trials may be limited, because the drugs are 
87  not yet approved for marketing, and these limitations may affect the ability to address the safety 
88  question of interest. In the post-market setting, the number and variety of trials available for 
89  inclusion are usually larger, as is the number of parties able to conduct the meta-analysis. In both 
90  pre- and post-market settings, the important principles guiding a well-planned and well-executed 
91  meta-analysis apply. 
92  This document focuses specifically on meta-analyses conducted for purposes of safety evaluation 
93  using data from RCTs. Meta-analyses conducted to evaluate a products effectiveness, either 
94  overall or within specific subgroups, are occasionally of interest to FDA, but the primary use of 
95  meta-analysis in the regulatory setting is for assessment of product risk. While meta-analyses of 
96  non-randomized studies may be informative for the assessment of certain safety outcomes, the 
97  issues related to such a meta-analysis are more complex, and the interpretation of the results 
98  more controversial. Meta-analyses of observational studies are therefore not addressed in this 
99  guidance. 
100  Meta-analyses are conducted for both exploratory and confirmatory purposes. The primary focus 
101  of this guidance, however, is on meta-analyses with predefined hypotheses that are designed to 
102  confirm a suspected risk associated with a drug rather than on exploratory meta-analyses.  
103  The subsequent sections of this guidance provide a detailed discussion of the important elements 
104  used in evaluating meta-analyses for regulatory purposes. In section III, the importance of the 
105  quality and relevance of the component trials included in a meta-analysis and the quality of the 
106  data from those trials are discussed. In section IV, the importance of prespecification and 
107  transparency in designing, conducting, and reporting a meta-analysis is described. In section V, 
108  the use of recommended statistical methods is discussed. In section VI, we summarize these 
109  technical considerations and discuss how they may be factored into a regulatory decision. 
  3  Contains Nonbinding Recommendations 
Draft  Not for Implementation 
 
110  Section VII provides two examples illustrating the range of meta-analyses conducted for safety 
111  evaluation and FDAs use of the evidence provided by each.  
112  III.  THE QUALITY AND RELEVANCE OF CANDIDATE TRIALS  
113  A.  Basic Principles 
114  Deciding what trials to include in a meta-analysis is an important step in the design and conduct 
115  of a high-quality meta-analysis. The major determinants for this decision should be the quality 
116  and relevance of the individual trials and the data collected in those trials. The component trials 
117  of a meta-analysis should be able to address the safety objectives of the analysis and be of 
118  sufficient quality to provide evidence useful for regulatory decision-making. The following are 
119  important factors to consider in determining whether the individual trials and associated data are 
120  of sufficient quality and relevance to ensure the validity of the meta-analysis: 
121    The extent to which the component trials are consistent with established standards for 
122  the design and conduct of adequate and well-controlled clinical trials  
123    The quality and completeness of safety outcome ascertainment in each trial 
124    The appropriateness of exposure and follow-up periods for estimating risk 
125    The appropriateness of the component trials inclusion/exclusion criteria for defining 
126  the population at risk 
127    The appropriateness of the comparator used in each trial and of the doses for the test 
128  drug and comparator 
129    The relevance of the candidate trials to current medical practice 
130    The availability of subject level data from each trial 
131  These factors are discussed further in the subsections that follow. 
132  B.  Consistency with Standards for Adequate and Well-Controlled Trials 
133  The knowledge base, literature, and published guidelines for designing, conducting, and 
134  analyzing well-controlled clinical trials to demonstrate efficacy in support of an NDA or BLA 
135  are extensive and well-known (see, e.g., E9 Statistical Principles for Clinical Trials, 
136  International Council on Harmonisation (ICH) of Technical Requirements for Pharmaceuticals 
137  for Human Use). The same principles apply to the individual component trials of a meta-
138  analysis, and the extent to which the component trials satisfy these principles has strong bearing 
139  on the quality of the meta-analysis to which they contribute. Notably, however, trials that are 
140  well-designed to measure the effect of a drug on a particular efficacy outcome may not 
141  necessarily be well-designed to measure an effect on another outcome, particularly an 
142  uncommonly occurring safety outcome, as discussed further in section III.C. 
143  Some study designs may cause a candidate trial to be discouraged from inclusion in the meta-
144  analysis. For example, randomized withdrawal studies, in which all subjects initially receive the 
145  drug and are then randomized to either remain on the drug or withdraw to a placebo or active 
146  control drug, may not be recommended for a safety meta-analysis. In these studies, subjects who 
147  cannot tolerate the test drug are excluded from the randomized portion of the trial, and the study 
  4  Contains Nonbinding Recommendations 
Draft  Not for Implementation 
 
148  population may therefore not accurately represent the population at risk. Additionally, depending 
149  on the period of exposure needed for the adverse effect to occur, the initial exposure to the drug 
150  may result in events in both randomized groups and an underestimate of the relative risk. 
151  Crossover studies in which subjects receive different treatments at different periods of time may 
152  not be recommended for evaluating safety outcomes, if exposure to a treatment in one period can 
153  result in an adverse event occurring in a later period. Washout periods for safety outcomes may 
154  need to be longer than for efficacy outcomes. Other non-standard study designs such as 
155  enrichment trials, trials with add-on therapies, adaptive trials, and trials stopped at interim may 
156  raise similar issues. 
157  C.  Outcome Definition and Ascertainment 
158  A high-quality meta-analysis has a carefully defined outcome variable with appropriate 
159  ascertainment procedures prospectively implemented in the component trials such as specific 
160  protocol-defined procedures for data collection and adjudication of safety outcomes. For 
161  example, if the outcome of interest is myocardial infarction, the protocol might instruct the 
162  investigators to collect laboratory and electrocardiogram data for suspected events during the 
163  trial. The results of these procedures might then be subject to adjudication by an independent 
164  panel to strengthen the evidence that a case event is real. Such procedures, however, are used 
165  primarily to assess effectiveness outcomes (does the treatment reduce myocardial infarction 
166  rates) and are not commonly used to assess safety outcomes, unless there is a specific concern 
167  known and planned for prior to study start (e.g., cardiovascular outcomes in studies of Type 2 
168  diabetes drugs; suicidal events in studies of antidepressant drugs). Although prospective 
169  collection and adjudication of safety outcomes are desirable, they are usually not feasible, 
170  particularly in the most common setting of evaluating a new, unanticipated safety signal with 
171  data from trials already completed.  
172  When the component trials are not prospectively designed to produce accurate ascertainment of 
173  the meta-analysis safety outcome, retrospective identification and adjudication of events will 
174  usually be recommended. In this situation, the safety outcome of interest should be clearly 
175  defined, and the identification and adjudication of events should be performed while masked. For 
176  example, in the antidepressants and suicidal events meta-analysis (section VII, Example 1), 
177  where suicidality was not specifically assessed, predefined search criteria were applied to 
178  adverse event data collected in the component trials. Based on the results of the search, 
179  narratives of candidate events were created, and a group of experts masked to treatment 
180  assignment classified the events into validated suicidal outcome categories. This resource 
181  intensive effort required subject-level data not directly available in the original trial datasets. A 
182  detailed meta-analysis protocol was developed that described the procedures necessary for 
183  obtaining and adjudicating the outcome data of interest prior to implementing those procedures.  
184  Measurement bias (such as an over- or under-estimation of the rate of events because of 
185  imprecise or individualized interpretation of adverse event reporting) factors into determining 
186  whether outcome ascertainment is sufficient for a high-quality meta-analysis. Biases common to 
187  both treatment and control groups can occur when an outcome variable does not accurately 
188  represent the safety outcome of interest (e.g., is not specific enough, causing many irrelevant 
189  events to be reported, or is too narrowly defined, causing many events to be missed). Both 
  5  Contains Nonbinding Recommendations 
Draft  Not for Implementation 
 
190  reporting problems may result in reduced power or a biased effect measure, but will not 
191  completely eliminate the ability to detect an effect. Of greater concern are reporting problems 
192  that can affect treatment groups differently, as they can eliminate the ability to detect an effect 
193  when one exists or that create the appearance of an effect when one does not exist.  
194  Biased ascertainment of outcomes is one important concern in unmasked trials, where 
195  investigators or subjects may unconsciously, or consciously, under- or over-report medical 
196  events based on the known treatment assignment. Even in double-masked trials, there is a 
197  potential for differential bias to occur in safety reporting, especially when safety outcomes were 
198  not of primary interest in designing the trial. For example, a drug may cause discoloration of the 
199  urine, which in turn may lead to more evaluations and subsequent diagnoses of kidney disease. If 
200  anticipated, the trial protocols could have included an evaluation for kidney disease at scheduled 
201  times during the trials, thereby reducing the potential for biased reporting of that safety outcome. 
202  Several strategies should be considered to minimize the impact of measurement bias. The use of 
203  safety outcomes that can be diagnosed readily and unambiguously, often called hard outcomes, 
204  can help minimize bias due to outcome ascertainment in a meta-analysis. For example, if vital 
205  status at the end of the study is known for all patients in all of the component trials, then use of 
206  death as the safety outcome effectively avoids any potential for ascertainment bias. If ischemic 
207  cardiovascular outcomes are of interest, ascertainment of myocardial infarction and stroke will 
208  be less prone to ascertainment bias than less specific events such as transient ischemic attack or 
209  angina. Excluding the less specific events or events that are difficult to ascertain objectively will 
210  probably reduce the power of the meta-analysis to detect a safety signal as well as the precision 
211  of the risk estimate that results, but the reduction in ascertainment bias may outweigh these 
212  losses. Precision and power can be quantified and reported with the meta-analysis results, 
213  whereas bias is typically unknown and difficult to measure. In general, reducing bias in a meta-
214  analysis should be given greater weight than increasing precision and power. 
215  It is important to define the period within which the safety outcome of interest is to be measured. 
216  For example, a safety outcome corresponding to the occurrence of anaphylactic events may call 
217  for the primary focus to be placed on the period of initial drug exposure, with a secondary focus 
218  on the entire drug exposure period. Including events beyond the initial exposure period may 
219  result in underestimation of the risk attributable to the drug. In cases where it is known that the 
220  effect of the drug diminishes when the drug is stopped, it might be recommended for the primary 
221  analysis to count outcomes only during the time a subject is on the drug (such as an on-treatment 
222  analysis).  
223  Ideally, outcome definition and ascertainment should be as uniform as possible across the 
224  component trials. Trial-to-trial differences can introduce heterogeneity in safety outcomes, 
225  increasing the variability of the meta-analytic estimate of risk. Differences in outcome definition 
226  and ascertainment may be confounded with other trial design or subject population 
227  characteristics, making observed differences in risk measures difficult to interpret.  
228  Outcome definition and ascertainment are particular problems for meta-analyses that rely 
229  exclusively on published trial data. Information taken from published articles about the 
230  component trials may be incomplete or lack specificity. Publications may not report on the safety 
  6  Contains Nonbinding Recommendations 
Draft  Not for Implementation 
 
231  outcome of interest, and even when the outcome is reported, important details may be lacking, 
232  including whether the event occurred on or off randomized treatment and whether the outcome 
233  was defined a priori and uniformly across trials. Protocols, study reports, and subject-level data 
234  from the component trials are often important to determine whether the trial outcomes are 
235  adequate for supporting a high quality meta-analysis. 
236  The definition of the safety outcome, the source data and any adjudication procedures that may 
237  have been employed should be prespecified in the meta-analysis protocol and consistently 
238  applied to all component trials, if possible (see Section IV.B).  
239  D.  Duration of Exposure and Length of Follow-Up  
240  The duration of exposure and length of follow-up for each of the candidate trials should be 
241  factored into the criteria for trial inclusion. For an outcome with delayed appearance, such as 
242  cancer or bone injury, the inclusion of short-term trials may not be recommended. When subject-
243  level data are available, analysis methods can be used to identify and account for differences in 
244  trial duration across studies (see Section V). Without subject-level data, it may not be possible to 
245  account for differences in duration, depending on the level of detail provided by the summary 
246  information available from each trial, and some trials may need to be excluded as a result. 
247  Subjects prematurely stopping assigned drug or withdrawing from the trial can affect the 
248  comparability of subject groups with respect to safety outcomes ascertained over the course of 
249  the treatment or study period. The dropout pattern may result in dissimilar observation time 
250  between the two groups, resulting in more opportunity to observe the safety outcome in one 
251  group compared to the other. Simple adjustments for person-time of observation may not be 
252  sufficient to correct for non-comparability, because these adjustments assume constant hazards 
253  across time. The risk of the event may not be constant over time if, for example, the safety 
254  outcome tends to occur either early or late during treatment. Time-to-event analysis may also be 
255  insufficient if the dropout rates are indicative of informative censoring; for example, if the 
256  adverse events resulting in early discontinuations are similar to or predecessor events of the 
257  safety outcome.  
258  When reviewing the component trials of a meta-analysis, it is important to consider the 
259  possibility of differential follow-up and informative censoring. Examining summary statistics 
260  and graphics by subject group of on-assigned drug time and follow-up time is usually helpful for 
261  this purpose, as is an examination of the stated reasons for stopping assigned-drug or 
262  discontinuing participation in the trial by subject group. The criteria for excluding individual 
263  trials for these reasons should be specified a priori and described in the meta-analysis protocol 
264  and analysis plan (see Section IV). If incorporated in the trial inclusion and exclusion criteria 
265  (applied to determine the component trials of the meta-analysis), a review to identify differential 
266  dropout rates should be performed masked to the safety outcome measurements. Regardless of 
267  the decision on inclusion, data summaries should be provided in the meta-analysis report to 
268  permit consideration of these issues.  
  7  Contains Nonbinding Recommendations 
Draft  Not for Implementation 
 
269  E.  Subject Populations 
270  Wherever possible, the subject population for component trials should reflect the patient 
271  population hypothesized to be adversely affected by the drug. For cardiovascular safety 
272  outcomes, for example, trials that enrolled subjects with pre-existing cardiovascular risk factors 
273  may improve the ability of the meta-analysis to detect any risk associated with the drug. 
274  Conversely, including trials that excluded subjects with certain risk factors may limit the ability 
275  to detect risk. The inclusion/exclusion criteria of the component trials should be reviewed to 
276  determine if the corresponding subject populations are consistent with the objectives of the meta-
277  analysis. 
278  F.  Dosing and Comparators 
279  Although uniformity of dosing regimens and therapeutic indications studied across component 
280  trials is desirable, it may be that trials including other doses or conducted in other indications can 
281  contribute to the meta-analysis. For example, in some circumstances, it may be assumed that if a 
282  safety event is not observed at doses higher than the dose or doses approved, it should not occur 
283  at the approved dose or doses. In this scenario, including trials with dosing higher than the 
284  approved dose might be used to rule out an association. Information on dose response 
285  relationships may also support a possible relationship between drug use and safety outcomes. 
286  Similarly, including trials for indications outside the indication of specific interest may be useful 
287  in a safety meta-analysis, if it can be assumed that the association would not depend on the 
288  indicated use. Such assumptions can be examined to some extent through sensitivity analyses 
289  conducted on subsets of trials at particular doses or in particular indications (see Section V.D). 
290  The suitability of the comparator drugs in the candidate trials should also be factored into the 
291  meta-analysis inclusion criteria. In some situations, the ideal comparator is a placebo, since a 
292  placebo cannot cause the safety outcome under investigation. However, placebo-controlled trials 
293  may not be feasible or ethical in certain disease areas. If trials with active drug comparators are 
294  used, attempts should be made a priori to determine if the active comparator is associated with 
295  the safety outcome of interest. The protocol specifications for concomitant therapy in the 
296  individual trials are also relevant, since concomitant therapies may be associated with the safety 
297  outcome. 
298  G.  Relevance to Current Medical Practice  
299  Changes over time in the practice of medicine may affect the usefulness of some trials for 
300  contributing data to a meta-analysis. Older trials may no longer be relevant, if medical practice 
301  has changed such that current practices are able to prevent or reduce the occurrence of the safety 
302  outcome under investigation.  Sensitivity analyses can be used to examine estimated risks as a 
303  function of the dates the component trials were conducted to determine if calendar trends pose a 
304  problem.  
  8  Contains Nonbinding Recommendations 
Draft  Not for Implementation 
 
305  H.  Availability of Subject-Level Data 
306  The availability of subject-level data is an important consideration in deciding which studies to 
307  include in the meta-analysis. For reasons already discussed, subject-level data improve the 
308  quality of the meta-analysis by providing the ability to evaluate important quality factors of the 
309  component trials and possibly correct for any deficiencies identified, particularly poor outcome 
310  assessment or insufficient exposure periods. Subject-level data also allow for a broader range of 
311  analysis methods to be used and an examination of subgroups (see Section V). Note, however, 
312  that in some cases, meta-analyses based on only trial-level summary data may be able to identify 
313  or rule out risks associated with a drug. If so, then the criteria for determining which trials to 
314  include in a trial-level meta-analysis should be carefully considered; the principles described in 
315  this section apply to trial-level meta-analyses just as they do to subject-level meta-analyses. 
316  I.  Quality over Quantity 
317  There is often a desire to include as many trials as possible in a meta-analysis to both increase 
318  the sample size and enhance the generalizability or external validity of the findings. Including 
319  trials that are of poor quality, however, does not accomplish this. The findings from a meta-
320  analysis of a limited set of trials, selected with careful attention to trial and data quality, the 
321  intended use of the product, and combined using appropriate statistical methods, will yield a 
322  more informative answer to the safety question under investigation than a broader set of trials 
323  that includes trials of poor quality.  
324  The criteria used to decide which of the candidate trials will be included in a safety meta-analysis 
325  should be carefully developed, taking into consideration outcome ascertainment and exposure 
326  periods as well as other factors described in the previous subsections. The choices of subject 
327  populations, dosing regimens, comparator arms, background therapy, standard of care, and other 
328  trial features that comprise the meta-analysis inclusion criteria will affect the validity and 
329  interpretation of the meta-analysis findings. Broad inclusion criteria (such as including trials 
330  where outcomes may not be reliably assessed) will likely compromise the internal validity of the 
331  meta-analysis without necessarily improving the external validity. The criteria for trial inclusion 
332  should be well-documented in advance of conducting the meta-analysis. This topic is discussed 
333  in section IV.  
334  Trial inclusion decisions are particularly important for network meta-analyses, which are 
335  designed to assess safety concerns about one drug relative to another, when the two may not 
336  have been studied in the same randomized trial (Ohlssen, Price et al. 2014). Direct comparisons 
337  between drugs within the individual trials included in a network meta-analysis are used to form 
338  indirect comparisons between the two drugs of interest. Because some of the subject group 
339  comparisons are made across trials, it is important that the trials involved in a network meta-
340  analysis be similar in design, subject populations, outcome definitions, and medical practice. 
341  Although the principles in this guidance apply to network meta-analyses, network meta-analyses 
342  have unique considerations beyond what is discussed in this guidance. 
  9  Contains Nonbinding Recommendations 
Draft  Not for Implementation 
 
343  IV.  THE IMPORTANCE OF PRESPECIFICATION AND TRANSPARENCY  
344  The extent of the information that should be considered both before and following the conduct of 
345  a meta-analysis to adequately establish prespecification and transparency is discussed in this 
346  section.  
347  A.  Potential for Bias, Multiplicity, and Other Errors 
348  Meta-analysis is a form of retrospective research in that most meta-analyses are conducted based 
349  on published clinical trials or trials already completed and whose results are known. It is 
350  important to minimize the potential for bias and other errors from sources that are often 
351  characteristic of retrospective research, including: 
352    Prior knowledge of individual study results when selecting the studies to be included 
353  in the meta-analysis 
354    Inclusion of the hypothesis-generating study in a meta-analysis designed to confirm 
355  the hypothesis 
356    Inability to determine the impact of multiplicity on the reported results 
357  Special care is recommended when including trials whose results regarding the safety outcome 
358  of interest are known prior to the conduct of the meta-analysis. Information describing the 
359  knowledge base at the time the meta-analysis was planned will aid in determining the extent of 
360  possible bias that may affect interpretation of the results (e.g., trial outcomes influencing 
361  selection of trials). Prespecification of the criteria used to decide which trials to include before 
362  decisions about individual trials are made is a major mechanism to minimize bias and can help 
363  lessen the impact of this knowledge on the validity of the meta-analysis findings.  
364  As stated earlier, our focus is on meta-analyses conducted to confirm a hypothesized safety risk. 
365  If a safety hypothesis was generated from the results of a specific clinical trial, then drawing 
366  inference from a meta-analysis that includes that trial is problematic. In this case, hypothesis test 
367  results and confidence intervals about the risk estimate are not readily interpretable. If the goal of 
368  the meta-analysis is to summarize existing information and not to make formal inference, then 
369  including the motivating trial may be reasonable. If the motivating trial is included, sensitivity 
370  analyses should be performed with and without the motivating trial to investigate its impact on 
371  the meta-analysis results (See Section V). 
372  Another problem frequently encountered when evaluating the evidence provided by a meta-
373  analysis is the potential for spurious findings due to multiple hypotheses being tested, multiple 
374  outcomes being evaluated, multiple or iterative analyses being conducted and multiple subject 
375  subgroups being investigated (Bender, Bunce et al. 2008). The result is inflation of the Type I 
376  error probability associated with the tests of hypotheses, making the meta-analysis conclusions 
377  difficult to interpret. When each of these sources of multiplicity is not well-described in advance, 
378  it is impossible to apply a statistical method of adjustment for multiplicity because the full range 
379  of factors that were evaluated cannot be determined. And even when the analysis plan does 
380  contain a clear description of the sequence of tests to be conducted (across hypotheses, 
381  outcomes, subgroups, etc.), there may be too little power available for each of the tests to 
  10  Contains Nonbinding Recommendations 
Draft  Not for Implementation 
 
382  confirm the hypothesized safety signal, when appropriate adjustments are applied. Adequate 
383  planning and prespecification of meta-analysis objectives and tests of hypotheses may help 
384  minimize, to some extent, problems due to multiplicity.  
385  B.  Meta-Analysis Protocol 
386  Prespecification, completeness, and transparency are important principles in the reporting of a 
387  meta-analysis, and the reporting begins with the meta-analysis protocol. The protocol should 
388  contain a detailed description of the information available prior to designing the meta-analysis 
389  that motivated the research. Potential problems anticipated in designing the meta-analysis and 
390  the methods planned to manage those problems should be documented. The protocol should be 
391  finalized prior to conducting the meta-analysis and, importantly, be in place prior to the selection 
392  of the component trials. 
393  The meta-analysis protocol should be available through advance publication or other methods of 
394  distribution. This practice has been widely adopted for clinical trials via use of the web site, 
395  https://clinicaltrials.gov/. There are several repositories for the protocols, such as PROSPERO 
396  (Chien, Khan et al. 2012). Having protocols appear in the same publication as the meta-analysis 
397  findings is generally insufficient to provide such assurance.  
398  Following is a list of the broad topics a meta-analysis protocol should include. Each is discussed 
399  further in the paragraphs that follow: 
400    The planned purpose of the meta-analysis 
401    The background information available at the time of protocol development that 
402  motivated the meta-analysis  
403    The design features of the meta-analysis, including outcome definition and 
404  ascertainment, exposure periods and assessment, comparator drugs, and target subject 
405  population 
406    A description of the search strategy that will be used to identify candidate trials and 
407  the criteria that will be applied for trial selection  
408    The analysis strategy for conducting the meta-analysis, including planned subgroup 
409  analyses and sensitivity analyses 
410  Planned purpose: The planned purpose should be clearly stated in the protocol, with sufficient 
411  background material to explain the reason for conducting the meta-analysis. Examples include: 
412  to estimate a specific risk, to evaluate risk in a subgroup of patients, to identify risk factors or 
413  effect modifiers, to examine whether risk changes over time, or to assess accumulating evidence 
414  on product safety as ongoing studies of the product complete. The weight of evidence provided 
415  by a meta-analysis planned specifically to provide new information or update existing 
416  information about a hypothesized risk of a drug would be considered more compelling than that 
417  from a meta-analysis designed to explore safety signals or relationships among variables with no 
418  stated hypothesis. The distinction is analogous to that made between exploratory and 
419  confirmatory clinical trials in drug development, with the latter guided by pre-specified 
420  objectives reflected in a final protocol prior to study start. 
  11  Contains Nonbinding Recommendations 
Draft  Not for Implementation 
 
421  Background information: The protocol should describe the information available prior to 
422  designing the meta-analysis that served as motivation for the research. Examples include safety 
423  risks identified in a randomized clinical trial of the drug or another drug in the same class, 
424  potential relationships between exposure and safety outcomes shown in post-marketing studies 
425  of health care data, or potential relationships identified during review of spontaneous adverse 
426  event reports. 
427  Design elements: A clear prospective plan can help protect a meta-analysis against bias and 
428  inflation of Type I error by providing the rationale for each design element based on the 
429  knowledge and information available during planning. Without such a plan, it is difficult to 
430  determine which analyses were planned and which were exploratory or suggested as the analysis 
431  progressed. Important among the design elements is outcome ascertainment, including whether 
432  the outcome data were collected as part of the design of the individual trials or retrospectively 
433  collected as part of the meta-analysis; whether the outcome was actively collected from subjects 
434  or passively collected via subject adverse event reports; and whether the outcome was 
435  adjudicated, and, if so, how. Clear definitions of the outcome variable and the follow-up period 
436  for its ascertainment should be stated, with rationale for the choices thereof. The protocol should 
437  state the specific exposure of interest and the comparator. If multiple exposures (multiple doses 
438  of one drug or multiple drugs within a class) or comparators are to be combined, this should be 
439  stated, and the primary exposure and comparator should be identified. 
440  Search and selection criteria: The protocol should describe the search algorithm that will be used 
441  to identify candidate trials to be considered for inclusion in the meta-analysis. Details should 
442  include a description of the sources to be searched, such as the literature or online resources (e.g., 
443  https://clinicaltrials.gov/, https://www.accessdata.fda.gov/scripts/cder/drugsatfda). The trial 
444  inclusion criteria should be described in detail, with the rationale given for each factor used as a 
445  basis for trial selection (see Section III.D). The selection process should be masked to study 
446  outcome and described in the meta-analysis protocol. Note that even if results are known to some 
447  parties, it may be possible to find others who could apply the trial selection criteria for the meta-
448  analysis in an unbiased manner. 
449  Analysis strategy: The protocol should describe the primary analysis strategy for achieving the 
450  study objectives as well as any sensitivity analyses and subgroup analyses planned. The 
451  statistical methods for the primary analysis should be stated in the protocol, with additional 
452  details provided in the statistical analysis plan. The analysis plan should be finalized prior to 
453  conducting the meta-analysis, analogous to the recommendation that a clinical trials analysis 
454  plan be finalized prior to unmasking of treatment codes. Sensitivity analyses should be planned a 
455  priori to assess the impact of any unverifiable assumptions on the meta-analysis results. The 
456  factors that should be considered in choosing the statistical methods are discussed in section V. 
457  C.  Reporting Results from a Meta-Analysis 
458  Results of a meta-analysis should be reported in a way that provides transparency and full 
459  disclosure of the many decisions involved in conducting the meta-analysis. The report should 
460  provide enough detail about the selection of trials, the statistical methods applied in the analyses, 
461  the results of those analyses, and the rationale for and results of any sensitivity analyses carried 
  12  Contains Nonbinding Recommendations 
Draft  Not for Implementation 
 
462  out to enable an evaluation of the impact of bias and multiplicity on the findings and to assess 
463  their strength and credibility. The Preferred Reporting Items for Systematic Reviews and Meta-
464  Analyses (PRISMA) statement provides some recommendations on the reporting of systematic 
465  reviews and meta-analyses (Moher, Liberati et al. 2009). Although not all the PRISMA 
466  components directly apply to meta-analyses that are the focus of this guidance, they should be 
467  considered. 
468  The report should include the results of the search algorithm used to identify candidate trials and 
469  contain enough detail to evaluate the search.  The selection process used to determine which of 
470  the candidate trials were selected for inclusion in the meta-analysis, which should be by applying 
471  the pre-specified criteria, should also be reported. Accounting for the trials that were not selected 
472  and the reasons for their exclusion is as important as accounting for the trials that were selected. 
473  Characteristics of the individual trials included in the meta-analysis should be summarized, 
474  including individual trial design features, durations of exposure and follow-up periods, and 
475  patient populations. The report should describe the sources of any trial-level and subject-level 
476  data used in the meta-analysis. Summaries of subject-level characteristics should also be 
477  provided for the trials to be included in the meta-analysis, including basic demographics, 
478  concomitant medication usage, and other important factors thought to impact the exposure-risk 
479  relationship under investigation. 
480  Any departures from the planned statistical methods should be described, as well as the rationale 
481  for those departures. Additional sensitivity analyses determined to be needed after the protocol 
482  was finalized, because of characteristics of the particular trials selected, unanticipated data issues 
483  encountered during analysis (e.g., zero-event trials), or preliminary findings needing further 
484  exploration, should be described and justified.  
485  Results corresponding to the pre-specified test of hypotheses, supporting analyses, and 
486  sensitivity analyses should be provided in a clear and concise manner, with sufficient detail to 
487  aid in interpretation. Point estimates of absolute or relative risk should be accompanied by 
488  measures of uncertainty, e.g., confidence intervals. Forest plots are recommended for providing 
489  visual summaries of the results from each of the component trials relative to the results of the 
490  meta-analysis. These plots are useful in describing study-to-study heterogeneity. 
491  V.  STATISTICAL METHODOLOGY CONSIDERATIONS 
492  A.  Overview 
493  In this section, general recommendations for selecting the statistical methods that will be used to 
494  combine evidence from the component trials in a safety meta-analysis are discussed. It is not the 
495  goal of this guidance to propose any best method, as no method performs best in all settings, nor 
496  is it the goal to restate the relative performance of methods that are well-established and have 
497  been compared in the literature pertinent to safety meta-analyses. Rather, this guidance 
498  recommends that the statistical methods used in a meta-analysis be aligned with the analysis 
499  objectives and hypotheses under investigation and be consistent with the study designs and data 
500  collected in the individual trials. The choice of methods should be justified based on the stated 
  13  Contains Nonbinding Recommendations 
Draft  Not for Implementation 
 
501  objectives and documented in the protocol or analysis plan (see Section IV); sensitivity of the 
502  results to departures from assumptions required for correct application of the methods should be 
503  examined as part of the planned analysis strategy. Note that although this guidance generally 
504  recommends the use of subject-level data when available, it is recognized that meta-analyses 
505  conducted based on trial-level data only may be useful in certain settings (see Section VI.B.). 
506  The recommendations in this section apply to trial-level meta-analysis as well.  
507  The material in this section falls into three broad areas: statistical properties of the analysis 
508  methods, heterogeneity, and sensitivity analysis. 
509  B.  Statistical Properties of Risk Estimates and Hypothesis Tests 
510  The statistical approach for a safety meta-analysis should ensure that the estimator and/or 
511  hypothesis test have good statistical properties, namely that the resulting risk estimate is 
512  approximately unbiased and sufficiently precise, the standard error of the estimated risk is 
513  accurate, and the associated confidence intervals have accurate coverage properties. Tests of 
514  hypotheses about the risk should have good operating characteristics, i.e., the Type I and II error 
515  probabilities should be accurate, and the power maximized given the data available.  
516  An important principle involved in estimating risk from a meta-analysis is that the randomized 
517  comparisons of the individual trials should be maintained when analyzing the combined data. In 
518  other words, when comparing drug A to drug B, subjects randomly assigned to drug A in a single 
519  trial are compared to subjects assigned to drug B from the same trial and not to subjects from 
520  other trials. In the statistics literature, this is referred to as stratifying the analysis by trial. 
521  Intuitively, this implies that the overall comparative measure of risk is based on combining the 
522  comparative risk measures from the individual trials using recommended statistical methods. 
523  Stratifying the analysis by trial is preferred to combining data across all subjects in the 
524  component trials by subject group prior to estimating risk, often referred to as simple pooling, as 
525  this ignores the randomized comparisons of the individual trials and can produce misleading 
526  findings.  
527  When one or more of the trials included in the pooling does not employ a one-to-one 
528  randomization scheme, simple pooling of trial data can result in a phenomenon known as 
529  Simpsons paradox (Chuang-Stein and Beltangady 2011). When there are large sample size 
530  disparities among the trials with different randomization allocations, the impact of this 
531  phenomenon can be quite large. The hypothetical example in Table 1 illustrates an extreme 
532  example of Simpsons paradox in which, for each of four trials, the estimated risk of a safety 
533  event is identical for both Drug A and Drug B. With simple pooling, however, the risk for Drug 
534  A appears to be more than twice as high as that for Drug B (12.8 percent vs. 6.2 percent).  
  14  Contains Nonbinding Recommendations 
Draft  Not for Implementation 
 
535  Table 1. An Illustration of Simpsons Paradox from Incorrect Pooling of Data 
  Drug A  Drug B 
Trial  Events  Patients  Risk  Events  Patients  Risk 
1  1  100  1.0%  2  200  1.0% 
2  1  100  1.0%  2  200  1.0% 
3  200  1200  16.7%  50  300  16.7% 
4  2  200  1.0%  2  200  1.0% 
Total  204  1600  12.8%  56  900  6.2% 
536  It is sometimes of interest to combine multiple doses of a drug in one or more of the component 
537  trials to gain statistical power and improve the precision of the risk estimate in a meta-analysis. 
538  The combination of arms should be performed within each trial and the overall analysis should 
539  still be stratified by trial to avoid Simpsons paradox in this setting. 
540  Sparse data resulting from rare safety outcomes pose particular problems in a meta-analysis. The 
541  statistical methods chosen for the analysis should perform well when the number of outcome 
542  events is very small in one or more of the component trials or in one or more treatment groups 
543  within a trial. Some commonly used methods perform well when there are ample events, but not 
544  so well when events are sparse (Bradburn, Deeks et al. 2007). For example, inverse variance 
545  weighting involves estimating risk with a weighted estimate of trial results, where weights are 
546  computed as the inverse of the trial level variance estimates. With sparse data, the estimated 
547  variances may not be well-determined, resulting in an unstable risk estimate. If some of the 
548  component trials have no events, the choice of methods is even more limited. 
549  We do not recommend the use of continuity corrections, one approach for handling zero-event 
550  trials or trials with zero events in one or more treatment groups. Because it may not be apparent 
551  with some software packages if and how continuity corrections are incorporated, caution is 
552  needed to avoid their inadvertent use. Continuity corrections approaches generally involve 
553  adding small quantities to the zero event counts prior to analysis. Although their use allows zero-
554  event trials to be included in a meta-analysis, the results may be biased. Note that the use of ratio 
555  effect measures, such as the risk ratio or hazard ratio, is more challenging in the presence of 
556  zero-event trials than is the use of risk difference measures, such as the Mantel-Haenszel risk 
557  difference (Greenland and Robins 1985). Another approach is to consider Bayesian methods for 
558  meta-analysis (Sutton and Abrams 2001) (Spiegelhalter, Abrams et al. 2004), which can 
559  incorporate information on trials with no events, even when a relative risk measure is used. The 
560  performance of any proposed method for dealing with zero-event trials should be established and 
561  the choice justified for a particular meta-analysis application. 
562  The ability to replicate the results of a meta-analysis with an independent study will increase the 
563  persuasiveness of the findings. One such approach is to analyze one or more newly available 
564  trials to see if the results agree quantitatively and/or qualitatively with the results of an existing 
565  meta-analysis. Alternatively, an existing meta-analysis can be updated as new trials become 
566  available. Although this sequential approach to meta-analysis provides an efficient way to update 
567  risk estimates with new study results, the impact of repeated hypothesis tests about that risk 
568  should be taken into account (Whitehead 1997). 
  15  Contains Nonbinding Recommendations 
Draft  Not for Implementation 
 
569  C.  Heterogeneity 
570  In any meta-analysis, heterogeneity of the drug effect among the component trials is expected 
571  and should be addressed at the design stage. If there is strong reason to believe trials will have 
572  importantly different drug effects based on known factors such as characteristics of the trial 
573  populations, the specific interventions, or other trial design features, then the statistical analysis 
574  should account for this expected variation. This may involve use of a statistical model that allows 
575  for different effects based on known factors. Alternatively, it may be of interest to conduct 
576  separate analyses for distinct groups of trials that vary with respect to one or more important 
577  design factors. For example, if the set of component trials consists of both placebo-controlled 
578  and active-controlled trials, a reasonable approach would be to perform a meta-analysis for each 
579  group of trials separately, taking into account what is known about the active control effect. In 
580  some situations the trials may be so heterogeneous that it is not possible to conduct a meta-
581  analysis.  
582  The most common approach to account for residual heterogeneity in drug effects across trials, 
583  after accounting for expected heterogeneity attributable to known factors, is to incorporate 
584  individual-trial treatment effects in the analysis model as either fixed or random effects. The 
585  meta-analysis literature includes a great deal of discussion about choosing between the two 
586  (Borenstein, Hedges et al. 2010). In the context of a meta-analysis, use of a fixed effects model is 
587  often interpreted as assuming a common effect exists across the trials, in contrast to the use of a 
588  random effects model, where the effects are assumed to vary across trials according to some 
589  probability distribution. This distinction is not usually made in other, similar areas of application, 
590  e.g., in managing centers in a multi-center trial (Senn 2000). In the statistics literature on multi-
591  center trials (see, e.g., ICH E9), use of a fixed effects model is not as restrictive in that the model 
592  can specify either a common effect across centers or different, but non-random, effects for each 
593  center (i.e., by including the center by treatment interaction terms in the model). In the latter 
594  case, interest lies in estimating an average effect across the centers. Similarly, in meta-analysis, it 
595  may be desirable to allow effects to vary by trial with the inclusion of treatment by trial 
596  interaction terms in the fixed effects model, and, in this case, averaging across trials with 
597  appropriate methods provides the drug effect of interest.  
598  Use of a random effects model in a meta-analysis implies an interest in estimating the average 
599  effect for some larger population of trials that are believed to be adequately represented by the 
600  trials in the analysis, and this parallels use of a random effects model in a multi-center trial; i.e., 
601  interest lies in estimating the average effect for a larger population of centers for which the trials 
602  centers provide adequate representation. Arguments may be made against the use of random 
603  effects models in a meta-analysis based on the belief that the trials available for analysis are not a 
604  random sample of some larger population of trials  that is, all relevant trials are included in the 
605  meta-analysis. It has been pointed out, however, that even when there is no interest in making 
606  inference to a larger population of trials, use of a random effects model may produce more 
607  appropriate results, due to the better characterization of the between- and within-trial variance in 
608  the estimation process (Permutt 2003).  
609  Both frequentist and Bayesian methods are available for random-effects meta-analysis, and the 
610  difference between the two lies in the assumptions made about the distributions of the random 
  16  Contains Nonbinding Recommendations 
Draft  Not for Implementation 
 
611  effects, with Bayesian methods offering more flexibility (Muthukumarana and Tiwari 2016). 
612  Bayesian methods also allow multiple sources of variation to be incorporated in the modeling 
613  and estimation process. For example, in a meta-analysis designed to examine a specific risk for a 
614  class of drugs, one may assume there is a component of variation among different drugs within 
615  the class and a separate component among trials involving a single drug. To date, the Agency has 
616  limited experience in evaluating meta-analysis submissions that use Bayesian methods, but 
617  supports the consideration of Bayesian and other methods that achieve the desired properties 
618  discussed in this section. 
619  For safety meta-analysis, the goal is to determine whether a significant risk is causally related to 
620  exposure to the drug, and the power available for that test should be maximized. Use of a fixed 
621  effects model will usually provide optimal power for detection of risk and also reflects a primary 
622  interest in the average effect among only those trials included in the meta-analysis. The parallel 
623  with the establishment of efficacy for drug approval is relevant here. The selective populations 
624  included in premarket efficacy trials may not fully represent the broader patient populations seen 
625  in clinical practice, but are still central in making regulatory decisions. However, for the 
626  quantification of the risk itself, a random effects model might be more appropriate, as the 
627  incorporation of the between-trial variance might better reflect the uncertainty associated with 
628  the risk estimate. Under all scenarios, the statistical inference should properly reflect the 
629  assumptions made for the fixed or random effects model used; in particular, the variance of the 
630  estimator should properly reflect whether the trial effects are constant, non-constant, or random. 
631  D.  Sensitivity Analysis 
632  Sensitivity analyses play an important role in examining the impact of meta-analysis design 
633  decisions on the findings as well as the strength of evidence provided by the meta-analysis. The 
634  goal of any sensitivity analysis should not be to search for additional findings, but to support and 
635  understand the primary findings of the meta-analysis. Trial inclusion criteria, outcome definition, 
636  time period within which the safety outcome of interest is to be measured, and analysis method 
637  are examples of design characteristics that may be varied as part of a sensitivity analyses.  
638  For example, if a meta-analysis protocol and statistical analysis plan called for including only 
639  those safety events that occurred during exposure periods in the risk estimate, then a sensitivity 
640  analysis that included all reported events, regardless of whether subjects were on or off drug, 
641  could provide important information about the observed risk estimate. A decreased event rate in 
642  off-treatment periods could, in this example, support causality (depending on the hypothesized 
643  mechanism). Similarly, a meta-analysis that included one very large study contributing a large 
644  proportion of subjects and events could raise a concern that it was overly influencing the meta-
645  analytic results. A sensitivity analysis that excluded that study would have reduced numbers of 
646  subjects and events and lower power to yield a significant finding, but a risk estimate that was 
647  consistent with the original estimate would add to the weight of evidence of the finding. 
648  It is often of interest to examine the consistency of findings from a meta-analysis across 
649  subgroups based either on trial-level or subject-level characteristics. Trial-level factors that might 
650  be of interest include the comparator treatment, dose and duration of treatment, background 
651  therapy, and subject inclusion criteria. Subject-level factors may vary within trials, and subject-
  17  Contains Nonbinding Recommendations 
Draft  Not for Implementation 
 
652  level data are required to provide estimates for each subgroup. In the antidepressant meta-
653  analysis example of section VII, age was an important factor of specific interest, because the 
654  meta-analysis was motivated by an earlier meta-analysis of pediatric subjects. The number of 
655  subgroups to be examined should be kept to a minimum to avoid the consequences of multiple 
656  testing. Given the multiplicity issues, subgroup findings are seldom viewed as definitive in safety 
657  meta-analyses. 
658  VI.  STRENGTH OF EVIDENCE AND REGULATORY DECISIONS 
659  A.  Critical Factors in Determining the Strength of Evidence 
660  Regulatory decisions related to drug safety are generally taken after considering the totality of 
661  available evidence, which may include meta-analytic findings, as well as other factors such as 
662  risk-benefit considerations, availability of alternative treatments, biological and clinical 
663  plausibility of the drug-risk relationship, and available regulatory options. The strength of 
664  evidence provided by the meta-analysis may influence a regulatory decision by FDA. The factors 
665  discussed above that FDA generally considers in determining the strength of evidence with 
666  respect to a safety-related regulatory decision can be summarized as follows:  
667    Quality and appropriateness of the individual trials for the meta-analysis objectives 
668    Quality and completeness of safety outcome ascertainment 
669    Appropriateness of studied populations and exposure and follow-up periods 
670    Protocol adherence in the individual trials (e.g., compliance with investigational 
671  treatment, loss to follow-up, etc.) 
672    Availability and quality of subject-level data 
673    Prespecification and adequacy of documentation 
674    Prespecification and documentation of objectives, available knowledge, trial 
675  inclusion criteria, and choice of comparators, outcomes, statistical methods, and 
676  subgroups 
677    Documentation that trial outcomes were not used as part of the trial selection 
678  criteria  
679    Documentation of meta-analysis results including summaries of trials, subjects, 
680  outcomes, effect estimates, measures of uncertainty, and sensitivity analyses 
681    Appropriateness of statistical methods 
682    Approach used for combining trials  
683    Methods to handle sparse data or rare events  
684    Methods to address heterogeneity  
685    Sensitivity analyses 
686    Validity of uncertainty estimates (e.g., confidence intervals or credible intervals) 
687  Although not previously discussed, the magnitude of the estimated risk and associated measures 
688  of uncertainty are also important. A large estimated risk will generally be more convincing than a 
689  small to moderate one, because it will provide more assurance that an effect is real even in the 
690  presence of potential biases. Similarly, smaller p-values or narrower confidence intervals, both 
691  measures of uncertainty, provide additional assurance on the findings of the meta-analysis. For 
  18  Contains Nonbinding Recommendations 
Draft  Not for Implementation 
 
692  safety meta-analyses, however, there is potential for bias from both known (e.g., selection of 
693  trials based on their outcomes) and unknown (biases that cannot be identified from the data used 
694  to conduct the meta-analysis) sources. Given this difficulty, standard measures of uncertainty, 
695  such as significance levels, should be interpreted with caution. 
696  One approach to account for the many potential sources of bias and error in a meta-analysis is to 
697  replace the commonly used test size or alpha level for hypothesis testing,  = 0.05, with an 
698  arbitrarily lower value (e.g., 0.01 or 0.001) in order for the results to be considered convincing. 
699  The choice of a lower value would reflect the recommendation to compensate for known and 
700  unknown sources of potential bias as well as to minimize the impact of multiplicity resulting 
701  from multiple comparisons. Such an approach would be important if the meta-analysis is the 
702  only basis for decision-making, as it will explicitly reflect the higher degree of uncertainty that 
703  exists for meta-analysis results. At the same time, there are often other sources of safety 
704  information so that the significance level for the meta-analysis is only one of many factors taken 
705  into consideration. Consequently, no single test size (alpha level) and no single confidence level 
706  can be recommended for deciding the level of statistical significance for results from a safety 
707  meta-analysis to be relied upon. The potential for harm may be so serious that marginally 
708  significant findings could prompt regulatory consideration. In this setting, however, the sources 
709  of bias and error related to the meta-analysis should be identified and accounted for wherever 
710  possible. 
711  In addition to the magnitude of the observed effect and the level of uncertainty, the robustness of 
712  the risk estimate to appropriate sensitivity analyses can also support the strength of the meta-
713  analysis findings. The importance of sensitivity analyses is described in section V, as their results 
714  play an important role in determining the strength of evidence. Risk estimates that are reasonably 
715  robust to the inclusion or exclusion of particular studies, or to changes in the statistical analysis 
716  methods used and assumptions required for appropriate use of those methods, will carry a greater 
717  weight of evidence than estimates that vary widely with such changes.  
718  Similarly, risk estimates that are consistent across trials will also carry greater weight. In section 
719  IV, the use of forest plots or other graphical display of the study-specific risk estimates and their 
720  confidence intervals is advocated as a descriptive assessment of study-to-study heterogeneity. 
721  Absent any known cross-study differences, a high degree of similarity among study-specific 
722  results will strengthen the evidence provided by the meta-analytic summary risk estimate. 
723  Conversely, a large amount of variability among studies would make a marginal risk estimate (in 
724  terms of lack of statistical significance or small in magnitude) less persuasive.  
725  B.  Hierarchy of Evidence for Decision-Making 
726  The factors described above for evaluating the strength of meta-analytic findings can be used to 
727  define a hierarchy of evidence against which meta-analyses conducted or reviewed for regulatory 
728  purposes should be evaluated.  
729    A top tier meta-analysis is one that is prospectively planned prior to the conduct of the 
730  trials to be included, and where the component trials are designed with the meta-analysis 
731  objectives in mind. The trials have well-ascertained outcomes and exposure periods, and 
  19  Contains Nonbinding Recommendations 
Draft  Not for Implementation 
 
732  subject-level data are available for analysis. This level represents a gold standard not 
733  often realized in practice but useful as a benchmark in evaluating the quality of a meta-
734  analysis.  
735    The next level down is a prospectively planned meta-analysis based on existing trials that 
736  were designed for other purposes but for which the quality of the data and the 
737  ascertainment of outcomes and exposure are adequate to support the planned analysis. 
738  Further, all meta-analytic study plans and trial inclusion decisions were made without 
739  knowledge of the study outcomes for the safety events of interest.  
740    The lowest tier, representing the least useful evidence for regulatory decision-making, 
741  corresponds to meta-analyses for which prospective planning did not occur, or is in 
742  doubt, study outcomes and trial inclusion decisions were made with outcome data in 
743  hand, and one or more of the important quality factors is in question, e.g., lack of rigor in 
744  outcome ascertainment, lack of subject-level data for use in determining exposure, use of 
745  inappropriate statistical methods such as simple pooling of trial data, or other issues.  
746  Between the bottom and top tiers lies a broad range of meta-analyses for which an evaluation of 
747  the strength of evidence provided should include careful consideration of the important factors 
748  delineated in the previous subsections.  
749  The level of evidence from a meta-analysis that is based solely on study level summary data, 
750  either prospective or retrospective, is generally considered to be lower than one for which 
751  subject-level data are available, as the party conducting the meta-analysis has little ability to 
752  judge the quality or completeness of the data or the appropriateness of the analysis methods used. 
753  On the other hand, if the outcome is relatively judgment-free and well-ascertained (e.g. mortality 
754  or perhaps stroke rate), these meta-analyses may still play a role in regulatory decisions. A study-
755  level meta-analysis could be used as a first step to determine whether a more resource intensive 
756  subject-level meta-analysis is needed, perhaps based on the same studies. A hybrid would be a 
757  combination of studies for which subject level data are available for a subset; the mix would 
758  determine where in the hierarchy such a meta-analysis should be placed. The recommendations 
759  laid out in this guidance for producing high-quality meta-analyses apply regardless of the level 
760  (subject- or trial-level) of analysis involved.  
761  There are two categories of meta-analyses considered particularly problematic for the regulatory 
762  framework and worth mentioning here. The first includes meta-analyses reported in the literature 
763  with no prior publication or credible record of a protocol to guide the selection of studies or 
764  prespecification of study objectives and analysis strategy. This type of meta-analysis is likely 
765  insufficient for regulatory purposes, for the reasons outlined in section IV. Even if the studies 
766  included in the meta-analysis represent a reasonable subset of those available (as opposed to only 
767  published studies), without documentation of a prespecified plan for deciding which to include 
768  and identifying outcomes of interest, it is usually not possible to determine what was known at 
769  the time the studies were selected, what analysis methods were chosen, or how many different 
770  analyses were conducted, in what sequence, and for which study populations or subgroups. 
771  Evidence from such an analysis would generally be considered too weak to support regulatory 
772  decision-making without further confirmation of the findings.  
  20  Contains Nonbinding Recommendations 
Draft  Not for Implementation 
 
773  The second category includes meta-analyses that are based solely on safety results appearing in 
774  the literature. Limiting the meta-analysis to studies whose results appeared in publications about 
775  the exposure-risk relationship can introduce publication bias. This well-known phenomenon 
776  arises from the concern that studies failing to find a significant association between drug use and 
777  risk are not published at the same frequency as studies that show an association, and even among 
778  those published, bias may occur due to a failure to include certain safety outcomes in the 
779  publication and failure to include studies that did not show the outcome sought (Chalmers, Levin 
780  et al. 1987). Further, the information contained in the publication for each study may be lacking 
781  in detail, and without access to subject-level data, it may not be possible to rule out bias or severe 
782  heterogeneity in the results. For example, even if the results for the safety outcome of interest are 
783  reported for each trial, the details of how events were defined, measured, or adjudicated in the 
784  trials may not be clear, and it may not be possible to determine if the safety events of interest 
785  occurred on or off drug. Subject-level data are typically not available in publications of 
786  completed trials, limiting the ability to resolve these issues. 
787  In summary, a number of important factors should be involved in determining the credibility of 
788  evidence from a particular meta-analysis. These factors range from the knowledge about and 
789  documentation of eligible studies, both published and unpublished; the quality and relevance of 
790  the studies selected as well as the process and timing of selection; and the validity of the 
791  statistical analysis that supports the inferential conclusions and the strength of the findings, 
792  evaluated against sources of potential or real bias. Whether or not the findings of a meta-analysis 
793  influence regulatory decision-making will generally depend, in part, on the strength of evidence 
794  provided by the findings, as determined by a careful evaluation of the important factors 
795  described in this guidance.  
796  VII.  EXAMPLES 
797  A.  Example 1: Antidepressant Use and Suicidal Events in Adults 
798  This example illustrates the use of a meta-analysis to evaluate risks associated with a class of 
799  drugs and represents a prospectively planned meta-analysis of retrospective data, falling into the 
800  middle tier of the hierarchy of evidence discussed in section VI.B. The research hypotheses, 
801  study inclusion criteria, outcome measures, and statistical analysis plan were all specified prior 
802  to the conduct of the meta-analysis. Outcomes were uniformly adjudicated across studies, 
803  pooling of study data was accomplished with stratification, and subject-level data were available 
804  to explore subgroups as well as trends in risks across time. The interpretation of the findings, 
805  which resulted in a boxed warning for labeling of drugs in the class, reflects appropriate 
806  consideration given to the level of statistical significance, and to the consistency of findings.  
807  In 2004, FDA completed a meta-analysis of studies of pediatric patients that showed an 
808  association of antidepressant drugs and suicidal behavior and ideation (Hammad, Laughren et al. 
809  (2006). Unsolicited information provided by drug sponsors to FDA and published articles 
810  motivated this meta-analysis. Based on the meta-analysis findings and deliberations from a 
  21  Contains Nonbinding Recommendations 
Draft  Not for Implementation 
 
811  meeting in 2004 of FDA Advisory Committees (69 FR 471574) and further consideration by 
812  FDA, a boxed warning was added to the labeling of all antidepressants concerning use in 
813  pediatric patients. Other FDA Advisory Committees later asked FDA to explore the association 
814  in adult patients (71 FR 665455.) 
815  For this purpose, FDA planned and conducted a meta-analysis of randomized clinical trials of 
816  antidepressants. FDA is uniquely positioned to address this research question, because of the 
817  Agencys knowledge of marketed products in the drug class in question. The meta-analysis had 
818  several important features that supported its quality and utility for regulatory actions: (1) 
819  hypotheses generated from previous and independent evidence provided the meta-analysis 
820  objectives; (2) the meta-analysis was based on well-defined inclusion criteria and a complete set 
821  of the trials that met the inclusion criteria; (3) the meta-analysis employed rigorous and 
822  consistent outcome definitions across trials and patients; (4) the meta-analysis was based on a 
823  prespecified plan; and subject-level data was available. 
824  FDA requested from all manufacturers of antidepressants all available subject-level data from 
825  randomized placebo-controlled trials of antidepressants. Basing the meta-analysis on data 
826  available to sponsors, while not inclusive of all potentially available data, has some important 
827  advantages. Because of regulatory requirements, trials from drug manufacturers typically contain 
828  detailed subject-level data including medical history, baseline characteristics, subject 
829  dispositions, patient outcomes, and adverse events. Focusing on the relatively small group of 
830  drug manufacturers (nine) allowed for the timely acquisition of the large amounts of pertinent 
831  data. Overall, the FDA obtained subject-level data considered usable for 372 trials.  
832  The meta-analysis was prospectively planned but was based on previously collected data. 
833  Because the specific outcomes of interest were not systematically collected and adjudicated 
834  during the conduct of the trials, FDA provided specific instructions to the individual sponsors to 
835  conduct a retrospective identification and adjudication of potential suicidal behavior and ideation 
836  events from the subject-level data. The outcome definition required that the suicidal behavior and 
837  ideation events occurred on randomized treatment or within one day of stopping the randomized 
838  treatment. Based on adverse event reporting, potential events were identified with a specified 
839  algorithm. Based on blinded narratives of the events, qualified personnel classified the events 
840  into specific outcomes including: completed suicide, attempted suicide, preparatory actions 
841  toward imminent suicidal behaviors, and suicidal ideation based on the Columbia Classification 
842  Algorithm for Suicide Assessment (Posner, Oquendo et al. 2007). The overall process resulted in 
843  outcome measures that were consistently and rigorously defined across trials and subjects.  
844  The meta-analysis employed a prespecified plan that included the trial inclusion criteria, 
845  hypotheses, outcome definitions, analysis methods, sensitivity analyses, and subgroups. The 
846  primary analysis method incorporated stratification by trial and accounted for the sparse nature 
847  of the outcome events by using exact statistical methods for hypothesis testing. Sensitivity 
                                                 
4 Briefing package: 
http://web.archive.org/web/20040911055410/https://www.fda.gov/ohrms/dockets/ac/04/briefing/2004-4065b1.htm 
5 Briefing package: https://wayback.archive-
it.org/7993/20170405070114/https://www.fda.gov/ohrms/dockets/ac/06/briefing/2006-4272b1-01-FDA.pdf 
  22  Contains Nonbinding Recommendations 
Draft  Not for Implementation 
 
848  analyses were planned to examine the possibility and consequences of the following: differential 
849  exposure time between the randomized drug arms; heterogeneity of the effect measure across the 
850  trials; and trials with no events. The subject-level data allowed for the examination of important 
851  subgroups, including subject age, and for the examination of changing risk over time.  
852  FDA presented the meta-analysis findings to a 2006 meeting of FDA Advisory Committees 
853  (FDA 2006) and sought advice on the interpretation and possible regulatory actions based on the 
854  findings of the meta-analysis. The meta-analysis found that the overall association of 
855  antidepressant drugs and suicidal behavior and ideation was not statistically significant in adult 
856  subjects, in contrast to the FDA meta-analysis of pediatric subjects. However, the association 
857  was nearly statistically significant for young adults, and a clear pattern emerged with respect to 
858  patient age (see Figure 1). The result from the pediatric meta-analysis supported this trend.  
859  Based on the totality of the evidence, including results from the meta-analysis, FDA requested 
860  that manufacturers update the boxed warning on all antidepressants to include the risk of suicidal 
861  behavior and ideation associated with antidepressants for young adult patients in addition to 
862  pediatric patients. The warning states that the effect was not seen in adults over the age of 24, 
863  and for adults aged 65 and older, there was a reduction in risk. It should be appreciated that the 
864  clear pattern observed with respect to age and not just statistical significance led to the warning.  
Suicidal Behavior and Ideation 
 Psychiatric Indications
Age Class OR (95% CI)   [Sample Sizes]*
Pediatric Data 2.22   (1.40, 3.60)   [61/2414 26/2073]
18 to 24 1.55   (0.91, 2.70)   [47/3810 21/2604]
25 to 30 1.00   (0.60, 1.69)   [41/5558 27/3772]
31 to 64 0.77   (0.60, 1.00)   [147/27086 124/18354]
65 and Up 0.39   (0.18, 0.78)   [12/3227 24/2397]
Adult Overall 0.84   (0.69, 1.02)   [248/39729 196/27164]
0.1 0.3 1 3.2 10
Odds Ratio
865  *                 
866  Figure 1: FDA Meta-Analysis of Antidepressants and Suicidal Behavior and Ideation. 
867  Note: Pediatric results from previous FDA meta-analysis of pediatric patients (Hammad, 
868  Laughren et al. 2006). 
  23  Contains Nonbinding Recommendations 
Draft  Not for Implementation 
 
869  B.  Example 2: Tiotropium and Cardiovascular Events 
870  This example illustrates FDAs consideration of the cardiovascular safety of the drug tiotropium 
871  and shows how the relative strengths of a well-designed, large, long-term trial and a meta-
872  analysis based on published literature as well as other trial-level information (Michele, Pinheiro 
873  et al. 2010) were factored into a regulatory decision. Tiotropium bromide inhalation powder is a 
874  long-acting anticholinergic approved for use in treating bronchospasm associated with chronic 
875  obstructive pulmonary disease (COPD) and for reducing COPD exacerbations. The potential 
876  association of the drug with cardiovascular events was first reported to FDA based on an analysis 
877  of adverse events from 29 placebo-controlled trials conducted by the drug manufacturer. In 
878  particular, the simple pooled analysis of these studies showed that the drug had an excess number 
879  of strokes associated with its use. The pooled analysis was intended to identify potential signals 
880  for further evaluation and examine a range of adverse events. As is typical for such analyses, 
881  findings from the pooled analysis were not adjusted for multiplicity associated with examining 
882  multiple endpoints.  
883  Because of the severity of the clinical outcomes, FDA issued a communication informing the 
884  public of the potential safety signal and FDAs efforts to investigate the findings. The 
885  communication noted that data from a large, four-year study called UPLIFT (Tashkin, Celli et al. 
886  2008) would soon be available and would provide additional long-term safety data on the drug. 
887  Following the FDA communication, an article appeared on a meta-analysis of 17 randomized 
888  trials reporting a statistically significant increase in a cardiovascular composite outcome 
889  (cardiovascular death, myocardial infarction, and stroke) associated with inhaled anticholinergics 
890  (consisting of tiotropium and ipratropium) (Singh, Loke et al. 2008). At the same time, the 
891  results from UPLIFT had become available, and the initial review did not support a finding that 
892  tiotropium was associated with an increased risk of stroke, heart attack, or cardiovascular death.  
893  A comparison between the pooled analysis of 29 trials conducted by the manufacturer and the 
894  UPLIFT trial highlights some important differences between the two sources of safety 
895  information. Although the pooled analysis contained more than twice as many subjects as 
896  UPLIFT (13,544 versus 5,992), the study duration of UPLIFT (4 years) was substantially longer 
897  than the durations of the trials in the pooled analysis (1  12 months). Consequently, UPLIFT 
898  provided more than twice as many person-years of follow-up (17,721 person-years) than the 
899  pooled analysis (7,636 person-years). Additionally, UPLIFT prospectively collected data on 
900  death and adjudicated cause of death for all subjects, including subjects who withdrew from the 
901  study.  
902  In 2009, FDA convened an advisory committee meeting to discuss the results of UPLIFT and the 
903  published meta-analysis. The advisory committee concluded (11 votes to 1) that the results of 
904  UPLIFT adequately addressed the cardiovascular safety concerns that had been raised for 
905  tiotropium based on the initial pooling of 29 trials by the manufacturer and the published meta-
906  analysis. The committee noted methodological concerns with the published meta-analysis, 
907  including lack of accounting for differential withdrawal rates between treatment groups and the 
908  potential for publication bias due to including only studies reporting an increase in 
909  cardiovascular events with use of tiotropium in the meta-analysis. The committee also noted 
910  concerns about the heterogeneity of trial designs in the review, including differences in study 
  24  Contains Nonbinding Recommendations 
Draft  Not for Implementation 
 
911  drug, comparator drug, trial duration, and studied population. Based on the strength of the 
912  UPLIFT study findings and the methodological concerns of the published meta-analysis, FDA 
913  concluded that the available data did not support an association between the drug and adverse 
914  cardiovascular events.   
915  The tiotropium example shows that a meta-analysis based on trial-level summaries may not 
916  agree with a large trial that is well designed specifically with a safety outcome as a primary 
917  objective. However, the Agencys position on the safety and effectiveness of a drug is based on 
918  the best information available at the time. In the tiotropium example, FDA issued a series of 
919  public communications to apprise the public of the latest safety information available and FDAs 
920  intended course of action. The example shows FDAs intention to carefully evaluate potential 
921  safety risks while balancing the need to not unnecessarily discourage or restrict the use of safe 
922  and effective drugs. The example also shows FDAs intention to act in a transparent manner, to 
923  the extent possible, based on the available data to ensure the safe use of drugs. 
924  VIII.  REFERENCES 
925  Bender, R., C. Bunce, M. Clarke, S. Gates, S. Lange, N. L. Pace and K. Thorlund (2008). 
926  "Attention should be given to multiplicity issues in systematic reviews." J Clin Epidemiol 61(9): 
927  857-865. 
928  Borenstein, M., L. V. Hedges, J. P. Higgins and H. R. Rothstein (2010). "A basic introduction to 
929  fixed-effect and random-effects models for meta-analysis." Res Synth Methods 1(2): 97-111. 
930  Bradburn, M. J., J. J. Deeks, J. A. Berlin and A. Russell Localio (2007). "Much ado about 
931  nothing: a comparison of the performance of meta-analytical methods with rare events." Stat 
932  Med 26(1): 53-77. 
933  Chalmers, T. C., H. Levin, H. S. Sacks, D. Reitman, J. Berrier and R. Nagalingam (1987). 
934  "Meta-analysis of clinical trials as a scientific discipline. I: Control of bias and comparison with 
935  large co-operative trials." Stat Med 6(3): 315-328. 
936  Chien, P. F., K. S. Khan and D. Siassakos (2012). "Registration of systematic reviews: 
937  PROSPERO." BJOG 119(8): 903-905. 
938  Chuang-Stein, C. and M. Beltangady (2011). "Reporting cumulative proportion of subjects with 
939  an adverse event based on data from multiple studies." Pharm Stat 10(1): 3-7. 
940  Greenland, S. and J. M. Robins (1985). "Estimation of a common effect parameter from sparse 
941  follow-up data." Biometrics 41(1): 55-68. 
942  Hammad, T. A., T. Laughren and J. Racoosin (2006). "Suicidality in pediatric patients treated 
943  with antidepressant drugs." Arch Gen Psychiatry 63(3): 332-339. 
944  Higgins, J. P. T., Green, S., (editors). (2011). Cochrane Handbook for Systematic Reviews of 
945  Interventions, Version 5.1.0 [updated March 2011]. The Cochrane Collaboration, 2011. 
946  Available from www.handbook.cochrane.org. 
  25  Contains Nonbinding Recommendations 
Draft  Not for Implementation 
 
947  Michele, T. M., S. Pinheiro and S. Iyasu (2010). "The safety of tiotropium--the FDA's 
948  conclusions." N Engl J Med 363(12): 1097-1099. 
949  Moher, D., A. Liberati, J. Tetzlaff, D. G. Altman and P. Group (2009). "Preferred reporting items 
950  for systematic reviews and meta-analyses: the PRISMA statement." PLoS Med 6(7): e1000097. 
951  Muthukumarana, S. and R. C. Tiwari (2016). "Meta-analysis using Dirichlet process." Stat 
952  Methods Med Res 25(1): 352-365. 
953  Ohlssen, D., K. L. Price, H. A. Xia, H. Hong, J. Kerman, H. Fu, G. Quartey, C. R. Heilmann, H. 
954  Ma and B. P. Carlin (2014). "Guidance on the implementation and reporting of a drug safety 
955  Bayesian network meta-analysis." Pharm Stat 13(1): 55-70. 
956  Permutt, T. (2003). "Probability models and computational models for ANOVA in multicenter 
957  clinical trials." J Biopharm Stat 13(3): 495-505. 
958  Posner, K., M. A. Oquendo, M. Gould, B. Stanley and M. Davies (2007). "Columbia 
959  Classification Algorithm of Suicide Assessment (C-CASA): classification of suicidal events in 
960  the FDA's pediatric suicidal risk analysis of antidepressants." Am J Psychiatry 164(7): 1035-
961  1043. 
962  Senn, S. (2000). "The many modes of meta." Drug Information Journal 34(2): 535-549. 
963  Singh, S., Y. K. Loke and C. D. Furberg (2008). "Inhaled anticholinergics and risk of major 
964  adverse cardiovascular events in patients with chronic obstructive pulmonary disease: a 
965  systematic review and meta-analysis." JAMA 300(12): 1439-1450. 
966  Spiegelhalter, D. J., K. R. Abrams and J. P. Myles (2004). Bayesian approaches to clinical trials 
967  and health-care evaluation. Chichester, Wiley. 
968  Sutton, A. J. and K. R. Abrams (2001). "Bayesian methods in meta-analysis and evidence 
969  synthesis." Stat Methods Med Res 10(4): 277-303. 
970  Tashkin, D. P., B. Celli, S. Senn, D. Burkhart, S. Kesten, S. Menjoge, M. Decramer and U. S. 
971  Investigators (2008). "A 4-year trial of tiotropium in chronic obstructive pulmonary disease." N 
972  Engl J Med 359(15): 1543-1554. 
973  Whitehead, A. (1997). "A prospectively planned cumulative meta-analysis applied to a series of 
974  concurrent clinical trials." Stat Med 16(24): 2901-2913. 
  26