# Presentation: 698641

## Inference From Multiple Studies

- Charles Anello, Sc.D.
- Deputy Director, Office of Biostatistics
- Center For  Drug Evaluation and Research,FDA
- anello@cder.fda.gov
- Fourth Annual FDA\Industry workshop “The information Revolution -who is mining the store?” sponsored by the American Statistical Association, Biopharmaceutical Section, Jan. 17-18, 2002 Bethesda, MD

## Disclaimer

## Acknowledgement

- I would like to thank Meiyu Shen,Ph.D. for her assistance with programming and graphic displays used in this presentation and H.M. Jim Hung, Ph.D for his help.

## Topics

- Regulatory  Setting
- Inference from multiple studies
- Canner’s Data(Aspirin and MI)
- Some approaches to looking at Multiple Studies
-                       Meta-analysis( fixed and random effect models)
-                        Likelihood Ratio
-                        Radial Plot
-                        Phyp Plots based on Alternate Distribution Approach
-                        Hierarchical  Bayes Linear Model
- Heterogeneity
- Conclusions

## Regulatory Setting

- Inference from multiple studies will be considered in the context of Drug Development and a New Drug Application

## Regulatory Setting

- Sponsor’s initiate a series of non-clinical and clinical studies: IND phase.
- Sponsor submits the results of these studies,the supporting data and the label claims: the NDA phase.
- CDER’s technical staff (Med, Chem, Phar Tox and Stat) evaluate the submission from the perspective of “substantial evidence” Review Phase
- Postmarketing Surveillance (not considered)

## Regulatory Setting

- 1962 Kefauver-Harris Amendments to the FD&C Act : Efficacy requirement
- 1970 DESI  Summary Judgements: Defining adequate and well controlled studies
- 1986 NDA Rewrite :Documentation,statistical evidence, integrated summaries of safety and efficacy
- 1997 Food and Drug Moderization Act(FDAMA):modifies the concept of “substantial evidence”in some situations

## Regulatory Setting

- Historically, the focus is on the individual studies
- Although, 1970 staff manuals called for an “overall evaluation”
- 1970 FR 314.1 (356H) allowed for “expanded summary”  a) table of all investigators, b) age and gender, c) table of dosages d) summary for each claim , e) safety tabulations and e) overall conclusions

## Regulatory Setting

- The number of studies and the size of the data base depends on the sponsors goals and objectives for the product.
- There may be one or more phase III studies usually designed to stand on their own and answer questions about the safety and efficacy of a new drug.
- These studies may be addressing the same question or different questions. May involve the same or different populations, duration or endpoints.

## Inference from Multiple Studies 

- Usually a narrative reviews or overviews, not Meta-analyses
- Some pooling of studies occurs and provides some descriptive statistics which may help characterize the findings for special populations (age, gender or ethnicity)
- Pooling of selected studies within an NDA has been referred to as a “Meta-analysis” but unless preplanned, it is ad hoc pooling.
- The p-values  when there is ad hoc pooling do not have the usual  meaning, associated with hypothesis testing.

## Inference from Multiple Studies

- An recent approach that has been proposed by
- Huster WJ and Enas GG,”A Framework Establishing clear Decision criteria for Assessment of Drug Efficacy, Statistics in Medicine 17,1829-1838(1998)
- Includes the following ideas:
- Distinguishes between the exploratory and confirmatory phase
- Proposes ‘one programme, one p-value framework’
- Proposes likelihood approach for exploratory phase
- P-value for one confirmatory trial
- This proposal is beyond the scope of this talk.

## Inference from Multiple Studies

- This talk will focus on the situation where one has multiple phase III studies addressing the same question, in the same populations and endpoints.
- In these situations one deals with an “integrated summary” effecacy.
- Any integrated summary should be based on all the adequate and well controlled studies that address the same question regardless of their particular results

## Inference from Multiple Studies

- I will assume that for many regulatory situations there will be more than one confirmatory trial.
- That is there will be two or more studies addressing the same labeling question.
- I will also assume that classical hypothesis testing will remain the basic approach to regulatory decision making but other methods would provide support or help clarify conflicting results.

## A non-NDA example of Inference from Multiple Studies

- Canner, PL,”An overview of six clinical trials of Aspirin in Coronary Heart Disease”,Statistics in Medicine, vol.6,225-263(1987)

## Canner, PL,”An overview of six clinical trials of Aspirin in Coronary Heart Disease “:  Table 1

- 		Aspirin			Placebo
- Study	No. Pt         No. Dx         %Dx        No. Pt       No.Dx        %Dx                Z                 OR
- UK-1	615	    49	   8.0	 624	   67  	 10.7	-1.67	0.72
- CDPA	758	    44	   5.8	 771	   64 	  8.3	-1.90	0.68
- GAMS	317	    27	   8.5	 309	   32	  10.4	-0.79	0.81
- UK-2	832	  102	 12.3	 850	 126	  14.8	-1.53	0.80
- PARIS	810	    85	  10.5	 406	    52	  12.8   	-1.20	0.80
- sub total	3332	  307	    9.2	2960	  341	  11.5	--3.19	0.76
- AMIS	2267	  246	  10.9	2257	  219	  9.7	1.27	1.13
- total	5599	  553	    9.9	5217	 560	  10.7	-1.56	-1.59	0.90
- Statistics in Medicine, Vol. 6 255-263(1987)

## Canner, PL,”An overview of six clinical trials of Aspirin in Coronary Heart Disease”: Results

- First 5 trials show remarkable agreement OR (0.68-0.81)
- Test of overall association between reduction in total mortality and aspirin use was at observed p<.0014
- Test for homogeneity p=.96
- Adding AMIS the association test yields p=.11 and the homogeneity test p=.08
- Statistics in Medicine, vol.6,225-263(1987)

## A few tools for looking at Multiple Study Results(homogeneity)

- Meta-analysis (fixed, random,mixed models)
- Likelihood
- Radial Plot
- Behavior of P-value under Alternate Hypothesis
-  Hierarchical Bayes Linear Model

## Meta-analysis

- The formal evaluation of the quantitative evidence from two or more trials bearing on the same question. This most commonly involves the statistical  combination of summary statistics from various trials, but the term is sometimes used to refer to the combination of the raw data.
- 1998 ICH: 9 Statistical Principles defines Meta-analysis as:

## Fixed Effect Model

- Cochran’s Method
- Peto’s Method
- Cochran ,WG “The combination of estimates from different experiments’, Biometrics,10,101-129,1954

## Random Effect Model

- DerSimonian R. and Laird,N “Meta-Analysis in clinical trials”, Controlled Clinical Trials,7,177-188,1986

## Canner's Data (Aspirin and MI)

**Lower**

**Upper**

- AMIS

- .941

- 1.329

- CDP

- .483

- 1.013

- GASP

- .527

- 1.279

- MRC-1

- .522

- 1.054

- MRC-2

- .649

- 1.054

- PARIS

- .593

- 1.133

**Fixed Combined (6)**

**.816**

**1.020**

**Random Combined (6)**

**.717**

**1.023**

**0.1**

**0.2**

**0.5**

**1**

**2**

**5**

**10**

**For Asprin**

**For Placebo**

**Canner's Data (Aspirin and MI)**

- Comprehensive Meta Analysis

## Funnel Plot of Precision by Effect Size

**Precision (1/Std Err) of Log RR**

**Log Risk Ratio**

**Funnel Plot of Precision by Effect Size**

## Likelihood Approach

- The likelihood ratio(LR)  is used to define the evidence
- LR is the ratio of probabilities of the observed data under two simple hypotheses H1 and H0.
- Goodman SN, “Meta-analysis and Evidence”, Controlled Clinincial Trials 10:188-204(1989)

## Likelihood Approach

- Goodman argues:
- Under the null hypothesis, a small effect in a large trail can be as rare as a big effect in a small trial, producing the same z score and p value
- The absolute effect size, an essential part of the evidence is not reflected in the p or z values
- Traditional meta-analysis must pool effect estimates and test statistics separately
- The likelihood ratio integrates the effect size and measures of statistical strength
- Goodman SN, “Meta-analysis and Evidence”, Controlled Clinincial Trials 10:188-204(1989)

## Likelihood Approach

- Interpretation of Support: (informal guide H1 vs H0)
- Units of Support   Interpretation     LR    Z
-            0                   No Evidence     1      0
-            1                   Weak                2.7   1.4
-            2                    Moderate         7.4   2.0
-            3                    Strong              20    2.5
-           4             very strong       55    2.8
- Goodman SN, “Meta-analysis and Evidence”, Controlled Clinincial Trials 10:188-204(1989)

## Likelihood Ratio(Canner’s Data)

## Radial Plot

-  Bivariate scatter plot of y versus x
- 1 every estimate has unit standard error in the y - direction the y scale is drawn as a +/- 2 error bar to indicate this explicitly
- 2  the numerical  value of an odds ration can be read off by extrapolation a line from (0,0) through (x,y) to the circular scale drawn, inpartricular , the horizontal line corresponds to an odds ratio of 1
- Galbraith RF,”A note on graphical presentation of Estimated Odds Ratios from several Clinical Trials” Statistics in Medicine, Vol.7,889-894(1988)

## Radial Plot

- An approximate 95% confidence limits can be read of by extrapolatin lines form (0,0) through (x,y+2), (x,y-2)
- Points with large              fall near the origin, that is, points with small           fall away from the origin
- Galbraith RF,”A note on graphical presentation of Estimated Odds Ratios from several Clinical Trials” Statistics in Medicine, Vol.7,889-894(1988)

## Radial plot(Canner’s Data)

## Phyp plot

- Hung et.al argue:
- P-value based on a continuous test statistic has a uniform distribution over the interval[0,1], regardless od the sample size.
- P-value under the alternative hypothesis is a function of both sample size and the true value of the tested parameter
- Hung HMJ, Oneill RT, et.al, “The Behavior of the P-value when the alternative hypothesis is true”, Biometrics 53,11-22,March 1997

## Phyp-plot(Canner’s Data)

## Hierarchical Bayes Linear Model

- DuMouchel and Normand propose a unified Hierarchical Bayes Liner Model with three sources of varitation:
- Let  represent the common parameter pf interest, so that, for moderately large study sample sizes,
- Yi normal(, si2).
- i, such that,
- Yi | i, si2  normal(i, si2).
- Given i and si2, therefore, each study summary is independent and normally distributed. Futhermore, each study-specific mean, i, is assumed to arise from a distribution of effects with mean and variance (interstudy variation) such that
- i | , 2  ~  normal(, 2).
- Yi | i, si2  = i  + i, i  normal(0, si2),
- where i | , 2  = **x****i**** ******+ i and i  ~  normal(0, 2).
**** ~normal(**b**, D),  ~ ()
- DuMouchel W.,  Normand S-L, “Computer-modeling and Graphical Strategies for Meta-Analysis”,Chapter 6,”Meta-analysis in Medicine and Health Policy”, edited by Stangl DK and Berry DA, Marcel Kekker, Inc 2000 pp.127-177

## Tauplot No covar

## Slide 34

## Heterogeneity

- The usual tests of heterogeneity are know to have low power to identify heterogeneity.
- However,in the case of the Canner example, no matter one examines the data, the conclusion is clear that the AMIS trial is an outlier.
- The focus must then turn to why?

## Assessing the cause Heterogeneity

- Design operation issues (age,dose, duration of followup)
- Baseline characteristics(history of congestive heart failure, history of angina pectoris, History of ECG-documented arrhythmia, use of digitalis, use of nitrogylcerin, use of propanolol or other beta-blockers, other drug,pulse rate(aspirin group may have been more ill)
- Effect by year of follow up
- Non-fatal events (none observed)

## Assessing the causes of Heterogeneity

- Is this case there was some evidence that the heterogeneity may be related to the duration of the the trials or the point of assessment.
- But other explanations may be possible.

## Conclusions

- It is important to include all controlled trials that bear on efficacy(independent of their results)
- Consider how each study contributes (consistency or heterogeneity)
- Recognize that there are many reasons for results to differ (Bias and chance or that there are dealing with different questions)

## References

- Fleiss JL, “The Statistical basis of meta-analysis”, Statistical Methods in Medical Research, 1993: 2: 121-145
- Malec D, “A closer look at combining data among a small number of binomial experiments”, Statistics in Medicine, 2001:20:1811-1823
- Goodman SN, “Meta-analysis and Evidence”, Controlled Clinincial Trials 10:188-204(1989)

## References

- Hung HMJ, Oneill RT, et.al, “The Behavior of the P-value when the alternative hypothesis is true”, Biometrics 53,11-22,March 1997
- Galbraith RF,”A note on graphical presentation of Estimated Odds Ratios from several Clinical Trials” Statistics in Medicine, Vol.7,889-894(1988)
- Canner, PL,”An overview of six clinical trials of Aspirin in Coronary Heart Disease”,Statistics in Medicine, vol.6,225-263(1987)

## References

- Chang B-H, Waternaux, C, Lipsitz S,”Meta-analysis of binary data:which within study variance estimate to use”, Statistics in Medicine 2001:20:1947-1956
- Demets D, Methods for combining Rndomized  clinical trials:strength and Limitations, Statistics in Medicine,Vol.6, 341-348
- Olkin I,Meta-analysis: Reconciling the results of independent studies” statistics in Medicine, vol. 14,457-472(1995)

## References

- DerSimonian R and Laird N.,”Meta-analysis in  Clinical Trials”, Controlled Clinical Trials, 7,177-188,1086
- Cochran WG, “The Combination of Estimates from Different Experiments, Biometrics 10,101-129,1954