# Presentation: 124581

## Integrated Summaries of Safety and Integrated Summaries of Efficacy:  Is this Meta-Analysis?

- C.Anello
- Deputy Director, Office of Biostatistics
- Center for Drug Evaluation and Research
- Joint Statistical Meetings, Baltimore August 8-10,1999

## Disclaimer

- The views expressed are those of the   author and not necessarily those of the  Food and Drug Administration.

## Topics

- Objective of talk
- U.S. Regulatory System
- Integrated Summary of Safety and Efficacy
- Collective Evidence and Its Interpretation
- Conclusions

## Are Integrated Summaries of Safety and Integrated Summaries of Efficacy Meta- Analyses?

- Objective: Answer the Question:

## Drug Development and Drug Review

- Sponsor’s initiate a series of non-clinical and clinical studies: IND phase.
- Sponsor submits the results of these studies and the label claims: NDA phase.
- CDER’s technical staff (Med, Chem, Phar Tox and Stat) evaluate the submission from the perspective of “substantial evidence” Review Phase
- Postmarketing Surveillance

## Changing Laws and Regulations

- 1962 Kefauver-Harris Amendments to the FD&C Act : Efficacy requirement
- 1970 DESI  Summary Judgements: Defining adequate and well controlled studies
- 1986 NDA Rewrite :Documentation,statistical evidence, integrated summaries of safety and efficacy

## Changing Laws and Regulations

- 1997 Food and Drug Modernization Act (FDAMA): Modifies the concept of  “substantial evidence” in some situations

## 1998 ICH: 9 Statistical Principles  

- The formal evaluation of the quantitative evidence from two or more trials bearing on the same question. This most commonly involves the statistical  combination of summary statistics from various trials, but the term is sometimes used to refer to the combination of the raw data.

## Integrated Summary of Effectiveness (ISE)

- Introduction and Background
- Description of Trial Results
- Integration and interpretation of collective evidence
- Summary of all clinical trials
- Nomenclature for characterizing RCT results
- Methods for combining data or information when appropriate

## Integrated Summary

- Efficacy the focus is on the individual studies
- 1970 staff manuals called for  “overall evaluation”
- 1970 FR 314.1 (356H) allowed for “expanded summary”  a) table of all investigators, b) age and gender, c) table of dosages d) summary for each claim , e) safety tabulations and e) overall conclusions
- Safety data are often combined to obtain estimates of event rates

## Integrated Summary

- IES and IES are narrative reviews or overviews, not Meta-analyses
- Pooling of studies within an IS has been referred to as a “Meta-analysis” in submissions
- The pooling of studies results often relate to special populations (age, gender or ethnicity)

## Integrated Summary

- The goal of the integrated summary is to look at the totality of data submitted that of efficacy and not just to focus on the “pivotal” studies that support a particular claim

## Integrated Summary

- Looking at only the “the pivotal” studies to “show effectiveness” will over estimate the drug’s effect.  Looking at all the studies should enhance our understanding of how the drug works in different groups, age, gender, disease severity, dose

## Integrated Summary

- Include all controlled trials that bear on efficacy
- Include the key efficacy data
- Consider how each study contributes (consistency or heterogeneity)
- Examine evidence for dose response
- Examine the important of subgroups
- Relate the evidence in the application to the proposed label

## Nomenclature of Clinical trials

- Sometime the terms we use can get in the way of doing an integrated summary of efficacy and safety

## Nomenclature of Clinical trials(Terms whose meanings lack clarity)

- Pivotal Study
- Positive Study
- Supportive Study
- Equivocal Study
- Inconclusive Study
- Negative Study
- Failed Study

## Integrated Summary Characterizing of Each Study

- Objective
- Controlled trial (populations treated, endpoints, assignments, numbers randomized, method of observation, disposition of patients
- Intention-to-treat relative to full analysis set
- Patterns of completers, dropouts, terminations

## Integrated Summary 

- Group studies by Design and objective
- Grouped by clinical endpoints
- Summary dose response and exposure
- Subgroup analysis (age,gender,etc)
- Analysis of dropouts and deaths
- Integrated analysis (where appropriate)

## Integrated Summary:Collective Evidence

- “Substantial Evidence” is derived from the results of more than one controlled trial
- Both graphical and statistical methods have a role in this effort of understanding the collective evidence
- The collective evidence is more than the sum the individual studies

## Integrated Summary of Effectiveness

- Meta-analytic methods are beginning to be seen in regulatory submissions
- Some are planned integrated analysis  or others are unplanned integrated analysis.
- Are there other ways of looking at the totality of evidence?(Likelihood, Bayesian, Response Surface or regression methods)
- But for the most part these integrated analysis are not “meta-analyses”

## Integrated Summary of Effectiveness

- How was the decision to combine made?  With or without reference to the individual study results.
- Are the study designs similar
- Are the effect measures appropriate
- Are the effects homogeneous across studies
- Are appropriate statistical methods being used
- Studies selected in a manner to avoid bias

## What is Heterogeneity Telling us?

- It may be more informative to look at how the studies results vary
- It may be more informative  to explore the individual study characteristics

## Situations Where Combining Study Results Might Be Appropriate As an Exploratory Exercise Without Precise P-Values

- Safety evaluation
- Generating hypothesis
- Better estimates established  treatment effects
- Evaluate how the treatment effect varies with subgroups
- Evaluate additional Endpoints (e.g., mortality)

## Interpretation of ISE and ISS Statistical uncertainty when some studies are combined

- What is the type I error?
- Are the combined estimates biased?

## Example 1:Ad Hoc combining of results to increase the power

- A Meta-analysis for Coumadin (warfarin sodium) related to prevention of death, recurrent MI, etc
- Issue:  Should  combined results from several under powered studies be accepted as an independent second study?
- May not be appropriate, without pre-planned analysis plan

## Example 2: 

- A Meta-analysis used to Explore Efficacy for a Subgroup

- Seven clinical trials for treatment of HIV-infected individuals
- Trials partitioned into two subgroups based on baseline CD4 cell counts CD4 < 50 cells / mm3 or > 50 cells mm3
- Survival was measured effect
- Led to conclusion that exclusion of patients with CD4 < 50 cells / mm3 may result in prolonged and/or longer clinical trials

- (Kazempour, Kammerman and Farr, 1995)

## Example 3:

- Meta-Analysis Used to Explore Study Characteristics

- 9 clinical studies
- Incorporated study characteristics (Duration of treatment)

## Example 3: Meta-Analysis to Explore Study Characteristics

- Study	Treat. A		Placebo		(Months)		Ratio	   Conf. Interval
-    1		  51 / 75		  38 / 75		      1		 	2.07	    	     (1.07, 4.02)
-    2		100 / 250		100 / 255		      1	          	1.03	    	     (0.72, 1.48)
-    3		  46 / 70		  32 / 72		      1.5	 	2.40	    	     (1.22, 4.72)
-    4		  51 / 65		  41 / 64		      2		 	2.04	   	     (0.94, 4.46)
-    5		  35 / 75		  24 / 78		      1		 	1.97	   	     (1.02, 3.84)
-    6		  79 / 122		  58 / 125		      1.5	 	2.12	   	     (1.27, 3.54)
-    7		  68 / 85		  46 / 85		      2		 	3.39	   	     (1.72, 6.71)
-    8		  46 / 70		  33 / 73		      1.5	 	2.32	   	     (1.18, 4.56)
-    9		  42 / 55		  25 / 60		      1.5	 	4.52   	     (2.02, 10.13)

**Success Rates**
**With Treatments**

**Duration of**
**Treatment**

**Odds**

**95%**

**Each Study:**

**(Huque and Dubey ‘94)**

## Example 3: Continued: Meta-Analysis to Explore Study Characteristics

- X (months of treatment)		1.0	     1.5	       2.0
- Abs. Differences (÷)		    11.2	    18.2	      25.2

## Safety

- ... there is insufficient information about the drug to determine whether the product is safe for use under the conditions prescribed recommended or suggested in its proposed labeling (21 CFR314.125(B)(4), page 120 s of 1 April 1988

## Differences Between ISE and ISS 

- Critical efficacy endpoints identified during design
- Generally safety endpoints not known prior to trial
- Adverse events can occur in any body system
- Adverse events often occur at low incidence
- Sophisticated statistical analysis is often restricted to efficacy evaluation
- The tendency is to crudely combine safety information from multiple clinical trials
- Zero observed events does not mean a drug is safe

## Example 4: Hypothetical Data:  On Combining Adverse Events From Three Trials : The Issue of taking time into account

- Trial		Measure			 1	 2	  3
-     1		No. with event			  5
- 		No. exposed			500
- 		Cumulative rate		              0.01
- 		Crude rate = 5/500 = 0.01
-     2		No. with event			  2	  2
- 		No. exposed			200	100
- 		Cumulative rate		              0.01	0.03
- 		Crude rate = 4/200 = 0.02
-     3		No. with event			  1	  1	  1
- 		No. exposed			100	 50	 25
- 		Cumulative rate	               	0.01       0.03	 0.069
- 		Crude rate = 3/100 = .03

- Months of Exposure

**(O’Neill ‘96)**

## Are Integrated Summaries of Safety and Integrated Summaries of Efficacy Meta- Analyses?

- Objective: Answer the Question:

## Conclusions

- For ISE and ISS some preplanned combined analyses have been referred to as “Meta-analysis”
- For ISE and ISS there are aspects of a M-A but they are really narrative overviews of a complex nature
- For ISS evaluations there are likely to be more situations where Meta-analytic procedures will play an important role

## Conclusions

- For ISE evaluation there are likely to be few situations where meta-analytic methods will play the major role
- There are situations where planned combining of studies results within an ISE or  ISS may be useful

## Conclusions

- Need reason for integrating findings
- Need adequate protocol for doing an integrated analysis of safety or efficacy
- Need to be based on Intention-to-treat
- Need valid end points, similar across studies
- Need research on how to adjust the type I error and avoid bias in the setting of an ISS or ISE.