# Presentation: 518982

## Interpretation of Observed Differences in the Frequency of Events When the Number of Events is Small

**Interpretation of Observed Differences in the Frequency of Events When the Number of Events is Small**

- Milton Packer, M.D.
- Gayle and Paul Stoffel Distinguished Chair in Cardiology
- Professor and Director
- Center for Biostatistics and Clinical Science
- University of Texas Southwestern Medical School at Dallas
- US Food and Drug Administration
- Friday, February 18, 2005
- 8:15 AM

## Question

- •	How should we interpret differences in the 	observed frequency of events in a clinical 	trial when the number of events is small?

## Question

- •	How should we interpret differences in the 	observed frequency of events in a clinical 	trial when the number of events is small?
- • 	By “small”, I mean that the number of 		events would have provided < 70% power 	to have detected a true treatment difference 	— assuming an effect size similar to that 	generally encountered in clinical research.

## How Should Such a Difference Between Treatment Groups Be Interpreted?

- 33

- Placebo

- Drug

*Major Adverse Cardiovascular Event*

- Hypothetical trial of 3000 patients
- (1500 in each group)

## How Should Such a Difference Between Treatment Groups Be Interpreted?

- 33

- Placebo

- Drug

- RR=2.63
- 95% CI (1.39, 5.00)
- P = 0.002

*Major Adverse Cardiovascular Event*

## Remember

- •	P values are most easy to interpret 		when they refer to reproducibility of 		observed differences in predefined 		primary endpoints in trials adequately 		powered (> 80-90% power) to detect 		differences between treatments.

## Probability That Second Trial Would Find P < 0.05 Effect if Second Trial Were Identical to First Trial

-  0.10		   		37%
-  0.05		   		57%
-  0.01		   		73%
- 0.005	   	   		80%
- 0.001	   	   		91%

- P Value in
- First Trial

- Probability of  P<0.05
- in Second Trial

*O’Neill.  Cont Clin Trial 1997;18:550-6.*

## What If  . . .

- •	The event was not the primary 			endpoint of the study.
- • 	The event was not precisely defined 		before the start of the trial.
- • 	The trial was not adequately power to 	detect a treatment difference.

## This Frequently Happens . . .

- • 	With primary endpoints in pilot trials.
- •	With secondary endpoints.
- • 	With subgroup analyses.
- • 	With other measures of efficacy.
- • 	With physiological measurements.
- • 	With assessments of safety.

## Things to Worry About When Analyzing Incidence of Adverse Events

## Things to Worry About When Analyzing Incidence of Adverse Events

- 1. 	There are hundreds of adverse events 	(multiplicity of comparisons).

## Multiplicity of Comparisons

- • 	A typical large-scale clinical trial may 		describe as many as 500 individual 		terms describing adverse events.
- • 	If P value were calculated for each 		pairwise comparison, then one would 		— by chance alone — expect
- 	****	  25 events (5%) to have P ≤ 0.05
- 	****	    5 events (1%) to have P ≤ 0.01

## Things to Worry About When Analyzing Incidence of Adverse Events

- 1. 	There are hundreds of adverse events 	(multiplicity of comparisons).
- 2. 	Adverse events are spontaneous 		(nonadjudicated) reports.

## Adverse Events Are Spontaneous (Nonadjudicated) Reports

- • 	Adverse events are reported at the 		discretion of the investigator and then 	translated into standardized terms.
- • 	There is little uniformity as to how an 		event is identified, defined or reported.
- • 	Uncertainty increased when event is in 	field remote from investigator’s focus.

## Can We Fix This Problem by Blinded Post Hoc Adjudication?

## Can We Fix This Problem by Blinded Post Hoc Adjudication?

- • 	Rules guiding post hoc adjudication 		are inevitably influenced by knowledge 	that a treatment effect has been seen.
- 	—	Any bar set by the post hoc process 		can magnify or dilute the effect.
- • 	Adjudication is generally not applied to 	those without a reported event.

## Things to Worry About When Analyzing Incidence of Adverse Events

- 1. 	There are hundreds of adverse events 	(multiplicity of comparisons).
- 2. 	Adverse events are spontaneous 		(nonadjudicated) reports.
- 3. 	Analyses that depend on grouping of 		events are subject to bias.

## Thrombotic Cardiovascular Events

- Drug

- Myocardial infarction	5	 20
- Stroke	4	  4
- Sudden death	5	  6
- Unstable angina	6	  5
- Pulmonary embolism	3	  4
- Arterial embolism	0	  1
- Transient ischemic attack	5	  3
- Venous thrombosis	4	  6

## Thrombotic Cardiovascular Events

- Drug

- Myocardial infarction	5	 10
- Stroke	4	  8
- Sudden death	5	  6
- Unstable angina	6	  5
- Pulmonary embolism	3	  4
- Arterial embolism	0	  1
- Transient ischemic attack	5	  3
- Venous thrombosis	4	  6

## Thrombotic Cardiovascular Events

- Drug

- Myocardial infarction	5	  4
- Stroke	4	  4
- Sudden death	5	  9
- Unstable angina	6	 13
- Pulmonary embolism	3	  5
- Arterial embolism	0	  1
- Transient ischemic attack	5	 11
- Venous thrombosis	4	  6

## Thrombotic Cardiovascular Events

- Drug

- Myocardial infarction	5	  4
- Stroke	4	  4
- Sudden death	5	  9
- Unstable angina	6	 13
- Pulmonary embolism	3	  5
- Arterial embolism	0	  1
- Transient ischemic attack	5	 11
- Venous thrombosis	4	  6

## Analyses That Depend on Grouping of Adverse Events

- • 	Best to develop uniform definition of a 	“group” before classifying events.
- • 	When the process of developing a 		definition is started after a concern has 	been raised, those creating the definition 	have frequently already looked at the 		data and know (subconsciously) what 	kind of definition is needed to capture 		the events of interest.

## Things to Worry About When Analyzing Incidence of Adverse Events

- 1. 	There are hundreds of adverse events 	(multiplicity of comparisons).
- 2. 	Adverse events are spontaneous 		(nonadjudicated) reports.
- 3. 	Analyses that depend on grouping of 		events are subject to bias.
- 4. 	Small number of events results in 		extremely imprecise estimates.

## With Small Number of Events, Lack of an Observed Difference Does Not Rule Out Existence of True Difference

- 25

- 26

- Placebo

- Drug

- RR=1.04
- 95% CI (0.60-1.79)
- P = 0.89

- With Small Number of Events, Lack of an Observed Difference Does _*Not*_ Rule Out Existence of True Difference

*Major Adverse Cardiovascular Event*

## With Small Number of Events, the Finding of an Observed Difference Does Not Prove Existence of True Difference

- 13

- 33

- Placebo

- Drug

- RR=2.63
- 95% CI (1.39, 5.00)
- P = 0.002

*Major Adverse Cardiovascular Event*

## Effect Size ( 95% CI) in Trials With 20-500 Events Assuming Lower Bound ≈ 1.00

- Hazard Ratio

- 20
- events

- 1.0

- 2.0

- 3.0

- Effect Size ( 95% CI) in Trials With 20-500 Events Assuming Lower Bound ≈ 1.00

- 8.0

- 50
- events

- 100
- events

- 200
- events

- 500
- events

- Each trial has 2
- treatment groups,
- each with n=1500

## Trials Are Designed to Provide Precise

- Very precise
- estimate

- Very imprecise
- estimate

- Primary endpoint

- Secondary endpoint

- Specific adverse event

- Estimates For Primary Endpoints

- Start

## What Is Wrong With Imprecise Estimates? 

- What Is Wrong With Imprecise Estimates?

## What Is Wrong With Imprecise Estimates? 

- What Is Wrong With Imprecise Estimates?

## What Is Wrong With Imprecise Estimates? 

- •	Imprecise estimates are problematic if the intent 	is to stop and reach a conclusion.

- What Is Wrong With Imprecise Estimates?

## What Is Wrong With Imprecise Estimates? 

- •	Imprecise estimates are problematic if the intent 	is to stop and reach a conclusion.
- •	When calculated in the conventional manner, 	the 95% CIs (and the associated P value) of an 	estimate have meaning primarily in the context 	of a completed experiment.

- What Is Wrong With Imprecise Estimates?

## What Is Wrong With Imprecise Estimates? 

- What Is Wrong With Imprecise Estimates?

## What Is Wrong With Imprecise Estimates? 

- •	Viewed from the amount of data needed for a 		precise estimate, the adverse event data in a 		single study represents a snapshot in an ongoing 	experiment to characterize the safety of the drug.

- What Is Wrong With Imprecise Estimates?

## What Is Wrong With Imprecise Estimates? 

- •	Viewed from the amount of data needed for a 		precise estimate, the adverse event data in a 		single study represents a snapshot in an ongoing 	experiment to characterize the safety of the drug.
- •* **	*Therefore, performing an analysis of adverse 		events data is akin to interim analysis of primary 		endpoint data in an ongoing clinical trial.

- What Is Wrong With Imprecise Estimates?

## Interim Monitoring in Group Sequential Trials 

- Information Time (% of Expected Events)

- Interim Monitoring in Group Sequential Trials

- Treatment Difference (Z score)

- 2.0

- 0.0

- -2.0

- 4.0

- 6.0

- 8.0

## Interim Monitoring in Group Sequential Trials 

- Information Time

- 2.0

- 0.0

- -2.0

- 4.0

- 6.0

- 8.0

- Interim Monitoring in Group Sequential Trials

- Z score

- =0.05

## Interim Monitoring in Group Sequential Trials 

- Information Time

- 2.0

- 0.0

- -2.0

- 4.0

- 6.0

- 8.0

- Interim Monitoring in Group Sequential Trials

- Z score

- =0.05

## Interim Monitoring in Group Sequential Trials 

- Information Time

- 2.0

- 0.0

- -2.0

- 4.0

- 6.0

- 8.0

- Interim Monitoring in Group Sequential Trials

- Z score

- =0.05

## Interim Monitoring in Group Sequential Trials 

- Information Time

- 2.0

- 0.0

- -2.0

- 4.0

- 6.0

- 8.0

- =0.05

- Interim Monitoring in Group Sequential Trials

- Z score

## Interim Monitoring in Group Sequential Trials 

- Information Time

- 2.0

- 0.0

- -2.0

- 4.0

- 6.0

- 8.0

- =0.05

- Interim Monitoring in Group Sequential Trials

- Z score

## Interim Monitoring in Group Sequential Trials 

- Information Time

- 2.0

- 0.0

- -2.0

- 4.0

- 6.0

- 8.0

- =0.05

- Interim Monitoring in Group Sequential Trials

- Z score

## Interim Monitoring in Group Sequential Trials 

- Information Time

- 2.0

- 0.0

- -2.0

- 4.0

- 6.0

- 8.0

- =0.05

- Interim Monitoring in Group Sequential Trials

- Z score

## Coronary Drug Project 

- Coronary Drug Project

- 3.0

- 2.0

- 1.0

- 0

- -1.0

- -2.0

- -3.0

- 20

- 40

- 60

- 80

- 100

- 0

**Z score**

**Months**

- Clofibrate better

- Placebo better

- 5.0

- 4.0

## Coronary Drug Project 

- Coronary Drug Project

- 3.0

- 2.0

- 1.0

- 0

- -1.0

- -2.0

- -3.0

- 20

- 40

- 60

- 80

- 100

- 0

**Z score**

**Months**

- Clofibrate better

- Placebo better

- =0.05

- 5.0

- 4.0

## Interim Monitoring in Group Sequential Trials 

- Information Time

- 2.0

- 0.0

- -2.0

- 4.0

- 6.0

- 8.0

- =0.05

- Interim Monitoring in Group Sequential Trials

- Z score

- =0.01

- =0.001

## An Important Note 

- •	This problem is related to the imprecision 		inherent in estimates based on small numbers 	of events — an imprecision not adequately 		quantified by conventional approaches to the 		calculation of confidence intervals.

- An Important Note

## Concept 

- Concept

- •	Reaching conclusions from data derived 	in an underpowered trial raises the same 	concerns as reaching conclusions based 	on an underpowered interim analysis in a 	definitive, adequately powered trial.

## IMPRESS (Omapatrilat in Heart Failure)

- a

- y

- s

- f

- r

- o

- m

- r

- a

- n

- d

- o

- m

- i

- z

- a

- t

- i

- o

- n

- P

- r

- o

- p

- o

- r

- t

- i

- o

- n

- w

- i

- t

- h

- e

- v

- e

- n

- t

- 39 events, RR=0.53
- 95%CI (0.27-1.02)
- P = 0.053

**0.15**

**0.10**

**0.05**

**0.00**

**0**

- 30

- 60

- 90

- 120

- 150

- 180

- 210

- 240

*Death or Hospitalization for Heart Failure*

- ACE inhibitor
- (n=284)

- Omapatrilat
- (n=289)

- Rouleau et al.  Lancet 2000; 356:615-20.

## OVERTURE Trial

- Omapatrilat (n=2886)

- ACE inhibitor (n=2884)

** **

**0.0**

**0.2**

**0.4**

**0.6**

**0.8**

**1.0**

- Months

**0**

**3**

**6**

**9**

**12**

**15**

**18**

**21**

**24**

- % Event Free Survival

- 1887 events, RR=0.94
- 95%CI (0.86-1.03)
- P = 0.187

- Packer et al.  Circulation 2002: 106:920-6.

*Death or Hospitalization for Heart Failure*

## Amlodipine in Heart Failure

- PRAISE-1

-   74/212

- 45/209

-      0.55

-   0.001

- (0.37,0.79)

-     Hazard

- Log-rank

-  Placebo

-  Amlodipine

-       Ratio

- P-Value

*All-Cause Mortality in Nonischemic Cardiomyopathy*

## Amlodipine in Heart Failure

- PRAISE-1

-   74/212

- 45/209

-      0.55

-   0.001

- (0.37,0.79)

-     Hazard

- Log-rank

-  Placebo

-  Amlodipine

-       Ratio

- P-Value

*All-Cause Mortality in Nonischemic Cardiomyopathy*

- PRAISE-2

-     262/826

-   278/826

-      1.09

-   0.32

- (0.92, 1.29)

## Experience With Vesnarinone and Losartan

- N Engl J Med 1993;329:149-55. N Engl J Med 1998;339:1810-6.
- Lancet 1997; 349: 747-57. Lancet 2000; 355:582-7.

## Definitive Trial Shows Reversal of Effect Seen in Earlier Pilot Trial

- N Engl J Med 1993;329:149-55. N Engl J Med 1998;339:1810-6.
- Lancet 1997; 349: 747-57. Lancet 2000; 355:582-7.

## Magnesium in Myocardial Infarction

- analysis

-   53/644

- 25/644

- 0.45

- < 0.001

- (0.28,0.71)

- Hazard

- Log-rank

-  Placebo

-  Magnesium

- Ratio

- P-Value

*All-Cause Mortality*

- LIMIT-2

-   90/1159

- 118/1157

- 0.74

- 0.04

- (0.55,1.00)

## Magnesium in Myocardial Infarction

- analysis

-   53/644

- 25/644

- 0.45

- < 0.001

- (0.28,0.71)

- Hazard

- Log-rank

-  Placebo

-  Magnesium

- Ratio

- P-Value

*All-Cause Mortality*

- LIMIT-2

-   90/1159

- 118/1157

- 0.74

- 0.04

- (0.55,1.00)

- ISIS-4

-   2103/29039

- 2216/29011

- 1.06

- 0.07

- (1.00,1.12)

## Metoprolol XL in Heart Failure

- RESOLVD

-    5/212

- 15/214

-      ------

-  < 0.05

- (1.01,5.63)

-     Hazard

- Log-rank

-  Placebo

-  Metoprolol

-       Ratio

- P-Value

*Effect on CHF Hospitalizations*

## Metoprolol XL in Heart Failure

- RESOLVD

-    5/212

- 15/214

-      ------

-  < 0.05

- (1.01,5.63)

-     Hazard

- Log-rank

-  Placebo

-  Metoprolol

-       Ratio

- P-Value

*Effect on CHF Hospitalizations*

- MERIT-HF

-     294/826

-   200/826

-      -----

-  < 0.001

- (-----, -----)

## What We Have Learned

- •	To achieve statistical significance in an underpowered analysis, the effect size must be extreme and the estimate must be imprecise.
- •	Yet, the more extreme the effects and the more imprecise the estimates, the less likely they will be reproduced in definitive clinical trials.

## Things to Worry About When Analyzing Incidence of Adverse Events

- 1. 	There are hundreds of adverse events 	(multiplicity of comparisons).
- 2. 	Adverse events are spontaneous 		(nonadjudicated) reports.
- 3. 	Analyses that depend on grouping of 		events are subject to bias.
- 4. 	Small number of events results in 		extremely imprecise estimates.

## What To Do?

## What To Do?

- •	The most important first step is to 		develop an approach to analyzing data 	in trials with small numbers of events 		which accurately reflects the true 		imprecision of the treatment effect 		estimate and its statistical significance.

## How Should Such a Difference Between Treatment Groups Be Interpreted?

- value for =0.05 is Z=1.96

- 13

- 33

- Placebo

- Drug

- RR=2.63
- 95% CI (1.39, 5.00)
- P = 0.002

*Major Adverse Cardiovascular Event*

## Interim Monitoring in Group Sequential Trials 

- Information Time

- 2.0

- 0.0

- -2.0

- 4.0

- 6.0

- 8.0

- =0.05

- Interim Monitoring in Group Sequential Trials

- Z score

## Interim Monitoring in Group Sequential Trials 

- Information Time

- 2.0

- 0.0

- -2.0

- 4.0

- 6.0

- 8.0

- =0.05

- Interim Monitoring in Group Sequential Trials

- Z score

## Interim Monitoring in Group Sequential Trials 

- Information Time

- 2.0

- 0.0

- -2.0

- 4.0

- 6.0

- 8.0

- =0.05

- Interim Monitoring in Group Sequential Trials

- Z score

## How Should Such a Difference Between Treatment Groups Be Interpreted?

- 33

- Placebo

- Drug

- RR=2.63
- 95% CI (< 0.8, > 6.0)
- P > 0.10

- If assume critical value for =0.05 is *boundary* Z score (across range of effect sizes)

*Major Adverse Cardiovascular Event*

## Boundary-Adjusted Confidence Intervals

- •	Appropriately describe the uncertainty inherent 	in the analysis of small number of events, 		markedly reducing the false positive error rate.
- • 	Yet, despite using a boundary-adjusted 		confidence interval. adverse effects that are 	known to be characteristic of specific drugs 	remain highly significant.
- • 	Do not provide a way to interpret trends 		observed in imprecise data.

## What Should We Do With Worrisome Trends in Imprecise Data?

## What Should We Do With Worrisome Trends in Imprecise Data?

## An Indisputable Truth

- “real” based on biological plausibility,
- because physicians can always be relied
- upon to propose a biological mechanism to explain the validity of an unexpected (and potentially preposterous) finding that happens to have an interesting P value.

## What Should We Do With Worrisome Trends in Imprecise Data?

- •	Look for confirmatory evidence in other 	studies (avoid being selective).

## Experience with Drug A

- Drug

- Trial 1	   	13		      33
- Trial 2	   	10		      15
- Trial 3	    	 5		       8
- Trial 4	    	 2		       0
- Trial 5	    	 8		      14

*Major Adverse Cardiovascular Event*

## The Cumulative Meta-Analyses 

- Information Time

- 2.0

- 0.0

- -2.0

- 4.0

- 6.0

- 8.0

- The Cumulative Meta-Analyses

- Z score

## Boundaries for Cumulative Meta-Analyses 

- Information Time

- 2.0

- 0.0

- -2.0

- 4.0

- 6.0

- 8.0

- Z score

- =0.05

- =0.01

- =0.001

- Boundaries for Cumulative Meta-Analyses

## Boundaries for Cumulative Meta-Analyses 

- Information Time

- 2.0

- 0.0

- -2.0

- 4.0

- 6.0

- 8.0

- Z score

- =0.05

- =0.01

- =0.001

- Boundaries for Cumulative Meta-Analyses

## Boundaries for Mg-MI Meta-Analyses 

- Information Time

- 2.0

- 0.0

- -2.0

- 4.0

- 6.0

- 8.0

- Z score

- =0.05

- =0.01

- =0.001

*Pogue & Yusuf, Cont Clin Trials 1997;18:580-93*

- Boundaries for Mg-MI Meta-Analyses

## What Should We Do With Worrisome Trends in Imprecise Data?

- •	Look for confirmatory evidence in other 	studies (avoid being selective).
- • 	Carry out definitive trial with the adverse 	event as primary endpoint (powered to 	detect meaningful treatment difference).

## Are Definitive Trials the Answer?

- • 	Definitive trial can address ascertainment 	and classification biases as well as 		concerns about multiplicity of comparisons 	and imprecision of data.

## Just Pause and Think 

- Just Pause and Think

## Do We Need to Be So Certain When Evaluating Safety Instead of Efficacy?

## Do We Need to Be So Certain When Evaluating Safety Instead of Efficacy?

- •	Some might advocate being less strict in 	reaching conclusions about *safety*, but 		saying that there is an adverse effect when 	there is none means millions will be 		deprived of an effective treatment.

## Conclusions

- •	The findings of controlled clinical trials are most 		easily interpreted when they represent the 			principal intent of the study.
- •	A non-principal finding is subject to many 			interpretative difficulties, including ascertainment 		biases and inflated false positive rates due to the 	multiplicity of comparisons and imprecision of 		estimates inherent in analysis of small numbers.
- •	The FDA, industry and academia remain 			in a quandary as to how to respond in a 			responsible fashion to observed differences in 		reported frequencies of adverse events.

## A Final Personal Note

- •	My presentation should not be construed as 		favoring any particular side in the current debate.
- • 	It is my view that — regardless of one’s position 		— it is critical to understand the limitations of what 	we know and to resist the temptation to reach 		conclusions before we are justified to do so.
- • 	Only by recognizing our ignorance will we be able 	to take the first step towards developing a 			rational approach that is in interest of all patients.