# Presentation: 245482

## Issues, Analysis Approaches, and Adaptive Testing in Active Controlled Trials*

**Sue-Jane Wang**
**DB2/OB/CDER/FDA**
**Office of Biostatistics Seminar**
**Date: April  21, 2000**

***Based on joint work with **_**H.M. James Hung**_**, **_**Yi Tsong**_**, and **_**Lu Cui**_** (1997-2000)**** ****The work on the Adaptive Testing  is to appear in “Statistics in Medicine”**

## * Wang, Hung, Tsong, Cui, and Nuri. 

**   ****“****Changing Study Objective in Clinical Trials” (1997)**
*** Tsong, Higgins, Wang, and Hung.  **
**   ****“****Statistical Considerations for Clinical Trials with Active Control**
**     ****Treatment” (1999) **
*** Wang, Hung, Tsong, and Cui **
**   ****“****Group Sequential Testing for Superiority and Non-inferiority**
**     ****Hypotheses in Active Controlled Trials” (1999)***** Wang and Nevius**** **
**   “****OB review round on issues of NI design of Critilip - IND” (1998-1999)**
*** Hung and Wang **
**   ****“****Review on Methods of Analysis in Active Control Trials” (1999)**
*** Wang, Nevius, Wilson, O’Neill, Birenbaum, Himmel,  Meyer, Temple. **
**   ****“****Study designs of Surfaxin for Infant RDS syndrome - IND” (1999-2000)**
*** Wang and Hung **
**    ****“****Issues and analysis approaches for active control  trials”(2000)**

## ACKNOWLEDGMENTS

**This research work was supported by CDER/FDA Review Science Research Funds #96010A and #FY99/00-08 **
**Thanks to Dr. Edward Nevius for his encouragement and helpful comments**

## OUTLINE

**Issues**
**Design for both “S” & “NI” objectives**
**Endocrinology & Metabolism drug product**
**Cardio-Renal drug product**
**Group sequential testing**
**Analysis Approaches and their validity to conclude “T” is superior to “P” (imputed)**

## Difficulties with Testing Non-Inferiority in Active Control Trials

**No concurrent placebo for comparison**
**Cannot replicate the effect of active control relative to placebo within the current active control trial**
**Determination of the noninferiority range is often a point of controversy* which rests upon reliable external or historical data about the effect of the active control**

***Temple, R. (1993, **_**Proceedings of the Biopharmaceutical Section, ASA**_**)**

## STATISTICAL FORMULATION

**Superiority ****“S”**
**1-sided equiv****(eg,Blackwelder, 1982)** “**NI”**
**Therapeutic equiv.** (e.g.,Fleming,1987)

**H****S****: ******** = 0   vs. K****S****: ******** > 0**
**H****E****: ******** ********-********E**** vs. K****E****: ******** >-********E**
**H****E****:|********|********E**** ****vs. K****E****: |********|< ********E**

-                              **|<********E****>|<********E****>|****   <----------------*---------****+****------*------****+****----------------*------->****   ****	****          ********C**** - ********S**                                  ******C****                             ********P**

-    ******P****     Placebo event rate  **
**   ********T****    Treatment event rate**
**   ********C****    Active control event rate **
**    ********E****   ****   ****Therapeutic equivalence margin of interest**
**                                              ******** ****= ********C**** ****-**** ********T**** **

## A Case Study

**endpoint: death due to RDS at day-28 after birth**
**treatment: 3.25%; active-control: 2% **
**mean difference (C-T): -1.25%**
**superiority testing:  p=0.65 (failed to show superiority) **
**consider to test if death due to RDS with T is NI than C.**
**lower bound of 95% CI:   -2.1%**
**‡****-------------------------------------------------------------------‡**
**-****	****before analysis:****	****     ********E****=1% (can’t conclude NI)**
**after observing data: ****	****     define ********E****=2.2% (show NI)**
*** Wang, Nevius, Wilson, O’Neill, Birenbaum, Himmel, Meyer, and Temple. “Study designs of Surfaxin for Infant RDS syndrome - IND”  (1999-2000)**

## Non-inferiority objective not planned
Equivalence-range E not pre-specified
Ad-Hoc determination of E influenced by analysis of active-control trial for testing NI

Cannot preserve type I error rate w.r.t. HE 

Ref: Wang SJ, Hung HMJ, Tsong Y, Cui L, Nuri W. (1997, Proceedings of the Biopharmaceutical Section, ASA)

**Equivalence-range ********E ****not pre-specified**
**Ad-Hoc determination of ********E ****influenced by analysis of active-control trial for testing NI**
**Cannot preserve type I error rate w.r.t. H****E**** **
**Ref: Wang SJ, Hung HMJ, Tsong Y, Cui L, Nuri W. (1997, **_**Proceedings of the Biopharmaceutical Section, ASA)**_

**ISSUE**

## PLANNED TESTING FOR “S” AND “NI”(At the end of the trial)

- LL> 0

- LL > -

_**S-NI**_**Step 1: Test H****S****Step 2: Test H****E**         **(involve ********E****)**

**R****eject H****S****          ****“S”**** ** **Reject H****E****           ****“NI”**

******E  ****prespecified**
**Let    D = ^********C**** ****- ^********T**,
**LL**** (lower limit of confidence interval) = D - Z************D**** **
**Rejection region for H****S****:    ****LL**** > 0**
**Rejection region for H****E****:    - ********E**** < ****LL**** **

**PLANNED TESTING FOR “S” AND “NI”****(At the end of the trial)**

**Y**

**N**

**Y**

**N**

## NI-SStep 1: Test HE(involve E) Step 2: Test HS

- LL > -

- LL > 0

_**NI-S**_**Step 1: Test H****E****(involve ********E****)** **Step 2: Test H****S**

**I****nconclusive****     ****Reject H****E****          ****“NI”**

**N**

**N**

**Y**

**Y**

## 
Type I error rates for “S” and “NI” are controlled [Morikawa & Yoshida (1995) and Gould (1997)]

Power for concluding “NI” is low when sample size is planned for testing “S”   [Gould (1997)]

Possible heuristic difference between S-NI and NI-S
With S-NI, the initial sample size might be planned for “S”
With NI-S, the initial sample size might be planned for “NI”

**Type I error rates for “S” and “NI” are controlled [Morikawa & Yoshida (1995) and Gould (1997)]**
**Power for concluding “NI” is low when sample size is planned for testing “S”   [Gould (1997)]**
**Possible heuristic difference between S-NI and NI-S**
**With S-NI, the initial sample size might be planned for “S”**
**With NI-S, the initial sample size might be planned for “NI”**

## Group Sequential Testing for “S” and “NI”

**Ethical reasons may require interim analyses to show “S” or “NI” as early as possible**
**Sample size may need adjustment during the course of the trial (N****E**** >> N****S****)**

## Group Sequential Stepwise (GSS) Test

**Given the same sample size, GSS S-NI = GSS NI-S in terms of rejection regions for “S”, “NI”**

- LLi > -

- LLi>0

- LLi > 0

- LLi > -

- continue

**“****NI”**

**“****S”   **

**“****S”,“NI”****     Stop********“NI”      ****Stop**

- continue

**Stop **** **

**“****NI”**

**Stop **** **

_**S-NI**_** **** **

_**NI-S**_** **** **

**Y**

**Y**

**Y**

**Y**

**N**

**N**

**N**

**N**

**At the ith Interim analysis**** **** **

## GROUP SEQUENTIAL STEPWISE TEST(GSS Test)

**GROUP SEQUENTIAL STEPWISE TEST****(GSS Test)**

**S-NI approach**
**test for superiority with   N****S****   usually planned**
**test for non-inferiority if fails on superiority at an interim analysis**
**NI-S approach**
**test for non-inferiority with N****E **** usually planned **
**test for superiority only if non-inferiority is concluded at an interim analysis**

## Slide 15

## Group Sequential Closed (GSC) Test

**Given the same sample size, GSC S-NI = GSC NI-S in terms of rejection region for “S”, “NI”**

- LLi > -

- LLi>0

- LLi > 0

- LLi > -

- continue

**“****NI”**

**“****S”   **

**“****S”,“NI”****     Stop********“NI”      ****Stop**

- continue

**Stop **** **

**“****NI”**

**Stop ****only at the final**** **** **

_**S-NI**_** **** **

_**NI-S**_** **** **

**Y**

**Y**

**Y**

**Y**

**N**

**N**

**N**

**N**

**only ****at ****the ****final**** **** **

**At the ith Interim analysis**** **** **

## Slide 17

## Group Sequential Closed (GSC) Test

_**Group Sequential **__**Closed**__** (GSC) Test**_

## Interim Adjustment of Sample Size for “NI”

**When the study is initially planned to test for superiority, i.e., initial SS= N****S****, sample size may need to be increased for testing “NI” when observed sample path at the interim analysis indicates that ****“treatments differ” is not plausible**
**e.g., conditional power suggests little likelihood of “showing treatment difference”**
**Type I error rate may be inflated**** when sample size adjustment is based on interim sample path and critical values are generated using the concept of adjusted ********-spending function of Lan & Trost (1997)**

## Adjust P-value boundary for efficacy 

**0**

**5%**

-       **Information time**

**Lan & Trost (1997)**

-  **20%       40%     60%      80%     100%     130%**

-            **Reject  H****0**

**Continue the trial**

## Slide 21

## Need to  ADAPT  Group Sequential Closed Test Procedure (AGSC) 

**Need to  ****ADAPT****  Group Sequential Closed Test Procedure (****A****GSC)**** **
**(Wang, Hung, Tsong, and Cui. Proceedings of Biopharmaceutical Section ASA, 1999)**

## Adaptive Group Sequential Closed Test   (AGSC)

**For superiority, adapt test statistic according to the work of Cui, Hung & Wang (Biometrics, 1999)**
**For non-inferiority, extend the work for superiority to adapt test statistic by adjusting conditional expectation of test statistic**

## Strategy-1:  whether to re-estimate N  evaluated at a pre-specified interim time only

**Strategy-2:  whether to re-estimate  N  evaluated at any interim analysis based on conditional power**

**ADAPTIVE**** GROUP SEQUENTIAL ****CLOSED**** TEST **
**(Adaptive GSC Test)**

## Slide 25

## Simulation Results - Table 4 

**Simulation Results - Table 4 **
**type I error rate control for non-inferiority  **
**type I error rate control for superiority  **
**large gain in power for superiority **
**power for non-inferiority reasonable**
**Strategy-1:  less powerful**
**Strategy-2:  have the power for non-inferiority very close to the planned level**

## Slide 27

## SUMMARY

**We recommend Adaptive GSC Test**
**When true state of treatment efficacy is unclear**
**a superior treatment effect or a non-inferior treatment effect?**
**requires the interim analysis rule, N****S****, non-inferiority margin be pre-specified**
**Advantage of AGSC test**
**flexible**
**allow for studying both objectives with vigor**
**adapt the test procedures **
**allow for sample size re-estimation if NI is more plausible**
**“****n” saving**
**when pursuing both objectives and the likelihood is a superior treatment effect**
**type I error is preserved for both objectives:  ******** ****S****=******** ****NI**** = ******
**allow to stop the trial as early as possible for “S” or for “NI”**

## Treatment Effectivenessin A-C “NI” trial w/o placebo arm

**Effect of “T” similar to that of “C”?  **
**    “****T” preserve at least 50%, say, of the effect of “C”?**
**Statistically,  “T” no worse than “C” by ********E ****?**
**Ref: Tsong Y, Higgins K, Wang SJ, Hung HMJ. (1999, **_**Proceedings of the Biopharmaceutical Section, ASA**_**)**

**Treatment Effectiveness****in A-C “NI” trial w/o placebo arm**

## Therapeutic Equivalence (Non-inferiority)

**Effect of “C” relative to “P” established from external data (e.g., meta analysis)**
**Two-step process to define non-inferiority margin:**
**	****(1) what is the effect of “C”?**
**	****(2) preservation of effect of “C” at what level?**

**Therapeutic Equivalence ****(Non-inferiority)**

## Indirect “CI” comparison method(CBER, Temple, etc.)

******t****= pr{adverse event with treatment t}    t=T, C, P.**
**Effect of treatment assessed on:**
**	****	****Proportion Difference (D), **
**	****    *****Relative risk (RR),   RR****TP**** = ********T****  / ********P**
**	****	****Odds ratio (OR),**
**	****	****Hazard ratio (HR).**
*** CBER memos recommend RR be estimated by the respective upper confidence limits**

**Indirect “CI” comparison method****(CBER, Temple, etc.)**

## Indirect “CI” comparison method(CBER, Temple, etc.) - con’t

**“****effect size of T ******** (1- ********) x effect size of C [r.t. P]”**
**	****	****1- ********T****/********P ******** (1- ********) (1- ********C****/********P ****)**
**	****	****iff ********T****/********C ******** ************ ******** {********C****/********P ****+ ********(1- ********C****/********P ****)}/ (********C****/********P ****)**

**Indirect “CI” comparison method****(CBER, Temple, etc.) - con’t**

## Virtual Method(Hasselblad & Kong, Fisher, etc.)

**Directly estimate the effect of T relative to P by**
**	****	****	****T - P = (T - C) + (C - P)**
**	****	****	****var(T - P) = var(T - C) + var(C - P)**
**      ****	**** ****(T - C)  obtained from the active control trial**
**       ****	**** ****(C - P)  obtained from placebo-control trials**
**Assuming (T - C) and (C - P)**** ****are from statistically independent trials**

**Virtual Method****(Hasselblad & Kong, Fisher, etc.)**

## Virtual Method(Hasselblad & Kong, Fisher, etc.)

**	****	****log(T/P) = log(T/C) + log(C/P)**
**	****	****var(log(T/P)) = var(log(T/C)) + var(log(C/P))**
**Construct CI of T/P to examine if it is less than one**
**Compute p-value using z-statistic of log(T/P)**
**Construct CI of (1 -T/P)/(1- C/P) to examine if lower **
**	****confidence limit exceeds (1-********)**

**Virtual Method****(Hasselblad & Kong, Fisher, etc.)**

## Bayesian Method(Gould, Simon, etc.)

_**Gould**_**: ****Empirical Bayes****, and ****prediction theory**** **
**(determine likelihood of a significant difference**
**between T and P had there been P)**
_**Simon**_**: ****Model dependent**
**	****y= ********+ ******** x+ ******** z + ******
****** ****- effect of P;  ******** - effect of (C-P);  ********  - effect of (T-C)**
**Require prior distributions for ********, ********, and ********. **
**Interpret effect of T vs. P via posterior probability**

**Bayesian Method****(Gould, Simon, etc.)**

## Bayesian Method(Simon, [1999, Biometrics])

**In the example of his article, Simon uses **
**	********non-informative prior for effect of P**
**	********non-informative prior for effect of T r.t. P, and**
**	********meta-analytic estimate distribution as a prior for**
**       ****effect of C r.t. P,**** **
**Then, compute posterior probability**
****** ****Pr{effect of T r.t. virtual P is positive | data}**

**Bayesian Method****(Simon, [1999, Biometrics])**

## Bayesian Method(Simon, [1999, Biometrics])

**Bayesian Method****(Simon, [1999, Biometrics])**

****** ****Pr{effect of T r.t. virtual P is positive | data}**
**If this posterior probability is large (say, ******** 0.95), then conclude that  T is superior to P**
**Likewise, compute the posterior probability**
****** ****Pr{C has a positive effect and T preserves at least 100 (1-********)% of the effect of C | est. of (T-C)}**
**If this posterior probability is large (say, ******** 0.95), then **
**   ****conclude that T preserves at least (1-********)% of effect of C**

## Bayesian Method(Simon, [1999, Biometrics])

**Bayesian Method****(Simon, [1999, Biometrics])**

**Prior distributions do not have to be those used by Simon**
**Comment:****	**** With the non-informative priors used by Simon in that example, Bayesian method is equivalent to virtual method. For comparing T vs. P,**
****** ****one-sided p-value = 1 - posterior probability**

## Comparison of T vs. “P”(imputed, putative, virtual, vestigial placebo)

**Comparison of T vs. “P”****(imputed, putative, virtual, vestigial placebo)**

**False positive rate associated w/ comparison of T vs. “P”**
**- effect of C r.t. P, its precision from historical data (sample size)**
**- effect of C in historical data = effect of C in active controlled trial ???**
**- % of effect of C needing preservation**
**- precision of estimated difference between T and C in active control trial (sample size)**

## Approaches Used for Comparisons of T vs. “P”

**Approaches Used ****for Comparisons of T vs. “P”**

**Indirect Confidence Interval Method (CBER, Temple)**
**	****	****CBERL - 50%;        CBERP - 50%**
**	****	****CBERL - 80%;****	****    CBERP - 80%**
**Virtual Method (Hasselblad & Kong, Fisher, etc.)**
**	****	****H&K**
**Bayesian Method (Simon)**
**	****	****Bayes - 97.5%**
**Others not used: 99% CI, worst case drug effect, etc.**

## Simulation Study

**Simulation Study**

**10,000 runs**
**Sample size for A-C is fixed**
**In each run, sample size for active control trial is estimated using the estimated effect size (either point estimate or with confidence limit) for the control from external data (assuming meta analysis fixed- effect or random-effect approach is valid)**

## Slide 42

## Slide 43

## Slide 44

## Slide 45

## Slide 46

## Comparison of T vs. “P”(imputed, putative, virtual, vestigial placebo)

**Comparison of T vs. “P”****(imputed, putative, virtual, vestigial placebo)**

**“****N” for A-C trial estimated from historical trials:**
**using ************: Makuch & Simon (1978)**
**using Bayesian: Simon (1999)**
**Both approaches led to large sample size for current A-C trial, therefore, patterns of inflation of false positive rate are similar (Fig 6a and 6b)**
**Such sample size estimation can be much larger than 5000 of fixed sample size, amount of inflation larger than Fig.1 **
**Empirical type I error rates could increase to more than 0.9**

## Slide 48

## Slide 49

## Summary on [T vs. “P”]*

**Summary on [T vs. “P”]***

**False positive rate can inflate when :**
**	****effect of “C” of A-C trial “worse” than effect of “C” of historical P-C trials**
**Inflation ********  when**
**sample size ******** in either the A-C trial or the historical P-C trials (Fig. 1-3)**
**effect of C ******** (Fig. 5)**
***Wang and Hung (2000) “Issues and analysis approaches for active control trials”**

## Summary on [T vs. “P”]*

**All three approaches maintain type I error rate (say, 1-sided 2.5%) when**
**effect of C in external trials = effect of C in active trial**
**Q: how sure are we about ****	**
**	****	********C**** (historical trial) = ********C**** (current trial)  ?**
**- populations “same” ?**
**- background treatments and therapies “same” ?**
**- dosage for active control administration “same” ? **

**Summary on [T vs. “P”]***

## Summary on [T vs. “P”]*

**Virtual method = Bayesian* given same sample size**
**** use of flat priors for effect of P, effect of (T-P), and***
***   ******use of informative prior from est. distn. of (C-P)***
**Indirect CI:**
**	****preserving 80% effect of C is more conservative than preserving 50% effect of C**
**“****Indirect CI” is much more conservative than “Virtual”**
**It protects against worsening “C” to a great extent.**
**“****Virtual”method can be very sensitive to difference in effects of control between A-C and historical trials.**

**Summary on [T vs. “P”]***

## Our work on % change of control event rate presented at the OB seminar on April 21, 2000 will be presented based on the relative risk scale for the coming ICSA Applied Statistics Symposium on June 2, 2000.

**Terms used between the two presentations: **
**   ****ICI: Indirect Confidence Interval, i.e, CBER/FDA method**
**   ****Virtual method refers to Hasselblad&Kong, Fisher, etc.**