# Presentation: 538761

## Adaptive Statistical Inference Following Sample Size Modification Based on Interim Review of Effect Size

**H.M. James Hung (DB1/OB/OPaSS/CDER/FDA)**
**Lu Cui (Aventis Pharmaceuticals)**
**Sue-Jane Wang (DB2/OB/OPaSS/CDER/FDA)**
**John Lawrence (****DB1/OB/OPaSS/CDER/FDA)**
***Present in ASA Joint Meetings, New York, NY, August 12, 2002***

**Notes:**
Our RSR projects cover two subjects:  adaptation of design/analysis and quantification of strength of evidence in Phase III clinical trials

## 

                           Disclaimer

The views expressed in this presentation are not 
necessarily those of the U.S. Food and Drug 
Administration, nor of Aventis Pharmaceuticals.
Dr. Lu Cui was one of the primary investigators
of this research during his tenure with FDA.

**                           ****Disclaimer**
**The views expressed in this presentation are not **
**necessarily those of the U.S. Food and Drug **
**Administration, nor of Aventis Pharmaceuticals.**
**Dr. Lu Cui was one of the primary investigators**
**of this research during his tenure with FDA.**

**Notes:**
Our RSR projects cover two subjects:  adaptation of design/analysis and quantification of strength of evidence in Phase III clinical trials

## Acknowledgments

- The research was supported by FDA/CDER RSR Funds, #96-010A, #99/00-008, and #02-06
- _ _
- The materials of this presentation are selected from the main results of our RSR research work.
**Cui, Hung, Wang (1997 ASA; 1999 Biometrics)**
**Hung, Cui, Wang, Lawrence (2002, NJ Chapter of ASA Spring Symposium)**** **

## Selected References in Adaptive Sequential Design/Interim Analysis

**Bauer & Köhne (1994, Biometrics)**
**Bauer & Röhmel (1995, Stat. In Med.)**
**Lan & Trost (1997, ASA Proceedings)**
**Fisher (1998, Stat. In Med.)**
**Posch & Bauer (1999, Biometrical J.)**
**Kieser, Bauer & Lehmacher (1999, Biometrical J.)**
**Lehmacher & Wassmer (1999, Biometrics)**
**Müller & Schäfer (2001, Biometrics)**
**Berry (2002, ASA Biopharmaceutical Report)**
**Brannath, Posch & Bauer (2002, JASA)**
**.........****. etc**

##                      Problem       

**     **

** **

** **

** **

** **

** **
**  **

** **

- y1i  N(1 , 2),   i = 1, ..., n
- y2i  N(2 , 2),   i = 1, ..., n
-   = 1 - 2 ,       2 = 1 (assumed for ease of presentation)
- Test   H0:   = 0   vs.    H1:  > 0
- with two-sample  Z  statistic
- To detect   =   at sig. level  and power 1-,
-     n = 2(z+z)2/2

-                      Problem

##   What is   (assumed effect size)?

- Wish to studyif   0 (target value) > 0
- (1) 0 may be a fixed number
-   e.g., as a public health question, will a medical product that has long been used to treat a disease population reduce morbidity/mortality?
- - plentiful experiences in safety, cost, treatment
-     administration, etc.
-     Choose       0

##     What is   (assumed effect size) ?

- (2) 0 may be a moving target
-   e.g., in new drug development,
-   - very limited data on  (from Phase II trials,
-      external information, etc)
-   - little safety experience in human use
-   - cost and other considerations
-   - clinical value of effect size not assessed at
-     design stage
-     Choose    based on “educated guess” of a range of plausible effect sizes

-  _Common strategy_
-      Use fixed-information design that considers possibly unexpected outcomes
-  This is **always good** because
-  - commonly used statistics have good statistical
-     properties: sufficiency & efficiency
-  - avoid data-dependent adjustments that may induce
-     biases (statistical or operational) making the
-     results possibly not interpretable

-  **But in drug development ....**
-  - true effect size depends on a primary parameter
-    (e.g., mean treatment difference) and nuisance
-    parameters (e.g., standard deviation, background
-    event rate)
-  - clinical value of true effect size depends on
-    benefit/risk assessment (subjective)  that might be
-    doable only in the late of the trial, hard to reach
-    consensus
-  - effect size for detection may depend on $ 
-     benefit/risk/cost consideration
- .......... etc

##      Sample Size Re-estimation

-      Sample Size Re-estimation

- Need of ‘n’ re-estimation in drug development has been well recognized
- Increasing ‘n’ (or amount of statistical information)
- based on **nuisance** parameters w/o breaking blind
-   - has little effect on type I error
-   - may preserve the intended power level
-   - needs no statistical adjustment (e.g., estimate, CI)
**Wittes & Brittain (1990), Gould (1992), Gould & Shih (1992)**
**Shih (1992, 1993, 1995), Birkett & Day (1994) **
**Jennison & Turnbull (1999, book),  ......... etc**

**Notes:**
In our previous work, we have shown that type I error rate may be inflated or statistical power.....

_**Caveat**_
**Lan (1997, ASA talk), Liu (2000, ICSA talk)**
**    **Knowing the components of the variance can
- lead to estimation of treatment difference; hence
- sample size re-estimation based on variance
- might affect type I error depending on how it is
- processed (e.g., by obtaining TSS & WSS)

**Notes:**
In our previous work, we have shown that type I error rate may be inflated or statistical power.....

- Increasing ‘n’ (or amount of statistical information) based on the internal data path may substantially inflate type I error, compromise statistical power, bias the estimate, invalidate CI

**Notes:**
In our previous work, we have shown that type I error rate may be inflated or statistical power.....

##  Sample Size Re-estimation

**     **

**     **

** **

** **

** **

** **

-  Sample Size Re-estimation

**Question:**
- At an interim time of a trial, if the observed treatment difference suggests to test a smaller effect size (worthwhile), one **wishes** to increase sample size.  Then, what adjustments are needed to perform valid statistical testing?

##     Sample Size Re-estimation

-     Sample Size Re-estimation
-          (non-sequential trial)
- Plan to enroll  n=100  subjects/group to detect
-  = 0.46  at   = 0.025  and power  90%
- After 40 subjects per group contribute data,
- the estimate of    suggests to increase  n  to a total of  m = 200/group  for detecting new assumed  * = 0.325  with power  90%

## Without sample size increase, final Z test is

-                     **Z-statistic          Z-statistic**
**                    ****of 40 patients    of 60 patients**
- With such data-dep ‘n’ increase (100  200), one possible way of adjustment is use of adaptive test [e.g., Cui, Hung, Wang (1999) ]

**Z of 160 patients**

**Lan (2001 FDA/CDER/OB Minisymp, 2002 FDA/ASA wkshop,**
**         ****2002 ICSA Symposium)**

## U and Z are standard normal under  H0

**U and Z are standard normal under  H****0**
**If U > ****1.96**** , then conclude  ******** > 0  **
**  ****Significance level =  0.025**
**U more powerful than Z  w/o increasing ‘n’**

**Estimation & CI for  ******** ---**
**  ****Lawrence & Hung (2002, ENAR talk)**
**    **Construct consistent estimator and valid CI for  
-    Adaptive test  U  is Z-ratio of the consistent
-    estimator
**Jennison & Turnbull (2001, 4th International Meeting on Methods in Biopharmacy)****   **

##    Sample Size Re-estimation

-    Sample Size Re-estimation
-         (group sequential trial)
- Plan to have n =100 subjects/group to detect
-  = 0.46  at   = 0.025  and power  90%
- Use Lan-DeMets procedure with O-F type -spending function for monitoring
- After 50 subjects per group contribute data,
- the estimate of    suggests to increase sample size to a total of  m = 200/group  for detecting
- new assumed * = 0.325 with power  90%
-  = 200/100     b = (-0.5)/(1-0.5) = 3

## Suppose that an interim analysis will be performed

- when 125 subjects/group contribute data.
- The corresponding information time  t  on the original time scale is
-  t = {(125 - 50)/3 + 50}/100 = 0.75
- Keep the initially chosen O-F -spending function
- unchanged. The critical value for the original Z test (w/o ‘n’ increase) at  t = 0.75  is   C0.75 = 2.36.

## The CHW adaptive test at  t = 0.75  is

- The CHW adaptive test at  t = 0.75  is
-     **  **
**                   ****Z test of 1st 50       Z test of the additional**
**                   ****subjects/group       75 subjects/group**
** **
- If U0.75 > 2.36 , then stop the trial and conclude
- that  H0 is rejected.  If the trial continues to the end, the final** **adaptive test is

- U1.0 > 2.01  reject H0

##    Sample Size Re-estimation

- Adaptive test  U:
- - has type I error rate attained at the targeted level and can have a large power increase
- - uses the initially chosen rejection boundary and so its implementation is very easy
_**Note**_**:** The above discussions are based on asymptotic  (i.e., ‘sufficiently large’ sample size) theory
** **

**Notes:**
In that project, a simple approach was proposed to properly adjust .....

This adaptive test has the type I error rate maintained at the desired level and its power can be substantially improved as a result of sample size increase.  The test can be implemented very easily in the sense of preserving the flexibility of interim analysis schedule Lan and DeMets procedure has.

## Adaptive test  U: 

- - reduces to the standard fixed-sample (sequential or non-sequential)  Z  test if sample size is not changed.
- - uses ‘fixed weights’ completely determined by the initially selected fixed-information design for the component Z statistics.
- - validity (type I error rate) holds so long as ‘n’ modification rule (e.g., maximum sample size allowed, effect size threshold for ‘n’ increase, etc.)  does not depend on future data

**Notes:**
In that project, a simple approach was proposed to properly adjust .....

This adaptive test has the type I error rate maintained at the desired level and its power can be substantially improved as a result of sample size increase.  The test can be implemented very easily in the sense of preserving the flexibility of interim analysis schedule Lan and DeMets procedure has.

- The adaptive test  U  has another look using a combination of p-values from the incremental group data **[Lehmacher & Wassmer (1999), Brannath, Posch & Bauer (2002)]**
** **

**Notes:**
In that project, a simple approach was proposed to properly adjust .....

This adaptive test has the type I error rate maintained at the desired level and its power can be substantially improved as a result of sample size increase.  The test can be implemented very easily in the sense of preserving the flexibility of interim analysis schedule Lan and DeMets procedure has.

## Sample size re-estimation criterion

- 1)  based on the observed ******s** at time  s, re-estimate sample size such that conditional power
-    CP(******s**)
- = Pr{U rejects H0 at the end | **Z****s** , ****** ****= ********s**}**= 1-******
**     ****Cui, Hung and Wang (1999, Biometrics)**   ** **
**2) CP(*********) = 1-********  at new assumed effect size ********* **
**     ****Lawrence (2002, in review)**
**3) Unconditional power at 1- ******** via simulation**
**     ****Cui (2002, Encyclopedia of Biopharm. Statistics)**

**Notes:**
In that project, a simple approach was proposed to properly adjust .....

This adaptive test has the type I error rate maintained at the desired level and its power can be substantially improved as a result of sample size increase.  The test can be implemented very easily in the sense of preserving the flexibility of interim analysis schedule Lan and DeMets procedure has.

## Some other sample size re-estimation procedures

**Bauer & Köhne (1994, Biometrics)**
**Proschan & Hunsberger (1995, Biometrics)**
**Lan & Trost (1997, ASA Proceedings)**
**Fisher (1998, Stat. In Med.)**
**Shen & Fisher (1999, Biometrics) **
**Lehmacher & Wassmer (1999, Biometrics)**
**Müller & Schäfer (2001, Biometrics)**
**Liu & Chi (2001, Biometrics)**
**Brannath, Posch & Bauer (2002, JASA)**
**Li, Shih, Xie & Lu (2002, Biostatistics)**

## Issue of Estimation

**Estimation** following data-dependent sample size change is an important issue, particularly the effect estimate may be used to plan future superiority trials or active-control non-inferiority trials.

## Issue of statistical efficiency

_**Jennison & Turnbull (2001, 4th International Meeting**_
_**on Methods in Biopharmacy)**_
-     Fixed sample size design considering the anticipated ‘n’ change rule in sample size planning may give greater power with less average sample size than adaptive design
_**Tsiatis & Mehta (2002, Biometrika?)**_
**    **Better efficiency of standard group sequential design using likelihood ratio test over adaptive design

## Example 1   

- D1)  Plan  n  to detect = at =0.025, =0.10
-         At  t = 0.5, observe 0.5
-           0.5  -0.27  accept H0 and stop the trial
-              [  Pr( 0.5/ < -0.27 |   0.6 )  0.05  ]
-           0.5    continue the trial w/o change
-         -0.27 < 0.5 <   increase  n  to
-                 m = min{ (/0.5)2n , Nmax=4n }

##                       Pr{rej H0}           (Ave. SZ)/n

**                ****---------------------    -------------------**
**       ********/********    D0      D1                     D1        **
**  ****------------------------------------------------------**
**        ****.0    .025    .025                  2.848    **
**        ****.1    .051    .073                  2.986    **
**        ****.2    .095    .180                  3.048    **
**        ****.3    .162    .357                  3.032    **
**        ****.4    .254    .569                  2.941    **
**        ****.5    .367    .754                  2.789    **
**        ****.6    .494    .873                  2.590    **
**        ****.7    .621    .935                  2.363    **
**        ****.8    .737    .965                  2.127    **
**        ****.9    .831    .980                  1.898    **
**      ****1.0    .900    .989                  1.690    **
**  ****---------------------------------------------------------------**
**D0: plan fixed sample size ‘n’ for ******** = ******** & use  Z**
**D1: use CHW       ** **                  **

## D2)  Use fixed sample size design by planning

-         sample size  n*  to detect  0.635  with same
-          and , in anticipation of use of the above
-        ‘n’ modification rule

##                       Pr{rej H0}            (Ave. SZ)/n

**                ****----------------------    ------------------**
**       ********/********    D0      D1       D2         D1      D2 **
**  ****------------------------------------------------------**
**        ****.0    .025    .025     .025      2.848   2.480**
**        ****.1    .051    .073     .074      2.986   2.480**
**        ****.2    .095    .180     .174      3.048   2.480**
**        ****.3    .162    .357     .334      3.032   2.480**
**        ****.4    .254    .569     .533      2.941   2.480**
**        ****.5    .367    .754     .723      2.789   2.480**
**        ****.6    .494    .873     .865      2.590   2.480**
**        ****.635           .900     .900      2.513   2.480**
**        ****.7    .621    .935     .947      2.363   2.480**
**        ****.8    .737    .965     .983      2.127   2.480**
**        ****.9    .831    .980     .996      1.898   2.480**
**      ****1.0    .900    .989     .999      1.690   2.480**
**  ****---------------------------------------------------------------**
**D0: plan fixed sample size ‘n’ for ******** = ******** & use  Z**
**D1: use CHW      D2: use fixed sample Z test** **                  **

## In this example, by properly planning in anticipation of the ‘n’ adaptation rule, fixed sample size design can have advantage in statistical efficiency.

- But the adaptive design competes reasonably well but has flexibility.
- e.g., use different ‘n’ modification rule
-    including interim change Nmax, futility
-    rule, etc.
** **

## Moreover, one can adjust sample size such

- that conditional power at new assumed  *  given
- the past sample path achieves a desired level.
- * depends on obs. effect size  t , n, Nmax,
- external data, etc. and usually cannot be known
- at the initial design stage.
- No way to compare such adaptive design with
- fixed sample size design in terms of statistical
- efficiency, unless * is assumed to be known
- before interim examination of sample path

## Even if * is known prior to the start of the trial, 

- adaptive information design with the test  U  can still compete reasonably well.
- Example 2
- E1)  Plan  n  to detect = at =0.025, =0.10
-    At  t=0.5, observe 0.5
-    Determine new sample size  m  such that
-    CP(*) = 0.90  for the new assumed  * = 0.6
-    Set   n  m  Nmax = 4n
-    0.5  -0.27  accept H0 and stop the trial

## E2)  Suppose that * = 0.6  is known from the start.  So plan  n*  to detect  =*  at =0.025, =0.10.

- Do not change  n*  throughout the trial.
- Use the conventional  Z  test.

##                       Pr{rej H0}            (Ave. SZ)/n

**                ****----------------------    ------------------**
**       ********/********    D0      E1       E2         E1       E2 **
**  ****------------------------------------------------------**
**        ****.0    .025    .025     .025      2.694   2.778**
**        ****.1    .051    .070     .078      2.805   2.778**
**        ****.2    .095    .169     .190      2.844   2.778**
**        ****.3    .162    .334     .367      2.809   2.778**
**        ****.4    .254    .541     .580      2.718   2.778**
**        ****.5    .367    .736     .771      2.582   2.778**
**        ****.6    .494    .871     .900      2.408   2.778**
**        ****.7    .621    .942     .966      2.214   2.778**
**        ****.8    .737    .974     .991      2.018   2.778**
**        ****.9    .831    .988     .998      1.830   2.778**
**      ****1.0    .900    .994   1.000      1.655   2.778**
**  ****---------------------------------------------------------------**
**D0: plan fixed sample size ‘n’ for ******** = ******** & use  Z**
-  **                  **

## In drug development

- Trials

- Trial 1

- Trial 2

- Trial 3

- Trial 3

**Phase II**

**Phase III**

-                                                     .........
- Data from former trial(s) may be used sequentially to adjust the plans for next trials

## The factors (e.g., cost, flexibility, ethics, etc.)

- for the total development program are more
- relevant than some statistical optimality criteria
- applied to individual trials
- Adaptive designs are naturally appealing for drug
- development.
- Adaptation allows proper combination of individual trials in their entirety or in part.
- Planning for adaptation is important [Bauer et al (2002, Method Inform Med)].

##    Operational Conduct Issues

-  Sample size re-estimation based on unblinded
-  data might open rooms for operational biases.
-  During adaptive change, only unblind data that
-  are necessary to be unblinded in order to avoid
-  operational bias.  Standard Operation Procedure
-  (SOP) must be in place in the protocol and trial
-  conduct must comply with the SOP.

##      Operational Conduct Issues

- Sample size re-estimation based on unblinded
- data might open rooms for multiple analyses that might lead to more protocol amendments and other changes of design elements. These types of changes or amendments (potentially driven by current data) may introduce problems in the  interpretation of the results. Care is needed for
- such designs.

##      Operational Conduct Issues

**But ....**
-   Most of the operational conduct issues related to sample size re-estimation based on unblinded data are also encountered in traditional designs with interim analyses

##     Operational Conduct Issues

- Recommend that sample size modification **if needed** be performed by an independent third party that has no conflict of interest issue.
- What took place after sample size (or any design) modification needs to be documented .

##                Summary

- Standard fixed-information design is always good. The statistics have good statistical properties.
- Depending on the applications, this design needs to permit sample size adjustment when it falls short, e.g., in studying endpoints for which
- - prior data are not very informative to provide
-   reasonably good educated guess about effect size
- - minimum clinical meaningful effect not available

##                   Summary

- Conventional design properly adapted for sample size adjustment can be useful with
- - proper planning to avoid any operational conduct
-   change that may lead to bias
- - proper adjustment of statistical analysis method
-     estimation issue needs attention and research
- - benefit/cost/risk assessment of the adaptation for
-    each specific application

##                   Summary

-    Try to get a reasonable    as best as one can and
-    plan sample size
-                                      **** ****
-     Examine the interim sample path with possibly
-     external data to determine  *
-       If  * is substantially smaller than , then
-       - may increase sample size
-       - may elect to stop hopeless trial
-       - adjust statistical inference

##                   Summary

- Fully adaptive design with proper planning is very attractive in development of medical product
- - change sample size or randomization allocation
- - change study hypothesis (e.g., superiority vs. non-
-    inferiority or equivalence)
- - change test method
- - change the primary endpoint from one pre-
-    specified endpoint to another pre-specified one
- - drop futile or unsafe treatment arms
- and more......      More work to do 