# Presentation: 877900

## Evidence and P-Values Given Multiple Endpoints, Multiple Comparisons and Multiple Studies

***Huque@cder.fda.gov***
- Division of Biometrics III, FDA
- 9201 Corporate Blvd, Rockville, Maryland
- (*Slides to be presented on December 31, 2000, at the Joint Statistical Meeting, New Delhi, Sponsored by the International Indian Statistical Association (IISA)*
***Disclaimer: ***Views expressed here is of the presenter and not necessarily of the U.S. Food & Drug Administration

## Outline

***Part I:*** Regulatory concept of evidence (for adequate and well-controlled clinical studies)
***Part II:*** Multiplicity issue when “lumping”/combining clinical trials
***Part III:*** Multiplicity in clinical trials given multiple endpoints and treatment groups
- Growing use of multi-step procedures and closure   principle (some examples)
- Issues and concerns
- Final comments

**Notes:**
My presentation is divided into 3 parts:

First, I will quickly go over a few slides to give a feel about the regulatory concept of  evidence, so that, when we talk about issues and concerns regarding multiplicity and p-values, we can put things in percpectives

Then, I will talk about concerns regarding multiplicity when lumping or combining trials. By lumping, I mean ....

Also, I see growing use of multistep procedures for handling multiple comaparison/multiple comparison issues. I will share with you some concerns and issues regarding this, and will go over 3 examples briefly.

Slides on Example 2 & 3 has been c. revised, please ignore those in the handout. I will be glad to email you revised ones if you need them.

And then some final comments

## Approval of New Drug Applications (NDAs) Based On:

- “Substantial evidence of efficacy” for the intended use of the drug
-  Benefit outweighs the risk:   benefit-risk analysis

## Federal Food, Drug, and Cosmetic Act Section 505 (d)

- “Substantial Evidence”
- Evidence consisting of ***adequate and well-controlled*** investigations
- Includes clinical investigations by experts qualified by scientific training and experience to evaluate the effectiveness of the drug involved
- Fair and responsible evaluations/conclusions by such experts
- The drug will have the effect it purports or is represented to have under the conditions of use prescribed

## CFR 21 §314.126 (a)-(e) : Adequate and Well-Controlled Studies

***Key Characteristics:***
- clear statement of the objectives
- valid design and appropriate control
- proper selection of patients having the disease or condition
- proper treatment assignments (randomization)
- adequate measures to minimize bias (w.r.t. subjects, 	      observers and analysis of data)
- well designed and reliable methods of measuring responses
- The design is such that an analysis of data is possible that will be adequate to assess the effects of the drug

***NDA ***

- New similar indication of an approved drug

- Change in dosage

- Change in formulation

- Pediatrics

- Generics

***Evidence for NDA (New Drug Application)***
***vs. post-NDA submissions may vary***

##  An Example: Type I Error InflationWhen 2 Studies are Combined

- Consider  2 placebo controlled trials, equal sample size, equal allocation, equal variance, and endpoint compared by mean
***Null hypothesis***: assume global null, i.e, H0:1=0, 2=0
***Test statistics for separate analyses***: T1  Z1- , T2   Z1-
***Test statistic pooled analysis***: T=(T1 + T2)/2   Z1-2

**Notes:**
Next 2 slides give an example to show the inflation of the type I error when 2 studies are combined. This example is by one our colleage Dr. Laura Lu.

* Consider 2 placebo controlled trials of ....
* Null hypothesis is a global null, no treatment effect in either    trial.
* Tests for single trials are based on statistics T1 and T2 each is    an alpha level test ...
* Say, the sponsor wants to do a pooled analysis if one or both      tests are not rejected. The pooled test statistic is given by T...     and say the  pooled analysis is an alpha square level test 

## An  Example of Type I Error Inflation: Pooling 2 Studies (After Testing for Single Studies )

- Type I error inflation in pooled analysis - dashed area

- (T1 + T2)/2 = Z1-2

- T1 = Z1-

- T2 = Z1-

**Notes:**
Grapically, the region of rejection for the first study test based on T1 is above this horizontal line, and the region of rejection for the second study test based on T2  is to the right of this vertical line.

The overall type I error of the two independents tests given the global null is alpaha square

The region of rejection for the pooled test is above this 135 degree line.

The shaded area shows an infaltion of the type I error under the global null by the pooled test

## Uncontrolled Decision Error Under Partial Null 

***Scenario:*** Consider 5 trials with  2 treatments active and placebo; n=100
- patients randomized to each arm. For i=1, ..., 5 studies, control response  rates
- 0i same  for all trials, and i = ( Ti  - 0i ) are true difference in response  rates.
- _______________________________________________________________
- Control    True Treatment    Prob. at least 2 Studies        #Prob. MA (Fixed Eff.)
- Rate          Differences          Significant at  =.025          Significant at  =.05
- _____      _____________    _________________         ___________________
- 10%        (0, 0, 0, 0, 10%)                 .0514                            .1119
- 10%        (0, 0, 0, 10%, 10%)            .3032                            .3814
- 50%        (0, 0, 0, 0, 10%)                 .0312                            .0965
- 50%        (0, 0, 0, 10%, 10%)            .1200                            .1606
- _______________________________________________________________
- # 10,000 simulated fixed effect meta-analyses. P-Value for each MA by MH-test

**Notes:**
Generally, if there is no safety concern, regulatory decision is made in favor of the efficacy of a drug,  if at least  2 adequate and well-conrolled studies  show significant result in favor of the drug, say,  with a 1-sided p-vlaue of less than  .025 in the right direction.

A naive application of this decison rule is problematic without some kind of consistency restriction on the other trilas for which the results are not significant in favor of the drug.

 This slide shows this example. Consider 5 trials.....

## Some Recent Work

- Such some of these problems can be alleviated by putting some consistency constraints on the p-values of the studies which do not show significance at the desired level 
- Methodology: for multiple studies obtain optimal “cut off points” for p-values that protect the overall ,  and at the same time give some assurance about the consistency of results across studies for the overall result to be interpretable.

**Notes:**
Dr. Qian Li has recently done some in this regard for controlling the decision error in the regulatory setting.

She has proposed some flexible designs on putting some consistency constriants on trials which dis not meet statistical criteria of p less than .025

basically, she proposes ......

## Controlling Decision Error Under Partial Null 

***Scenario:*** Consider 3 trials with  2 treatments active and placebo; n=100
- patients randomized to each arm. For i=1, 2 studies, control response  rates
- 0i same  for all trials, and i = ( Ti  - 0i ) are true difference in response  rates.
- _______________________________________________________________
- Control    True Treatment    Prob. at least 2 Stuides         #Prob. at least 2 Stuides
- Rate          Differences          Significant at  =.025          Significant at  =.025
- _____      _____________    _________________         ___________________
- 50%        ( 0, 0, 30%)                 .0492                                .0118
- 50%        ( 0, 0%, 20%)              .0415                                .0100
- 50%        ( 0, 0, 10%)                 .0151                                .0038
- _______________________________________________________________
- # and the 3rd study significant at  =.25

**Notes:**
This slide shows the idea behind the Qian's work.

If there is no constriant on the third study which does not show significane, then the decison error rate in column 3 is infalted 

However, if put a restriction of alpha=.25 on this 3rd study, the decison error is controlled

## Type I Error (Decision Error) for Partial Nulls (Multiple Endpoints Case)

- Each individual endpoint planned with power=.90 and =.025 (1-sided)
- _______________________________________________
- 	          T=2          ---------------- T=1 ------------------------
- r 		Rule 3/3      Rules  2/3       Rule 2/3*        Rule 3/3
- ________________________________________________
-  0            .020             .044               .006                  .001
- .4            .023             .046               .0196                .004
- .8            .024             .039               .035                  .009
- 1.0          .025             .025               .025                  .025
- _________________________________________
- *: 2/3  rule + the p-value for the 3rd endpoint < .15 (1-sided)

**Notes:**
The infaltion of the decision error also occurs in the multiple endpoint situations (e.g., for arthritis trials) when the clinical criteria is to show significance in 2 out of 3 endpoints

Suppose 2 of the 3 endpoints has effect, but one has no effect.   Then 3/3 rule does protect the decision error rat at alpha=.025.

However, if  only one endpoint has an effect and the other 2 are null, then the decison error is not controlled for the 2/3 rule at alpha=.025. But on putting, some cosnsistency constrian on the other  2 endpoints improves the situation

## Multi-Step & Closure Principle for Handling Multiple Comparison/ Multiple Endpoint Problems 

***Closure principle*** used in recent developments of multi-step procedures
- Refereces: Peritz (1970) Marcus et al (1976), Westfall et al (1999 SAS pub.), Hsu & Burger (1999), Hochberg & Tamhane (1987), Peter Bauer (96, 97, 98), and others
- Allows the use of -level test at each step to assure Type I FWE of 
- Easy to setup and apply
- However, there are issues and concerns

## Closure Principle (Peritz, 1970; Marcus et al,1976)

- Let W be the set of null hypotheses closed under intersection: wi, wj  W implies wi  wj  W
- Any null hypotheses w is tested and rejected at level   by means of a test  *if and only if*  all hypotheses w that are included in w ( w  w), and w  W, have been tested and rejected at level .
- The probability of making no Type I error rate by this procedure is at least (1-)

**Notes:**
The idea is simple

If you have a family of null hyportheses which is closed under intersection

Then a null hypothesis in this family is rejected by an alpha-level test procedure if and only if all sub-set null hypotheses of this hypothesis in that family is also rejected by the same alpha-level test procedure 

## Applications of the Closure Principle - 3 Clinical Trial Examples

_***Example 1***_*** - ***Tests for non-inferiority, statistical superiority, and clinical superiority - all in one trial -- Morikawa & Yoshida (1995, JBS)
_***Example 2***_*** - ***Tests for dose effects in a clinical trial with a control and K dose groups -- Hsu & Berger (JASA, 1999), Hochberg & Tamhane(1987)
_***Example 3***_***: ***Tests for treatment effects in a two-arm clinical trial with an active and a placebo group having two endpoints Y1 and Y2***  ***-- JSM 1999

## Example 1 - Level  tests for non-inferiority, statistical superiority, and clinical superiority - all in a single trial

- For a clinical trial with an active and a control treated groups, and
- 1 >0 and  2 >0 ,
- H03 = ( - 0)  - 1,   A3 = ( - 0) > - 1  (clinical non-inferiority)
- H02 = ( - 0)  0,   A2 = ( - 0) > 0             (statistical superiority)
- H01 = ( - 0)  2,   A1 = ( - 0) > 2       (clinical superiority)
- Define, for i=1,2,3,   H'0i =  ji H0i    and    A'i =  ji Ai .
- Then  H'03      H'02      H'01 .
- Therefore, a direct application of the closure principle makes
- the following test procedure a closed test procedure (next page).

## Example 1 (cont’d)

- Let  L= (Ÿ - Ÿ0) - t ;1- *** . ****S* ( 1/n + 1/n0) 1⁄2
***Step 1 - ***Reject  H'03 if {L > - 1} and go the next step, else stop
***Step 2 - ***Reject  H'02 if {L > - 1}  {L > 0} and go to the next step, else stop. Note that {L > - 1}  {L > 0} = {L > 0}
***Step 3 ***Reject  H'01  if    {L > - 1}  {L > 0 }  { L >  2 }

## Example 2 - Consider a trial with a control and K dose group with unknown means 0 , 1,  2, ...,  K

- For i=1, 2, ..., K, and 1> 0, 2 > 0, consider,
- H0i = (i - 0)  a,   Ai = (i - 0) > a ,   where,
- 	 a = - 1     (Setup for clinical inferiority trials)
- 	 a = 0         (Setup for statistical superiority trials)
-  	 a = 2       (Setup for clinical superiority trials)
- CTP using order statistics (next page)
- Hsu & Berger (JASA, 1999), Hochberg Tamhane (1987, Wiley)

## Example 2 (Cont’d)

- Let Ti =  {(Ÿi - Ÿ0) - a} /*** ***{*S* ( 1/ni + 1/n0) 1⁄2}, i=1,2, ..., K.
- Let T(1)  T(2)  ...  T(K)  be the K order statistics with critical
- values  T(1);1- , T(2);1- ....,   T(K);1- .
***Step 1:  ***Reject  H(0K) = ((K) - 0)  a  in favor of  A(K) =  ((K) - 0) > a   if   T(K) > T(K);1- and go to Step 2, else stop.
***Step 2: ***Reject  H(0(K-1)) = ((K-1) - 0)  a  in favor of  A(K-1)= ((K-1) - 0) > a   if   T(K-1) > T(K-1);1- and go to Step 3 else stop.  ...
***Step K: ***Reject  H(01) = ((1) - 0)  a  in favor of  A(1)= ((1) - 0) > a   if   T(1) > T(1);1-

## Example 3: Testing for treatment effects in a two-arm trial with an active and a placebo group having two endpoints Y1 and Y2 

***Null and alternate hypotheses - endpoint 1***
- 	H01 = (1 - 01)   a1,   A1 =  (1 - 01) > a1
***Null*** ***and alternate hypotheses - endpoint 2***
- 	H02 = (2 - 02)  a2,   A2  =  (2 - 02) > a2
***CTP:*** Reject H01 in favor of A1 by using a global test procedure (e.g., O’Biren’s GLS test) if  both hypotheses H01 and H01  H02  are rejected. Similarly for H02
***CTP:*** based on order statistics (next page)

## Example 3 (Cont’d)

- Define for i=1, 2 endpoints,
- 	Ti =  {(Ÿi - Ÿ0i) - a} /*** ***{*S* ( 1/ni + 1/n0i) 1⁄2}
- Let T(1)  T(2) be the order statistics for T1 and T2 and  T(1);1- ,
- T(2);1- be the critical values for these order statistics.
***Step 1: ***Reject  H(02) =  ((2) - (02))  a(2)  in favor of  A(2) =  ((2) - (02)) > a(2)   if   T(2) > T(2);1- and  go to Step 2, else stop.
***Step 2: ***Reject  H(01) =   ((1) - (01))  a(1)  in favor of  A(1) =  ((1) - (01)) > a(1)   if   T(1) > T(1);1-

## Example 3 (Cont’d)

***An Ad-hoc  Stepwise Procedure  ***
***Step 1: ***Reject  H01 =  (1 - 01)   a1  in favor of
- 	A1 =  (1 - 01) > a1   if   T1 > t1;1- and go to Step 2A. If
- 	 T1 > t1;1-    (  '>   go to Step 2, else stop
***Step 2A: ***Reject  H02 =   (2 - 02)   a2  in favor of
- 	A2 =  (2 - 02) > a2   if   T2 > t2;1-
***Step 2B: ***Reject  H02 =   (2 - 02)   a2  in favor of
- 	 A2 =  (2 - 02) > a 2   if   T2 > t2;1-*
- where * <  ;  *, ** are determined by considering correlation between the two endpoints and Type I FWE  

## Multi-step and Closed Testing ProceduresSome Issues & Concerns

- Directional errors (Type III errors): problem with two sided tests, but OK with one sided tests
- For closed testing procedures adjusted p-values can be calculated (Examples: Huque & Sankoh (1997), JBS; Westfall et al (1999 SAS pub.)). However, the raw and adjusted p-values may differ in rank ordering.
- Confidence interval? Sample size and power?
- Different global test procedures when closed for getting a specific endpoint result may give different results
- Tests, in a closed testing procedure, are generally correlated - bootstrap or  permutation techniques helpful

## Final Comments

- “Substantial evidence”  is key to decision making for NDAs
- In general, literature based meta- analysis has not been helpful in this regard. Interpretation of the p-value is a major problem.
- In combining studies, decision errors/type I errors need to be controlled by proper design and analysis
- Multi-step and closure principles in general have been helpful in solving multiplicity problem for clinical trials. However there are issues and concerns.
- Use of these methods, however, require pre-specification at the protocol stage