# Presentation: 737525

## A Flexible Multiplicity Adjustment Approach and Its Application to Evaluating Several Independently Conducted Studies

- Qian Li and Mohammad Huque

## Abstract:

- 	Usually, in applying market approval of a new drug, more than one similarly designed clinical trials are conducted to support efficacy or safety claims. There has been no appropriate statistical approach that can evaluate evidence from many studies collectively. In this poster, we propose a new multiplicity adjustment approach to evaluate several similarly designed and independently conducted studies collectively. This multiplicity approach is flexible in selecting decision rules so that it is possible to find a decision rule which achieves the optimal power for a specified alternatives and a certain decision error. To control the decision error, a concept of overall hypotheses is proposed which are the combinations of individual study's hypotheses. As the number of studies increases, we could have many choices of overall hypotheses. Using this new approach, the maximum type I error can be derived under any overall hypotheses. A couple of examples are presented to illustrate the application of the method in evaluating several independently conducted studies.

## Outlines

- Motivation
- The flexible multiplicity adjustment approach for the case of two studies
- overall null and alternative hypotheses
- overall type I error or decision error
- power characteristics
- The general case of the multiplicity approach
- Examples
- Conclusions

## Motivation

- 	In many new drug application (NDA), usually more than one controlled, similarly designed clinical trials are conducted in order to provide convincing evidence for efficacy and safety claims. Some of the trials may show statistical significance, while some may not. How to evaluate the evidence from several independently conducted studies collectively can be a challenge statistical issue. There has not been a satisfactory method dealing with this problem.
- 	For a hypothetical example, when ten similarly designed trials are conducted, if two studies are statistically significant at 0.025 and the rest approaching to this significance level, this suggests convincing evidence. While if two studies are highly significant and the rest of the studies are far from statistically significant, this may suggest problematic evidence.
- 	For decision making, we need certain measurement for decision error. How should we evaluate the decision error in such settings? A new method is proposed to answer the questions.

**Notes:**
Problems:

counting the number of studies that are significant can be problematic. What if half studies are significant and half studies’ p-value close to 0.5? What if two studies are very significant and the rest are very close to significant and what if two studies are very significant and the rest studies’p-value are close to 1. The counting does not seem to provide internal consistency and no consistent standard to control decision error.

Combining studies together lose the insight information. 

## Define Terminologies

- 	The following terms will appear frequently in this poster, for convenience, they are defined here:
- _Positive study_: a study has true individual alternative
- _Negative study_: a study has true individual null
- _Successful study_: p-value is statistically significant at level 
- _Failed study_: p-value is not statistically significant at level 

## The New Approach - the case of two studies:

- 	This new approach can be written into the following formula
- PH0(p(1)1 p(2)2) ’
- 	where
- p(1) and p(2) are ordered p-values of p1 and p2
- where 1  2  1 are p-value cut points
- ’ is the decision error or overall type I error
- decision rule: if the minimum p-value is less than 1 and the maximum is no larger than 2, overall null H0 will be rejected at level ’.

## Overall hypotheses for the case of two studies

- Individual hypotheses for each studies
- Study 1      H01: 10   vs     HA1: 1>0
- Study 2	 H02: 2  0   vs    HA2: 2>0
- Where 1 and 2 represents treatment difference.
- Two overall hypotheses
- (1)		H01/2:   H01 H02 (intersection)
- 				vs
- 			HA1/2:   HA1 HA2 (union)
- (2)		H02/2:   H01 H02  (union)
- 				vs
- 			HA2/2:   HA1 HA2 (intersection)

## Two studies - decision error for H01/2:   H01 H02

- 	For such an overall null hypothesis, the null space is the third quadrant of coordinator 1 and 2.
- 	The maximum decision error can occur at (1=0, 2=0) and 1  and 2 satisfy 21 2 -12’
- 	More than one set of 1  and 2  can be found to satisfy the above inequality. For ’=0.05,
- if 1 = 0.025 then 2 =1;
- if 1 = 0.05 then 2 =0.525. ..

## Two studies - decision error for H01/2:   H01 H02

- 	The rejection regions for the two sets of cut points are shown in the shaded areas:

- 0.525

- 0.525

- 1=0.050
- 2=0.525

- 1=0.025
- 2=1.000

- 1

- p1

- p1

- p2

- p2

- 0.05

- 0.05

- 1

- 0.025

- 0.025

## Two Studies - decision error for	 H02/2:   H01 H02

- 	For such an overall null hypothesis, the null space is all the area except first quadrant.
- 	The maximum decision error occurs at (1= , 2=0) & (1=0, 2=) and the p-value cut point satisfies 1 =2’.

- 1

- 2

- (0,0)

## Power characteristics for two studies:

- 	The power function can be written as the follows:
- y2= F1(1) F2(2)+ F1(2) F2(1)- F1(1) F2(1).
- 	For normally distributed test statistics with p-value cut point ,
- Fi()=1-(-1(1- )-i),
- 	where i  is the mean of the standardized normal distribution for alternative.

## Power curve for two studies for the null hypothesis H01/2:   H01 H02 when 1= 2=2.0 and ’=0.05. The test statistics are normal distributed with standardized means 1 and  2. The optimal power occurs at 1 = 0.065 and 2 = 0.417. The optimal power is 0.851. 

## Optimal power and p-value cut points for two studies with normally distributed test statistics and decision error ’=0.05.

## The new approach in general

- 	When there are more than two studies, the new method can be written as follows:
- 	PH0(p(1)1 p(2)2...  p(K)K) ’,
- 	Where
- K denote the number of studies
- where 1  2  ...  K  1 are p-value cut points
- p(1), p(2), ..., p(K) are ordered p-values of p1, p2, ..., pK
- ’ is the decision error based on such decision rule.

## Overall hypotheses can be many union and intersection combinations of the individual hypotheses.

- Individual hypotheses:
- H01: 1 0   vs    HA1: 1>0
- H02: 2 0   vs    HA2: 2>0
- ...
- H0K: K 0   vs    HAK: K>0
- Possible choice of overall alternative hypotheses:
- HA1/k: at least one study is positive
- HA2/k: at least two studies are positive
- ...
- HAk/k: all the studies are positive

- 	When the overall null hypothesis is H01/k: , which requires at least one study to be positive in the overall alternative, to control the decision error to be ’, the p-value cut points should satisfy the following inequality:

- 	When the overall alternative requires more than one study to be positive, HAm+1/k ,
- max overall type I error occur when m studies have power 1 to reject individual null
- i’s satisfy the following inequality
- 	and 1= 2= ... = m m+1.

## Power characteristics

- 	The general power function can be written as the following:
- 		yK=PHA(p(1)1 p(2)2...  p(K)K)
- 	It is tedious to write the power function when K>3, however, the power can be evaluated numerically. We can use the power the function to find the optimal p-value cut points to gain the optimal power constrained by the decision error inequality.

## In the next several pages, we present p-value cut points and power comparisons for different number of studies (K). The optimal p-value cut points are obtained based on the following facts:

- test statistics are normally distributed
- if individual study has power 90% to detect treatment difference at one sided =0.025 level, the standardized mean  needs to be 2.47.
- if individual study has power 17%, the standardized mean  needs to be 1.
- Decision criteria is to require at least two studies to be significant at level 0.025
- The overall decision error is listed in the following table:

## The following table lists the overall decision error for different number of studies. The decision error for HAK/K for different K are the same. 

## Optimal p-value cut points for K=3

- 			 HA1/3 		  HA2/3 		  HA3/3
- 1			0.025		0.025		0.025
- 2			0.025		0.025		0.025
- 3			0.350		0.159		0.086

## Optimal p-value cut points K=4

## Optimal p-value cut points K=5

## Power comparison

## Example 1

-  	Three identically designed phase III studies were conducted. For decision making, we use the overall alternative HA1/3  and decision error ’=0.0252. The three p-value cut points need to satisfy the following inequality
- 	 	61 2 3 -3 3 12 -3 1 22 +13 0.0252.
- 	To require at least two studies to be significant at level 0.025, it is easy to obtain the third p-value cut point 3 . The p-value cut points are 0.025, 0.025, 0.350.
- 	The observed p-values were 0.0065, 0.0125, 0.06, which satisfy the decision rule.

## Example 2

- 	In this example, six similarly designed studies were conducted for two indications. The overall alternative hypothesis is chosen to be  HA4/6 and the overall decision error to be ’=0.025. Since there are two indications, the overall decision error was split ’=0.0125. Again to require at least two studies to be significant at level 0.025. The p-values cut point satisfy the following inequality:
- 			64 56 -3 6 42 -3 4 52 +43 0.0125
- 	and . 1  2  3  4.
- 	The optimal p-value cut points are 0.025, 0.025, 0.048, 0.122, 0.516.
- 	The observed p-values were 0.002, 0.003, 0.01, 0.027,0.071 for indication 1and 0.004, 0.016, 0.027, 0.387, 0.472 for indication 2. As it can be seen that indication 1 satisfied the decision rule.

## Conclusions:

- We have the flexibility to choose p-value cut point using this approach.
- This approach allows us to calculate type I error under different overall null hypothesis.
- This approach allows us to control decision error and internal consistency
- At the same time, it allows certain degree of variability
- This approach has potential to be used in correlated data