# Presentation: 181460

## Equivalence Testing

**Peter A. Lachenbruch**
**Director, Division of Biostatistics**
**CBER**

## Do Two Antigens Evoke Similar Responses?

**Do new antigen and standard antigen have the same mean response?**
**A challenge of each antigen is applied to two sides on the back of a subject.  **
**The application should be randomly determined**
**This assumes no systemic effects that could muddy the comparison**

## Two Antigens (2)

**After a period of time, the sizes of the wheals from each antigen is measured**
**To show the equivalence, we wish to show that the wheals are within a small limit of each other.**
**What is small?**
**What’s the correct method?**

## Two Antigens (3)

**Set the margin as a fraction (10% is often used) of the mean of the standard**
**Select an absolute value for the margin**
**this will often be based on previous experience, so may not differ much from the fraction.**
**Require the difference of the means to be within a margin of 0**

## Difference

**	****	****	****-0.3****	****	****0.3**
**  ****The heavy lines indicate a difference in mean level.  If the difference in means is in the heavy area, the means are different.  We want to show that the true mean difference is in the lightly shaded area.**

## Two Antigens (4) 

**Suppose we found the mean difference of the pairs and their standard deviation to be **

## Two Antigens (5)

**Concluding equivalence since t is “not significant” is an error because we can choose a small sample size and have no power to detect a difference.**
**Suppose the historically known wheal size was 3 cm, then 10% would be 0.3 cm**

## Two Antigens (6)

**We can compute a 90% confidence interval, and note if it is **_**entirely contained**_** in the equivalence region:**
**	****	****0.2±1.66*2.5/********100**
**	****	****(-0.215, 0.615)**
**This overlaps the region (-0.3, 0.3)**

## Confidence Interval

**Acceptable Region**
**                 ****-0.3          0.3**
**    ****-------------^----------^------------------**
**Observed Confidence Interval**
**                 ****-0.215                   0.615**
**    ****---------------^---------------------^------**
**  ****They don’t overlap.  We can’t conclude equivalence.**

## Two Antigens (7)

**Another method is to test the joint null hypothesis **
**H****01**: **μ****1** -** μ****2** **> δ  OR**
**H****02**: **μ****1** -** μ****2** **< -δ **
**By rejecting both of these hypotheses, we can conclude that | μ****1** -** μ****2****|** **< δ**

## Conclusions

**Discuss selection of margin**
**Sample size based on null hypothesis of **_**non-equivalence**_
**Use confidence interval or two one-sided tests**
**Show both ITT and PP analyses**
**Provide evidence of trial validity**
**design**
**efficacy of control**

## Concerns

**In comparative trials detection of  difference implies the trial had sufficient quality to detect it**
**In equivalence trials, sloppy design, execution, analysis move conclusions toward equivalence**
**Finding equivalence doesn’t mean that both treatments were effective**
**Sometimes a third, placebo arm is valuable**
**Some entry criteria let non-diseased patients in trial**

## Concerns (2)

**Some clearly effective drugs do not show effectiveness in each study**
**Population may be less responsive than in licensure study**
**Sample size may be too small to show difference**
**Must show that active control is superior to untreated control by a given amount **
**This is principal advantage over the historical control study**
**With large variability, it may be useful to include a placebo group**

## Concerns (3)

**Design - since much information is available, there is no excuse for poor design**
**Double blinding**
**Randomization**
**Inclusion/Exclusion criteria should be same as for trials of active comparator**
**Dosing and schedule for active comparator should be as licensed **
**Outcomes for active comparator should be similar to previous trials**

## Concerns (4)

**Paired or unpaired**
**Paired design can reduce variance substantially**
**If permanent change occurs (e.g., vaccination) a paired design is not possible**
**Analysis - which patients to include?**
**Intention to treat (ITT) - as randomized tends to obscure differences  - just what we don’t want**
**Per protocol (PP)- by treatment actually received - but can also lead to biases if there are unequal dropout patterns**

## Concerns (5)

**Useful to perform both ITT and PP analyses**
**Analysis may show:**
**Confidence interval lies entirely within the equivalence range - conclude equivalence**
**Some points of the confidence interval are outside the equivalence range - can’t conclude equivalence**
**It is possible for the two treatments to be **_**significantly**_** different, but the confidence interval lie within the equivalence range**

## Concerns (6)

**The test of the null hypothesis is not appropriate **
**A test of the relevant null hypothesis (H****0**:|******1** -** ********2**|>** ********) should be done**
**Even when TOST is used, the other concerns remain**

## Two Antigens (8)

**We may test that the difference is greater than 0.3 cm or less than -0.3 cm**
**We perform two one-sided t tests**

## Two Antigens (9)

**First we test the hypothesis that the mean difference is greater than 0.3.  If t****1**** < -1.66  reject the hypothesis (t****1**=-**0.4>-1.66, so we don’t reject)**
**We then test the hypothesis that the mean difference is less than    -0.3.  If **t2 **is greater than 1.66 we reject (t****2**=**2.0> 1.66, so we reject)  **

## Two Antigens (10)

**Since t****1**** does not reject H****01****:********d****>0.3  and t****2**** rejects H****02****:********d****<-0.3, we cannot reject the hypothesis of non-equivalence.**
**A 90% confidence interval is                 (-0.215,0.615)  which overlaps       (-0.3,0.3)**

## Testing for no difference

**The null hypothesis is that the two allergens are the same (H****0****: ********1**** = ********2****)**
**The alternative is that they are not.  H****a****:********1********2**
**Failing to reject the null hypothesis is **_**not**_** the way to show equivalence**

## No difference test

**If we fail to reject, this has not given much evidence that they really aren’t different**
**Study is not powered to show a difference?**
**Sloppy study conduct tends to show no effect**
**Wrong treatment assignment reduces difference**

## Equivalence

**The proper null hypothesis is H****0****:********1****-********2******** which says the difference of means is “too large”**
**The alternative is the converse.  We usually compute the power at H****0****:********1****-********2****=0**

## Equivalence model 

**This tests the null hypothesis that the treatments are different.  By rejecting it, we conclude the alternative: they are the **_**same**_**.  **
**Usually, the sample size requirements are a little larger than for a test of equality.  By rejecting the hypothesis of equality, we conclude they are **_**different**_**.**

## TOST=Two one-sided tests

**The null hypothesis is usually tested by performing two tests at the ******** level:**

## Equivalence

**	****	****	****   ****-0.3****	****	****0.3**
**   ****The heavy line is the region of equivalence.  If the mean difference is in this region, they are equivalent.**