# Presentation: 554638

## Analyzing Trials with Active Control Arms

- Non-Inferiority Analyses
- David Harrington
- Dana-Farber Cancer Institute

**Notes:**
I am David Harrington from the Dana-Farber Cancer Institute, a statistician member of ODAC.
The FDA has asked me to begin the meeting with some comments on statistical issues in the analysis of trials with active controls, or non-inferiority analyses.

## Disclaimers...

- No financial support from sponsor
- Expenses paid by FDA as a member of ODAC
- Will not discuss in detail or evaluate the analyses in the application, but will use some data for context

**Notes:**
I have no affiliation with the sponsor, and my expenses for this meeting will be paid by the FDA.
I am not going to speak about the application in general and will not evaluate the quality of the evidence for or against raloxifene. I will use some data from the application for context.

## Trials in Application

| Trial | Number of Participants | Population: Postmenopausal women | Primary Endpoint |
| --- | --- | --- | --- |
| STAR | 19,747 | High risk of breast cancer | Invasive Breast Ca |
| RUTH | 10,101 | With or at risk of coronary events | Coronary events, inv. Breast Ca |
| MORE | 7,705 | With osteoporosis | Vertebral fracture |
| CORE | 4,011 | With osteoporosis | Invasive Breast Ca |

**Notes:**
The application includes four studies examining raloxifene as a possible agent to reduce the risk of invasive breast cancer.  The trials have different designs and were done in slightly different subsets of post-menopausal women. Collectively, the RUTH, MORE and CORE studies were placebo controlled studies evaluating the effect of raloxifene in reducing the breast cancer, coronary events, or fractures. We will hear more about these trials from the sponsor and the FDA. 
I will focus on aspects of the analysis of the STAR Trial.

## Main Statistical Issue in Analysis of STAR

- Application contains non-inferiority (NI) analysis of Raloxifene vs Tamoxifen in STAR trial.
- Primary endpoint is invasive breast cancer
- Raloxifene is test agent
- Tamoxifen is an ‘active control’
- Analysis of active control trials uses information outside of current trial to infer effect of a study drug (raloxifene) vs placebo in absence of a direct comparison
- See Rothmann et al. (2003), Temple & Ellenberg (2000), many others...
- What questions should be asked about NI analysis?

**Notes:**
The STAR trial compared raloxifene with tamoxifen in the prevention of invasive breast cancer in post menopausal women at high risk for breast cancer as estimated by the Gail model.   
The efficacy analysis of the STAR trial is a non-inferiority analysis of raloxifene vs tamoxifen.  The primary endpoint is invasive breast cancer, raloxifene is the test agent and tamoxifen is the active control.
Clinical trials with active controls, instead of control arms that use a placebo or no treatment at all, are becoming more prevalent in cancer as more effective drugs are discovered and marketed.  Most often, these trials take the form of superiority trials, where investigators seek evidence sufficient to change the standard of care with treatments that are more effective than standards of care.  But trials with therapeutically active control arms are sometimes done to determine whether to adopt a drug or  that may be nearly as good as a current standard but has a more favorable side-effect profile or is easier to administer.  These non-inferiority trials are more difficult to interpret, and there are important questions that we should ask about them.
After some background, I will close this talk with a list of those questions.

## Tamoxifen vs Placebo: NSABP P1 Trial  Subset of 7,998 Women  50 years old

**Tamoxifen vs Placebo: NSABP P1 Trial  Subset of 7,998 Women ******** 50 years old**

- 1.6

- 1.4

- 1.2

- 1.0

- 0.8

- 0.6

- 0.4

- RR = 0.47 (0.33 - 0.66)

- Favors Tamoxifen

- Favors placebo

- Interpretation: T reduces the rate of invasive breast cancer incidence on placebo by 53% (confidence interval 34% to 67%)

- Relative Risk for Invasive Br Ca: Tamoxifen / Placebo

**Notes:**
Tamoxifen value as a breast cancer prevention drug was established in the NSABP trial P1 and three other trials.  This application relies largely on the NSABP trial P1 for historical data on Tam.  This slide shows the results of P1 in the subset of approximately 8,000 women at least 50 years old.  The population is not a perfect match for the STAR population of post menopausal women, but apparently menopausal status at the start of therapy was not reliable recorded in P1.
This slide shows the ratio of the incidence rate of invasive breast cancer on the tamoxifen arm to the incidence rate on placebo in this population.  Values to the left of one indicate that tam had a smaller rate of breast cancer than on placebo.
In this slide we see the effect of tam compared to placebo through risk reduction.  The RR of 0.47 estimated in the trial is a 53% reduction in the incidence rate of invasive breast cancer compared to placebo. This estimate is consistent with the 51% reduction in risk found in the entire P1 study population. The lines extending beyond the box show 95% confidence intervals for the RR.  Even with a conservative interpretation, the right most point in the confidence interval shows a RR risk of 0.66 and indicates a 34% reduction in breast cancer incidence rate.

## Tamoxifen vs Placebo: NSABP P1 Trial Subset of Women  50 years old

- RR = 2.12 (1.52 - 3.03)

- Interpretation: P increases the rate of invasive breast cancer incidence compared to Tam by 112% (confidence interval 52% to 303%)

- Relative Risk for Invasive Br Ca: Placebo / Tamoxifen

**Notes:**
This slide shows the same result, in the same population, but instead shows the ratio of breast cancer incidence rates with placebo in the numerator.  This might be thought of as the increased risk of breast cancer for women not treated with tam or any other breast cancer prevention agent.  
The relative risk of 2.12 shown here suggests the rate of breast cancer incidence on P was 112% higher on P than on T 
I am going to use this picture later when talking about the issues in the analysis of  R vs T. 
These results had to be considered at the time that STAR was designed. 

## Potential Designs for Evaluating Raloxifene

- A
- N
- D
- O
- M
- I
- Z
- E

- Raloxifene

- Tamoxifen

- Placebo

**Potential Designs for Evaluating Raloxifene**

- STAR designed as superiority trial R vs T

- 85% power if
- RR (R/T) < 0.67
- 95% power if
- RR (R/T) > 1.56

- Observed RR = 1.02 Application has non-inferiority analysis of STAR

**Notes:**
At the time the STAR trial was initiated, there were several possible designs for evaluating Raloxifene.  
A superiority trial of raloxifene vs tamaxofen would seek to establish that raloxifene was either superior or inferior to tam.
A non-inferiority trial of R vs T would seek to establish that R provided much of, if not all of, the benefit of T.
A placebo controlled trial of R vs P, arguably a difficult trial to do in the face of the P1 results, would give a direct measure of the value of R compared to no treatment. 
A three arm trial of R vs T vs P would have provided a direct measure of the efficacy of R or T vs placebo in the same population and allowed a direct comparison of the relative efficacy of the two drugs.
The NSABP chose a superiority design for STAR, with 85% power to detect a relative risk of  0.67, R to T, or a 33% reduction in breast cancer incidence on R.  The NSABP also included  calculation of the likelihood of declaring R inferior to T it the relative risk of R to T exceeded 1.56  The value 1.56 indicates a 56% increase in breast cancer on R vs T, exactly half the 112% increase on P vs T.  The NSABP used this power calculation to show that if R was less that half as effective as T, the STAR trial would be likely to conclude that R was in fact inferior.
The observed relative risk in STAR, R compared to T, was 1.02, indicating the in the STAR population, the invasive breast cancer incidence rates in the two groups were nearly identical.
The application from the sponsor contains a non-inferiority analysis of STAR, an analysis that seeks to show that while R may be no better than T, and could be less effective, it compares sufficiently well to T to make it a viable option as a breast cancer prevention agent.
What are the elements of this NI analysis?

## Tamoxifen vs Placebo: NSABP P1 Trial Subset of Women  50 years old

- 2.2

- 2.0

- 1.8

- 1.6

- 1.4

- 1.2

- 1.0

- 0.8

- 0.6

- 2.4

- 2.6

- RR = 2.12 (1.52 - 3.03)

- Favors Tamoxifen

- Favors Placebo

- Relative Risk for Invasive Br Ca: Placebo / Tamoxifen

- NI margin: 50% of Active control effect retained. 56% increased risk on P

-  1.84

- 1.28

- RR = 1.56

**Notes:**
Here is the earlier slide showing the increased risk of invasive breast cancer on placebo compared to T.  The goal of a NI analysis is to show that a test agent does not lose `too much’ efficacy compared to an established agent.
Values to the right of 1 indicate I increased risk of invasive breast cancer on placebo, compared to Tam.   The STAR trial replaces placebo by R; values to the right of 1 would indicate that R looked more like a placebo compared to T, that is, lost efficacy.
In a non-inferiority design, trial investigators would set a non-inferiority margin (say 1.56), take as the null hypothesis the assumption that the new test agent was inferior, and power the trial to detect a non-inferiority margin that was smaller than the boundary value given by the null hypothesis.
There is considerable debate on how to set NI margins.  In the early active controlled trials, NI margins were set at a fixed value, often 1.25, allowing up to a 25% increased risk on the test agent.  Current thinking puts NI margins a specified fractions of the established effect.  1.56 puts the NI margin at 50% of the T vs P effect, 1.28 retains 75% of the T effect, 1.84 loses 75% of the T effect, or retains 25% of the T effect.
Despite the ambiguity in specifying a NI margin, there are two valuable reasons for setting these margins at design stage:  the trial can be adequately powered to detect non-inferiority of the test agent, and, perhaps more importantly, pre-specification prevents the increased false positive rate that can arise in the post hoc specification of hypotheses.
In the absence of a NI design, a NI analysis can be done on a trial as long as the definition of the NI margin has not been suggested by data in the trial at hand. I will say more about these analyses in a bit.

## Possible Outcomes of Active Control Trials

- Favors Active Control

- Favors Test Tx

- Relative Risk: Test Treatment / Active Control

- NI margin: 50% of Active control effect retained. 56% increased risk on P

- Placebo vs Active Control: RR = 2.12, with C.I.

**Notes:**
This slide will show possible outcomes of active control trials.  As in the previous slide, values to the left of 1 indicate a reduced risk on the test treatment, and so favor the test treatment, and values to the right indicated indicate increased risk on the test treatment, or favor the active control. For context, I have marked the placebo vs active control effect at 2.12 (the value in the subpopulation of older women in  P1), indicated its confidence interval with these orange bars, and marked a NI margin at 50% of this effect, and showed the margin of error that the NI effect inherits from the uncertainty in the estimate of the active control effect.
In the first two trials, you don’t need a weatherman to tell you which way the wind is blowing.  The first indicated superiority of the test agent compared to the active control, the second indicates inferiority.
The third trial requires careful interpretation.  In this trial, the test treatment has a slightly higher risk  ....

## Goal of Active Control (NI) Analysis

- How would test treatment compare to placebo, had placebo been present in the trial?
- Estimate effect of T compared to active control (C) in current trial (T vs C)
- Use data from previous trials to estimate effect of P vs C, along with a margin of error for that effect
- Combine these estimates to evaluate putative effect of  T vs P
- Sometimes done by estimating the range of percent retention of the P vs C effect consistent with data (confidence interval for NI margin)
- If T has fewer side effects than active control, T may be useful even if not as effective as C

**Notes:**
In a non-inferiority analysis, the goal is to infer how both the test treatment and the active control would have performed against placebo, had placebo been present in the trial.  Broadly speaking, the steps of the analysis are presented here. These steps include
How would test treatment compare to placebo, had placebo been present in the trial?
Estimate effect of T compared to active control (C) in current trial (T vs C)
Use data from previous trials to estimate effect of P vs C, along with a margin of error for that effect
Combine these estimates to evaluate putative effect of  T vs P 
Sometimes done by estimating the range of percent retention of the P vs C effect consistent with data (confidence interval for NI margin)
If T has fewer side effects than active control, T may be useful even if not as effective as C

## Most important issue when evaluating NI analyses...extrapolation

- NI analyses are based on one or more previous trials and a current trial and use information gained in potentially different settings.
- More careful labeling is helpful...
- Current trial: T test treatment, C2 active control, P a putative (unobserved) Placebo
- Previous trial(s) C1 the same control, P1 the placebo

**Notes:**
NI analyses are based on one or more previous trials and a current trial and use information gained in potentially different settings.
More careful labeling is helpful...
Current trial: T test treatment, C2 active control, P a putative (unobserved) Placebo 
Previous trial(s) C1 the same control, P1 the placebo

## Assumptions in NI Analysis

- T vs C2 well conducted.
- P vs C2 =  P1 vs C1 if placebo had been present in current trial  (assay sensitivity)
- P1 vs C1 effect has not changed since prior trials, or any change can be modeled
- Uncertainty in P1 vs C1 effect can be estimated
- Both within and between trial variability relevant
- P1 vs C1 sometimes estimated from meta-analysis of prior trials.
- Clinically relevant non-inferiority margin was specified before analysis
- Assumptions all used in inference

**Notes:**
T vs C2 well conducted.
P vs C2 =  P1 vs C1 if placebo had been present in current trial  (assay sensitivity)
P1 vs C1 effect has not changed since prior trials, or any change can be modeled
Uncertainty in P1 vs C1 effect can be estimated
Both within and between trial variability relevant
P1 vs C1 sometimes estimated from meta-analysis of prior trials.
Clinically relevant non-inferiority margin was specified before analysis
Assumptions all used in the inference

## Some Questions to ask about NI analyses

- Is the claim of NI supported by a biological rationale?
- Might the effect of Active Control (vs placebo) have been different in current trial?
- Changes in administration of agent
- Differences in populations using the drug or in endpoint determination
- Has long term follow-up changed the thinking of the value of the active control?
- Does the analysis use the best available historical data on active control to estimate both treatment effects and uncertainty in the estimate?
- Justification for omitted trials...

**Notes:**
Is the claim of NI supported by a biological rationale?
Might the effect of Active Control (vs placebo) have been different in current trial?
Changes in administration of agent,
Differences in populations using the drug or in endpoint determination
Has long term follow-up changed the thinking of the value of the active control?
Does the analysis use the best available historical data on active control to estimate both treatment effects and uncertainty in the estimate?  
Justification for omitted trials...

## Questions...

- Is an estimated NI margin clinically relevant? Was it specified in advance of the analysis?
- Is a reduced therapeutic effect for the test agent balanced by other benefits?
- What is the margin of error (confidence interval) in the estimate of possible loss of efficacy?
- Are results consistent across related endpoints?
- As in all trials, treatment effects measured in NI analyses are estimates of population effects, not predictions of efficacy for individuals
- Is there a clear signal to the treating clinician on when to use the active control vs the new treatment?

**Notes:**
Is an estimated NI margin clinically relevant? Was it specified in advance of the analysis? 
Is a reduced therapeutic effect for the test agent balanced by other benefits?
What is the margin of error (confidence interval) in the estimate of possible loss of efficacy?
Are results consistent across related endpoints?
As in all trials, treatment effects measured in NI analyses are estimates of population effects, not predictions of efficacy for individuals
Is there a clear signal to the treating clinician on when to use the active control vs the new treatment?