# Presentation: 822210

##  Measuring Effectiveness in Trials of Acute Bacterial Sinusitis

**John H. Powers, MD**
***Lead Medical Officer***
***Antimicrobial Drug Development and Resistance Initiatives***
***Office of Medical Policy***
***Center for Drug Evaluation and Research***
***U.S. Food and Drug Administration***

## Introduction

**Regulatory and scientific history related to noninferiority trials and demonstration of effectiveness**
**History of discussions regarding NI trials in antimicrobials and acute bacterial sinusitis trials**
**Evaluation of historical evidence of magnitude of effects of antimicrobials in placebo controlled trials in acute bacterial sinusitis**
**Conclusions regarding current knowledge of evaluation of effectiveness and *****in vitro***** resistance on outcomes**

## Regulatory/Scientific History

**1938 – pre-market requirement for safety only (based on Elixir of sulfanilamide deaths)**
**1962 –requirement for demonstration of effectiveness to balance any potential harms of therapy (thalidomide)**
**President Kennedy sent recommendations to Senate Committee: ****“an undefined standard of ‘substantial evidence’ of effectiveness was inadequate in terms of assuring that drugs that reach the market have been shown to be effective for the claims made for them.”**
**Congress defines that only source of “substantial evidence” of effectiveness is “adequate and well-controlled trials”**
**1970 – FDA publishes regulations that provide criteria for defining “adequate and well-controlled trials”**

## Adequate and Well-Controlled

**Clear statement of objectives**
**Study design permits valid quantitative comparison with a control**
**Select patients with disease (treatment) or at risk of disease (prevention)**
**Baseline comparability (randomization)**
**Minimize bias (blinding, etc.)**
**Appropriate methods of assessment of outcomes**
**Appropriate methods of analysis**
**21 CFR 314.126**

## Regulatory/Scientific History

**1985 – recognition of issues with trials designed to show “similarity” rather than superiority of drugs**
**“****If the intent of the trial is to show similarity of the test and control drugs, the report of the study should assess the ability of the study to have detected a difference between treatments. **_***Similarity of test drug and active control can mean either that both drugs were effective or that neither was effective.***_** The analysis of the study should explain why the drugs should be considered effective in the study, for example, by **_***reference to results in previous placebo-controlled studies***_** of the active control drug. **
**21 CFR 314.126 (b)(2)(iv)**

## Regulatory/Scientific History

**Noninferiority trials attempt to:**
**rule out how much *****inferior***** a new treatment might be compared to an already proven effective treatment **
**AND ensure control drug effect relative to placebo is consistent under the conditions of the trial**
**2000 – ICH-E10 Guidance on *****Choice of Control Group and Related Issues in Clinical Trials***
**Section 1.5 describes information necessary to select noninferiority margin determined by analysis of *****historical evidence of sensitivity to drug effects***** (HESDE) **
**magnitude by which control drug may be reliably and reproducibly shown superior to placebo in previous superiority trials**
**Effect of control relative to placebo should be well-characterized and consistent from trial to trials so that effect in current trial consistent**

## Regulatory/Scientific History	

**Well-designed experiments have ****positive**** and ****negative**** controls to determine results causally related to intervention and not conditions of experiment (internal validity)**
**Lack of “negative” controls in noninferiority trials means they lack intrinsic measure of internal validity**
**Data on effect of control relative to placebo external to NI trial**
**Similar potential biases to historical controlled trials**
**Conditions of experiment (e.g. enrolling patients who do not have disease, timing of outcomes beyond natural history of disease, etc.) may change effect of control and affect conclusions regardless of margin chosen**
**Demonstration of noninferiority does not necessarily mean drug  has demonstrated effectiveness relative to placebo**

## Regulatory/Scientific History

**Selecting of appropriate margin of inferiority integral to design of noninferiority trials**
**“****In practice, the noninferiority margin chosen usually will be smaller than that suggested by the **_***smallest effect size of the active control***_** because of interest in ensuring that some clinically acceptable effect size (or fraction of the control drug effect) was maintained.”**
**ICH E-10, section 1.5.1.1., page 11**
**“****Effect size” refers to magnitude of benefit of active control drug relative to placebo from previous placebo controlled trials**

## Quantitative Comparison with Control

**Three criteria before one can perform noninferiority trial:**
***Quantitative *****assessment of effect of control drug relative to placebo based on data from previous trials:**
**Reliable, well-characterized and reproducible effect from trial to trial **
**Based on trials that are adequate and well controlled**
**Evaluation of all previous superiority trials**
**Takes into account variability of the data ( confidence intervals rather than point estimates)**
**   ****	****	****	****	****	****THEN**
**2. Selection of margin that is less than effect of control relative to placebo to preserve some benefit of control based on clinical judgment**
**	****	****	****	****	****AND**
**3. Maintenance constancy of the effect of the control from trial to trial**
**Similar definition of disease, endpoints, timing of endpoints**
**Changes in medical practice, adjunctive therapies, antimicrobial resistance**

## Definitions and Determining Margin

**M(1)**** defined as magnitude of benefit of active control compared to placebo as measured in current trial but determined from *****data***** in previous superiority trials**
**M(2)**** defined loss of effect of test drug compared to active control where drug is still considered clinically useful based on clinical *****judgment***
**Results of trial demonstrate noninferiority if success rates with test drug minus success rates with active control exclude M(2); ****T – C < M(2)**
**BUT this only shows that test drug is effective relative to placebo when M(2) is less then M(1); ****M(2) < M(1)**

## Non-Inferiority MarginsQuantifying the Effects of a Control Drug

**Difference in Point Estimates**

## Non-Inferiority MarginsQuantifying the Effects of a Control Drug

**Difference in Point Estimates**

## Non-Inferiority MarginsDetermining the Effect of a Test Drug

**Difference in Point Estimates**

## Non-Inferiority MarginsDetermining the Effect of a Test Drug

**Difference in Point Estimates**

## Non-Inferiority MarginsDetermining the Effect of a Test Drug

**current trial**

**control**

**test**

**past trials**

**0**

**100**

**80**

**success rates**

**control**

## History with Antimicrobials/ABS

**Several meetings addressing issues in noninferiority trials in study of antimicrobials**
**Need to evaluate data on each indication to determine margins or whether data to support noninferiority trials**
**Issues with selection of patients with disease**
**Issues with defining outcomes and timing of outcomes**
**Issues with analysis (ITT and subgroup analyses)**
**October 2003 – AIDAC meeting on clinical trial design in ABS**
**No constellation of signs/symptoms predicts bacterial etiology so sinus puncture necessary at to define patients with disease**
**No studies correlating >ten days of symptoms with sinus punctures**
**Lack of evidence of specificity of radiographic findings with positive culture on sinus puncture**
**No evidence to support “presumed eradication” of organisms**
**Timing of outcome important in relation to ability of trial to evaluate effectiveness and time to resolution of symptoms may be most sensitive measure of outcome**
**Lack of evidence from previous placebo controlled trials to base any noninferiority margin – trials should be superiority trials**

## Evaluating Data from Literature

**Criteria for submission of published literature reports alone as evidence of effectiveness**
**Multiple studies**** conducted by different investigators where each clearly has adequate design and ****findings consistent**** across studies**
**High level of detail in published reports including clear and adequate descriptions of statistical analysis plans, ****prospectively determined analytical methods and study endpoints****, and full accounting of all enrolled patients**
**Clearly appropriate ****endpoints ****objectively assessed and ****not dependent on investigator judgment**
**Robust results achieved by protocol-specified analyses that yield consistent conclusions of efficacy and ****do not require selected post-hoc analyses, subsetting or reduced datasets**** (e.g. analysis of only responders)**
**Conduct of studies by groups with properly documented operating procedures**

***Guidance for Industry: Providing Clinical Evidence of Effectiveness for Human Drugs and Biological Products***** **
**p. 19.**

##  Similar disease definition

- 50

- 40

- 30

- 20

- 10

- 0

- 10

- 20

- 30

- 40

- 50

- Favors placebo

- Favors drug

- What One Would Need to Justify  a Margin of 10% in Acute Bacterial Sinusitis

** ****Similar disease definition**
** ****Similar endpoint definition**
** ****Similar timing of endpoint**
** ****Constancy of effect of control**

## ABS Placebo Controlled Trials

**Description of trials**
**21 trials compared antimicrobial to placebo in all languages but 4 not prospective, not randomized and/or no direct patient outcomes**
**17 randomized, prospective trials evaluated**
**2787 total patients (1132 ‘evaluable’ on placebo and 1331 ‘evaluable’) on various drugs/doses/durations**
**No quinolones**
**One trial in children with cefuroxime in children (no evidence of benefit)**
**Dates from 1964 to 2005 (41 year span)**
**8 trials published since 2000,  2 in 2005, showing that placebo controlled trials can and are being performed**
**3 published since last AIDAC meeting in 2003 including one in US, one in Europe and one in pediatric population **
**Demographics**
**Average age per trial was 37 years with age range 18-93 years**
**Average gender distribution per trial was 60% female in trials with adults or mixed populations**

## Lindbaek et al. 1998 n=70 

- 50

- 40

- 30

- 20

- 10

- 0

- 10

- 20

- 30

- 40

- 50

- Ganaca et al. 1973 n=50

- Lindbaek et al. 1996 n=127

- Haye et al. 1998 n=168

- Hansen et al. 2000 n=127

- Wald et al. 1986 n=93

- Varonen et al. 2003 n=146

- van Buchem et al. 1997 n=206

- deSutter et al. 2003 n=135

- Axelsson et al. 1970 n=142

- Norrelund et al. 1978 1978 n=135

- Kaiser et al. 2001 n=265**a** (77)**b**

- Bucher et al. 2003 n=251

- Garbutt et al. 2001 n=161

- Stalman et al. 1997 n=186

- Favors placebo

- Favors study drug

- Analysis of Efficacy in Placebo Controlled Trials in Acute Bacterial Sinusitis

**cefuroxime d14**

**amoxicillin-clavulanate d14**

**amox or amoxicillin-clav d14**

**azithromycin d8**

**pivampicillin d8**

**pencillin or lincomycin d10**

**amoxicillin d14**

**amoxicillin d14**

**amox or doxy or penicillin d 14**

**amoxicillin or amox-clav d10**

**penicillin d7**

**azithromycin d14**

**amoxicillin or penicillin d10**

**cyclacillin (not specified)**

- Kristo et al. 2005 n=82

- Lindbaek et al. 1998 n=70** **

**doxycycline d10**

- Merenstein et al. 2005 n=135

**amoxicillin or penicillin d10**

**amoxicillin d10**

**a**

**b**

## ABS Placebo Controlled Trials

**No reliable, consistent magnitude of benefit of antimicrobials compared to placebo [M(1)], so no evidence upon which to base any noninferiority margin**
**Majority of trials do not provide evidence of benefit of antimicrobials compared to placebo; most trials powered to rule out differences of 15% to 35% which is amount needed to justify 10% to 15% margin**
**Trial with largest point estimate of benefit had lower bound of 95% confidence interval 11.3%; most of point estimates of treatment difference <10%**
**Point estimates success rates in placebo groups range from 29% to 95% and point estimate success with various drugs range from 35% to 93%**
**No evidence of decreasing complications or prevention of chronic sinusitis**
**Cannot select loss of effect of test drug relative to control [M(2)] since magnitude of benefit of control relative to placebo [M(1)] not clear**
**No constancy of effect of control**
**Different definitions used for enrollment criteria**
**None of trials used receipt of additional antimicrobial as an endpoint**
**Timing of fixed endpoints vary and not longer than 14 days for primary fixed endpoint; effects of drug compared to placebo disappear with time**

## Lindbaek et al. 1998 n=70 

- 50

- 40

- 30

- 20

- 10

- 0

- 10

- 20

- 30

- 40

- 50

- Ganaca et al. 1973 n=50

- Lindbaek et al. 1996 n=127

- Haye et al. 1998 n=168

- Hansen et al. 2000 n=127

- Wald et al. 1986 n=93

- Varonen et al. 2003 n=146

- van Buchem et al. 1997 n=206

- deSutter et al. 2003 n=135

- Axelson et al. 1970 n=142

- Norrelund et al. 1978 1978 n=135

- Kaiser et al. 2001 n=265* (77)

- Bucher et al. 2003 n=251

- Garbutt et al. 2001 n=161

- Stalman et al. 1997 n=186

- Favors placebo

- Favors study drug

- Analysis of Safety in Placebo Controlled Trials in Acute Bacterial Sinusitis

- Kristo et al. 2005 n=82

- Lindbaek et al. 1998 n=70** **

- Merenstein et al. 2005 n=135

**odds ratio 3.89 (2.09, 7.25)**

**p-value 0.017**

**GI AEs 3 drug, 0 placebo**

**6 excluded from analysis on drug, 2 on placebo**

## Lindbaek et al. 1998 n=70 

- 50

- 40

- 30

- 20

- 10

- 0

- 10

- 20

- 30

- 40

- 50

- Ganaca et al. 1973 n=50

- Lindbaek et al. 1996 n=127

- Haye et al. 1998 n=168

- Hansen et al. 2000 n=127

- Wald et al. 1986 n=93

- Varonen et al. 2003 n=146

- van Buchem et al. 1997 n=206

- deSutter et al. 2003 n=135

- Axelsson et al. 1970 n=142

- Norrelund et al. 1978 1978 n=135

- Kaiser et al. 2001 n=265**a** (77)**b**

- Bucher et al. 2003 n=251

- Garbutt et al. 2001 n=161

- Stalman et al. 1997 n=186

- Favors placebo

- Favors study drug

- Analysis of Efficacy in Placebo Controlled Trials in Acute Bacterial Sinusitis

- Kristo et al. 2005 n=82

- Lindbaek et al. 1998 n=70** **

- Merenstein et al. 2005 n=135

**a**

**b**

**gemi 009**

## In Vitro Resistance and Clinical Outcomes

**Given uncertainty about magnitude of treatment effect with any drug, correlation between in vitro resistance and clinical outcomes also unclear**
**Even with susceptible organisms, lack of correlation between microbiological outcomes and clinical outcomes in patients**
**“****Bacterial survival in the maxillary sinus despite a high concentration in the sinus illustrates that MIC values determined in the laboratory do not always mirror the sensitivity of the bacteria to antibiotics *****in vivo*****.”**
**Carenfeldt C et al. Scand J Infect Dis 1975;7:259-64.**
**Patient often recover clinically despite persistence of organism and differences in potency of antimicrobials does not necessarily translate into differences in clinical outcomes**
**Carenfeldt C et al. Acta Otolaryngol 1990;10:128-135. **

## Conclusions

**Need for demonstration of effect of control drug relative to placebo  in noninferiority trial has been noted in regulations since 1980’s**
**Evaluation of previous placebo controlled trials in ABS does not show reliable and reproducible magnitude of effects of antimicrobials relative to placebo for studying new drugs in clinical trials**
**Demonstration of noninferiority in acute bacterial sinusitis trials still leaves uncertainty as to whether this demonstrates effectiveness of drugs relative to placebo**
**Demonstration of effectiveness needed to balance any potential harms of therapy**

## Back Up Slides

## Substantial Evidence

**“****Upjohn contends that ‘the totality of materials, which include these 54 articles, the material submitted over the years since these products were certified, and the clinical experience in totality clearly satisfy the substantial evidence’. It says the clinical experience, widespread throughout the world, used by thousands upon thousands of doctors in 750 million doses is a very significant factor.”**
**Upjohn v. Finch, 1970 Appendix A, p. 12**

## Substantial Evidence

**“****The Commissioner concludes that Congress itself has described the type of evidence that is suitable to support claims of effectiveness. The claims must be supported by adequate and well controlled investigations. This means that the experimental factors must be so controlled that **_**the effectiveness of an anti-infective drug on the disease process in patients**_** can be compared with the effect of no treatment or of a recognized effective treatment of patients with the same disease or condition.”**
**Upjohn v. Finch, 1970 Appendix A, p. 12**

## Substantial Evidence

**“****Upjohn recognizes this fact, but contends that several in vitro studies reported, the mouse study, and the uncontrolled observations reported in the literature, when added to the evidence obtained from the studies in which controls were attempted, raise the quality of the data to the level required by the law.”**
**Upjohn v. Finch, 1970 Appendix A, p. 12**

## Substantial Evidence

**“****The *****in vitro***** studies are suggestive of some effectiveness...in laboratory experiments utilizing artificially cultured microorganisms as test systems, but because the studies are not at all correlated  clinical [trial] experience they cannot be used as a basis for concluding the drugs will have the effectiveness claimed for them when used to treat naturally occurring clinical disease in man.”**
**Upjohn v. Finch, 1970 Appendix A, p. 12**