# Presentation: 669262

## PulminiqTMCyclosporine Inhalation Solution

- Pulmonary Drug Advisory Committee Meeting June 6, 2005
-  Statistical Evaluation
- Jyoti Zalkikar, Ph.D.
- Office of Biostatistics
- Center for Drug Evaluation and Research
- Food and Drug Administration

## ACS001- Study Conduct

- Small (56 patients in total), Phase II, Single Center  (UPMC) Study
- Not designed as a confirmatory study – planned sample size was 136 patients.
- Investigators did not use Case Report Forms during the conduct of the study.
- These were generated retrospectively by the applicant.
- No prospective Statistical Data Analysis Plan or formal stopping rules.

## ACS001- Study Conduct

- Study Began in 1997.
- First 10 patients – received CyIS (Cyclosporine Inhalation solution) as part of the open label pilot phase
- Next 58 patients were randomized to either CyIS with propylene glycol as vehicle or propylene glycol alone.
- Two of these 58 patients did not receive any dose of the study medication and were excluded from all analyses.
- The enrollment was stopped in August 2001 at 68 subjects.
- Study was “Vehicle” Controlled.
- All subjects received systemic immuno-suppresion

##  ACS001- Study Design

- Patients were to be enrolled from 7-42 days, after transplant.
- 3 violations – 47 days, 53 days and 77 days
- Randomization was stratified by enrollment window (7-21 days and 22-42 days) and CMV (D+/R-) mismatch status.
- 2 violations - incorrect strata
- Patients were to be on treatment daily for the first 10 days and 3 times a week thereafter for a total of 2 years.
- 30 patients discontinued treatment early
- All Patients were followed for at least 33 months.

## ACS001 - Study Design

- Evaluation of Survival and Chronic Rejection were NOT the primary objectives.
- Pre-specified Primary Objective – Superiority over propylene glycol in terms of Rate of Acute Rejection (AR).
- Study _failed_ to achieve the primary objective.
- Mean number of AR higher in the CyIS arm (1.35) compared to the propylene glycol (1.20) arm.

## Survival Distributions

**Survival Distributions**

- Survival Probability
- Month	12	24	36	48	60
- CyIS	96%	88%	88%	84%	77%
- Placebo	83%	77%	60%	56%	45%

- Log-rank p-value=0.0165

## Challenge for the Review Team

- Given that the study had failed on the primary endpoint, determine if the observed survival difference attributable to Cyclosporine Inhalation Solution.
- Review found Several Major Weaknesses

## Baseline Imbalances

- Though randomized, due to small sample size, the study failed to benefit from randomization – several baseline characteristics that clinicians  consider to influence patient survival are not balanced between the two groups.
- Therefore, groups are not comparable for evaluating survival or chronic rejection.

## Observed Important Baseline Imbalances 

- 						CyIS (26)	   PG (30)
- Single transplant*		15 (58%)	  24 (80%)
-  [Double transplant]		[11 (42%)	   6 (20%) ]
- Not Emphysema*		17 (65%)	  11 (37%)
- Prior AR Grade 2+	  	 8 (31%)	  13 (43%)
- Donor with inotropic 	13 (50%)	  25 (83%) support*
- Time in ICU >10 days*	 1 (4%)	   6 (20%)
- * statistically significant at 10% level

## Nature of Imbalance – All patients had at least one risk factor

- 2

- 4

- 11

- 7

- 7

- 11

- 6

- 7

- 1

- 0

## Proportional Hazards Model

- Statistical Adjustments “one factor at a time” (Applicant’s briefing document and presentation) OR analyses with some (not all) of the factors included are NOT appropriate as the two groups are still not comparable due to imbalance with respect to the other factors.
- The sample size of the study is too small for simultaneous adjustment for all the factors of importance that show imbalance.

## The questionable validity of any further inferential statistical analyses performed on the data from this study lends to caution while interpreting the results.

## Blinding ?

- ABCD code as part of the patient number may have revealed the treatment assignment to the investigators.
- 	This together with the retrospective nature of the study makes it vulnerable to the introduction of bias, as inadvertent as it may be.

## Detection of the presence and magnitude of bias is difficult, but its possibility lends to caution when interpreting the study results.

## An Illustration of the effect of small perturbations in the assignment of patients

## Survival Distributions

**Survival Distributions**

- Survival Probability
- Month	12	24	36	48	60
- CyIS	96%	88%	88%	84%	77%
- PG	83%	77%	60%	56%	45%

- Log-rank p-value=0.017

## Small SampleSensitivity Analysis

**Small Sample****Sensitivity Analysis**

- Survival Probability
- Month	12	24	36	48	60
- CyIS	86%	79%	79%	75%	69%
- PG	93%	85%	67%	62%	50%

- Log-rank p-value=0.271

## Due to Small Sample Size, the study results are highly sensitive to small perturbations in the assignment of patients. 

## Discontinuations - Reasons

| Reason for Discontinuation | CyIS (N=26) | PG (N=30) |
| --- | --- | --- |
| Withdrawal of Consent |    6 |    0 |
| AE |    4 |    10 |
| Unsatisfied with their response (LOE)  |    1  |    2 |
| Protocol Deviations |    2 |    5 |
| Completed Treatment |   13 |   13 |

## Number of Doses

| Number of Doses  | CyIS (n=26) | PG (n=30) |
| --- | --- | --- |
| 1 to <10 doses | 3 (12%) | 3 (10%) |
| 10 to < 25 doses | 3 (12%) | 2 (7%) |
| 25 to < 42 doses |     - | 1 (3%) |
| 42 to < 78 doses | 3 (12%) | 2 (7%) |
| 78 to <150 doses |     - | 3 (10%) |
| 150 to < 222 doses | 3 (12%)    |  4 (13%) |
| 222 to < 294 doses | 7 (27%) |  6 (20%) |
| ≥ 294 doses | 7 (27%) |  9 (30%) |

## Sensitivity analyses

**All 11 patients excluded**
- All 6 CyIS patients treated as PG patients
- Define Treatment Failure as Death or Discontinuation within 25 doses due to AE or Withdrawal of Consent

## Survival Distributions Patients with less than 25 doses excluded

**Survival Distributions ****Patients with less than 25 doses excluded**

- Log-rank p-value=0.202

- Hazard Ratio = .51
- 95% CI = (.16, 1.4)

## Sensitivity analyses

- All 11 patients excluded
**All 6 CyIS patients treated as PG patients**
- Define Treatment Failure as Death or Discontinuation within 25 doses due to AE or Withdrawal of Consent

## Survival Distributions Patients with less than 25 doses are treated as PG patients

**Survival Distributions ****Patients with less than 25 doses are treated as PG patients**

- Log-rank p-value=0.194

- Hazard Ratio = .52
- 95% CI = (.17, 1.34)

## Sensitivity analyses

- All 11 patients excluded
- All 6 CyIS patients treated as PG patients
**Define  Treatment Failure as Death or Discontinuation within 25 doses due to AE or Withdrawal of Consent**

## Distributions of Time to Treatment Failure*

**Distributions of Time to Treatment Failure***

- Survival Probability
- Month	12	24	36	48	60
- CyIS	73%	65%	65%	61%	55%
- PG	77%	70%	53%	53%	43%

- Log-rank p-value=0.556

- * Treatment Failure = Death or Discontinuation Within 25 Doses due to Adverse Event or Withdrawal of Consent

## The data from patients who  discontinued early and are not expected to have any influence of the treatment they are assigned to, have a big impact on the study results and that lends to caution when interpreting the study results.

## Lung Function – FEV1

- Number in bracket is sample size
- 				       _Month__	__CyIS (26)__	__ PG (30)__	_
- Mean(FEV1_Pre-enroll)	0	1.923 (15)	1.371 (17)
- Mean(FEV1_3m)		3	2.175 (26)	1.787 (26)
- Mean(FEV1_6m)		6	2.326 (25)	1.912 (27)
- Mean(FEV1_12m)		12	2.268 (24)	1.867 (24)
- Mean(FEV1_18m)		18	2.135 (22)	1.789 (22)
- Mean(FEV1_24m)		24	2.123 (20)	1.643 (19)
- _______________________________________________
- Information on Pre-enrollment FEV1 is grossly incomplete – 43% missing – Difference in pre-enrollment FEV1 between CyIS and Propylene Glycol  is statistically significant.

## Lung Function – FEV1

- 17

- 26

- 27

- 24

- 22

- 19

- 15

- 26

- 25

- 24

- 22

- 20

## Time to BOS*

**Time to BOS***

- Log-rank p-value=0.214

- No at Risk
- CyIS     26          26           24           20          16           12          11            9             5            0
- PG       30          27            21          18          14             8            6            4             3            1
- * Patients who died without diagnosis of BOS, as defined by the Applicant, were censored at the time of last follow-up for BOS.

## Additional Data

- 10 patients in Study ACS001 in CyIS open label phase – Limited information
- Baseline Characteristics	       CyIS- OL	CyIS-R      PG
- 	Donor Age > Patient Age  	20%    15%    20%
- 	CMV D+/R- Mismatch 		40%    19%    23%
- 	Gender Mismatch			50%    27%    33%
- 	Single Transplant			70%    58%    80%
- 	Double Transplant			30%    42%    20%
-  	Emphysema				70%    35%    63%
- 	Time in ICU > 10 days		 0%     4%     23%

## Survival Distributions

**Survival Distributions**

## Summary

- Primary Endpoint of Acute Rejection - No Treatment Effect
- Risk Factor Imbalance - Caution
- Blinding Concerns - Caution
- Sensitivity to small perturbations in assignment - High
- Discontinuations - Caution
- Lung Function - FEV1 and BOS - No Treatment Effect
- Additional Data - Not supportive

## The Question Still Remains...

- Is the Observed Survival Difference, between the two arms in the randomized portion of the trial,
- due to Cyclosporine Inhalation Solution
- or
- due to other factors?