# Presentation: 933708

## Clinical Assessment – Part II
William Sacks, PhD, MD

**William Sacks, PhD, MD**

**COMPUTERIZED THERMAL IMAGING, INC****.**
** *****Breast Cancer System         BCS 2100******                  *****P010035**** **
** **
**Radiological Devices Advisory Panel**
**December 10, 2002**

- Sacks - Clinical Assessment

- 12/10/02

## Safety of the BCS has two aspects:
 Adverse events
 Accuracy of
    BCS output 

** ****Adverse events**
** ****Accuracy of**
**    ****BCS output **

- Sacks - Clinical Assessment

- 12/10/02

## There were only 4 minor adverse events, two of which were thought not to be device related, during a clinical trial involving 2407 subjects.

- Sacks - Clinical Assessment

- 12/10/02

## From the point of view of the BCS output:

 Safety is more closely
   related to sensitivity
   (cancers), or FNR, but
   also FPR
 Effectiveness is more
   closely related to
   specificity (benign masses)

** **_**Safety**_** is more closely**
**   ****related to **_**sensitivity**_
**   ****(cancers), or FNR, but**
**   ****also FPR**
** **_**Effectiveness**_** is more**
**   ****closely related to**
**   **_**specificity**_** (benign masses)**

- Sacks - Clinical Assessment

- 12/10/02

## POINTS TO BE COVERED

 What BCS 2100 (BCS) is and
     is not intended to do
 How BCS does it
 What the clinical trial
     demonstrated
 Labeling issues

** ****What BCS 2100 (BCS) is and**
**     ****is not intended to do**
** ****How BCS does it**
** ****What the clinical trial**
**     ****demonstrated**
** ****Labeling issues**

- Sacks - Clinical Assessment

- 12/10/02

## CLINICAL SUBMISSIONS
 PMA 
 Amendment 4
 Amendment 5
 Amendment 7

** ****PMA **
** ****Amendment 4**
** ****Amendment 5**
** ****Amendment 7**

- Sacks - Clinical Assessment

- 12/10/02

## After reviewing the PMA, the FDA sent a letter to CTI listing a number of deficiencies.  CTI’s response was Amendment 4.

- Sacks - Clinical Assessment

- 12/10/02

## In Amendment 4, for their conclusions concerning BCS effectiveness (p. 11), CTI retrospectively selected from the PMA data one of two analytical indices (Se/Sp, not ROC) and two of three lesion types (masses, and arch. dist., not microcalcifications).

- Sacks - Clinical Assessment

- 12/10/02

## In Amendment 4 the revised labeling further deleted architectural distortion, and referred to masses alone.

- Sacks - Clinical Assessment

- 12/10/02

## The FDA sent another deficiency letter to CTI.  The response was Amendment 5.

- Sacks - Clinical Assessment

- 12/10/02

## Amendment 5 was offered as a test of BCS in additional subjects, because Amendment 4 had contained retrospective selections.  CTI called the additional dataset PPMA (Post-PMA).  

- Sacks - Clinical Assessment

- 12/10/02

## Amendment 5 confined its analysis of the PPMA data to the newly chosen analytical index (Se/Sp) in the newly chosen subgroup (masses).

- Sacks - Clinical Assessment

- 12/10/02

## In addition to presenting data on a new set of subjects (PPMA), Amendment 5 also contained analysis of the combined datasets from Amendment 4 and the PPMA.

- Sacks - Clinical Assessment

- 12/10/02

## Because of the retrospective selections in Amendment 4, FDA asked CTI to justify combining that data with the PPMA.  CTI’s response was Amendment 7. 

- Sacks - Clinical Assessment

- 12/10/02

## In Amendment 7 CTI applied a Bonferroni correction in an attempt to compensate for
 retrospective selection of
   data in Amendment 4 and 
 the smallness of the
   additional (PPMA) sample
   in Amendment 5.

** ****retrospective selection of**
**   ****data in Amendment 4 and **
** ****the smallness of the**
**   ****additional (PPMA) sample**
**   ****in Amendment 5.**

- Sacks - Clinical Assessment

- 12/10/02

## Two overriding issues:
 Adequacy of data
 Interpretation of 
    data, i.e., do the
    data demonstrate S&E
    of BCS?

** **_**Adequacy**_** of data**
** **_**Interpretation**_** of **
**    ****data, i.e., do the**
**    ****data demonstrate S&E**
**    ****of BCS?**

- Sacks - Clinical Assessment

- 12/10/02

##   Adequacy of data

Can data from Amendment 4 contribute to judgment of S&E, when it consists of retrospective selection of mammographic masses, because the Se was too low in microcalcifications and in the tested population as a whole?

**Can data from Amendment 4 **_**contribute**_** to judgment of S&E, when it consists of retrospective selection of mammographic masses, because the Se was too low in microcalcifications and in the tested population as a whole?**

- Sacks - Clinical Assessment

- 12/10/02

## Adequacy of data (cont.) 

 Is a Bonferroni
 correction applicable in
 this context?

** ****Is a Bonferroni**
** ****correction applicable in**
** ****this context?**

- Sacks - Clinical Assessment

- 12/10/02

## Adequacy of data (cont.)

Are the data from
the PPMA alone (in
Amendment 5) adequate to
judge S&E?

**Are the data from**
**the PPMA alone (in**
**Amendment 5) adequate to**
**judge S&E?**

- Sacks - Clinical Assessment

- 12/10/02

## Interpretation of the data
It is noteworthy for the following discussion that no formal hypotheses were explicitly put forward for testing, either in the PMA or in the subsequent amendments.

**It is noteworthy for the following discussion that no formal hypotheses were **_**explicitly**_** put forward for testing, either in the PMA or in the subsequent amendments.**

- Sacks - Clinical Assessment

- 12/10/02

## To qualify as a testable hypothesis there must be a quantitative criterion, whereby either
  a point estimate may imply 
      rejection, or
  a confidence interval may
      entail exclusion.

**  ****a point estimate may imply **
**      ****rejection, or**
**  ****a confidence interval may**
**      ****entail exclusion.**

- Sacks - Clinical Assessment

- 12/10/02

## There were two implicit hypotheses:

 that the ROC area for BCS and 
   mammography combined would exceed that
   of mammography alone (with statistical
   significance) 
   [Protocol, Section 6.0 Statistical Analysis]

 that the point estimate for sensitivity
   would be at least 99.3% (in at least
   75% of simulations with the data)
   [PMA, Section 5.8.7.2, p. 489].

** ****that the **_**ROC area**_** for BCS and **
**   ****mammography combined would exceed that**
**   ****of mammography alone (with statistical**
**   ****significance) **
**   ****[Protocol, Section 6.0 Statistical Analysis]**
** ****that the point estimate for **_**sensitivity**_
**   ****would be at least 99.3% (in at least**
**   ****75% of simulations with the data)**
**   ****[PMA, Section 5.8.7.2, p. 489]****.**

- Sacks - Clinical Assessment

- 12/10/02

## The protocol otherwise contained only non-quantitative statements of what CTI hoped to achieve in the clinical trial.           For example, 
The objective of the study is to determine if the (BCS 2100), when used in conjunction with clinical examination and/or diagnostic mammography, increases the ability of physicians to differentiate benign from malignant, or suspicious, breast abnormalities. [Protocol, Section 3.1 Study Objective]

**The objective of the study is to determine if the (BCS 2100), when used in conjunction with clinical examination and/or diagnostic mammography, increases the ability of physicians to differentiate benign from malignant, or suspicious, breast abnormalities. ****[Protocol, Section 3.1 Study Objective]**

- Sacks - Clinical Assessment

- 12/10/02

## In the original PMA submission, the comparison of ROC areas failed to achieve statistical significance, except as an artifact of too few points in the mammography-alone curve, so it was not pursued in any of the amendments.

- Sacks - Clinical Assessment

- 12/10/02

## In addition, the sensitivities failed to achieve a level of 99.3% with 75% confidence, in any of the datasets.

- Sacks - Clinical Assessment

- 12/10/02

## SPECIFICITY

**Am. 4 **
**(n=90)**
**(Se 100%,**
**Sp 18.0%)**

**Combined**
**(n=105)**
**(Se 99.0%,    Sp 19.2%)**

**72.1%**

**99.7%**

**94.1%**

**98.8%**

**95.6%**

**96.7%**

**Reference ****Se 99.3%**

**SPECIFICITY**

**SENSITIVITY**

**100%**

**   ****0%**

**Chance Line**

- Sacks - Clinical Assessment

- 12/10/02

## Potential safety and effectiveness in U.S. population:

- Sacks - Clinical Assessment

- 12/10/02

## Percent of U.S. biopsies potentially obviated by the BCS if used on all eligible women:

 1.3 million U.S. women biopsied each year,
 of which ~45% (585,000) have
    mammographic masses, 
 of which ~80% (468,000) are benign, 
 of which 15% to 20% (70,000 to 94,000)
    would be BCS(-) and saved a biopsy.
 Thus ~5% to 7% of 1.3 million U.S. biopsies
    would be obviated,
 and that’s if BCS were used on all 585,000 women
    with mammographic masses on their way to biopsy.
 In addition to saving biopsies on these benign
    masses, approximately 1% to 6% of the
    malignant masses, and 0.5% to 3% of all breast
    cancers, might be delayed in diagnosis.

(

** **
** ****1.3 million U.S. women biopsied each year,**
** ****of which ~45% (585,000) have**
**    ****mammographic masses, **
** ****of which ~80% (468,000) are benign, **
** ****of which 15% to 20% (70,000 to 94,000)**
**    ****would be BCS(-) and saved a biopsy.**
** ****Thus ~**_**5% to 7%**_** of 1.3 million U.S. biopsies**
**    ****would be obviated,**
** ****and that’s if BCS were used on **_**all**_** 585,000 women**
**    ****with mammographic masses on their way to biopsy.**
** ****In addition to saving biopsies on these benign**
**    ****masses, approximately **_**1% to 6%**_** of the**
**    ****malignant masses, and **_**0.5% to 3%**_** of all breast**
**    ****cancers, might be delayed in diagnosis.**
**  **
**    **
** **
**     **
**(**

- Sacks - Clinical Assessment

- 12/10/02

## POINTS TO BE COVERED

 What BCS 2100 (BCS) is and
     is not intended to do
 How BCS does it
 What the clinical trial
     demonstrated
 Labeling issues

** ****What BCS 2100 (BCS) is and**
**     ****is not intended to do**
** ****How BCS does it**
** ****What the clinical trial**
**     ****demonstrated**
** ****Labeling issues**

- Sacks - Clinical Assessment

- 12/10/02

## Labeling issues

 Size of mass
 Depth of mass

** ****Size of mass**
** ****Depth of mass**

- Sacks - Clinical Assessment

- 12/10/02

##     Size of mass

Effect of small lesion size on device sensitivity difficult to evaluate, since only 2/105 cancers < 5mm.

**Effect of small lesion size on device sensitivity difficult to evaluate, since only 2/105 cancers < 5mm.**

- Sacks - Clinical Assessment

- 12/10/02

##   Size of mass  (cont.)

| Number of masses | 
Malignant | 
Benign |
| --- | --- | --- |
| <0.5 cm | 2 | 12 |
| 0.5 cm - 1.0 cm | 50 | 188 |
| >1.0 cm | 53 | 185 |

**(Amendment 4, p.8, Table A4.4,**
**Amendment 5, p.23, Table A5.14)**

- Sacks - Clinical Assessment

- 12/10/02

## 
   Depth of mass 

With the chosen IOS threshold (20.59) there was no definite effect of lesion depth on BCS result, but...

**   **_**Depth of mass**_** **
**With the chosen IOS threshold (20.59) there was no definite effect of lesion depth on BCS result, but...**

- Sacks - Clinical Assessment

- 12/10/02

##  Depth of mass (cont.)

...the effect of lesion depth was difficult to evaluate, since depth is not easily gauged on mammography.

**...the effect of lesion depth was difficult to evaluate, since depth is not easily gauged on mammography.**

- Sacks - Clinical Assessment

- 12/10/02

## Conclusions

**    **

- Sacks - Clinical Assessment

- 12/10/02

##            In summary

 Only 4 out of 2407 subjects had an adverse event,
    all minor.
 There were no explicit quantitative hypotheses.
 There were two implicit quantitative hypotheses.
 Neither hypothesis was fulfilled.
 Most data were selected retrospectively.
 Bonferroni correction is not applicable in this
    context.
 Using the trial results, if the BCS were in general
    use in the U.S., it would obviate 5% to 7% of the
    1.3 million biopsies per year. 
 Approximately 1% to 6% of these obviated biopsies
    would turn out to be malignant, and their
    diagnoses would thus be delayed.

** ****Only 4 out of 2407 subjects had an adverse event,**
**    ****all minor.**
** ****There were no explicit quantitative hypotheses.**
** ****There were two implicit quantitative hypotheses.**
** ****Neither hypothesis was fulfilled.**
** ****Most data were selected retrospectively.**
** ****Bonferroni correction is not applicable in this**
**    ****context.**
** ****Using the trial results, if the BCS were in general**
**    ****use in the U.S., it would obviate 5% to 7% of the**
**    ****1.3 million biopsies per year. **
** ****Approximately 1% to 6% of these obviated biopsies**
**    ****would turn out to be malignant, and their**
**    ****diagnoses would thus be delayed.**
**    **

- Sacks - Clinical Assessment

- 12/10/02