# SUMMARY OF SAFETY AND EFFECTIVENESS DATA

## A. GENERAL INFORMATION

> Device Generic Name: System, Image Processing, Radiological
>
> Device Trade Name: ImageChecker® CT CAD Software System
>
> Applicant's Name and Address: R2 Technology, Inc.
>
> 1195 W. Fremont Avenue
>
> Sunnyvale, CA 94087
>
> PMA Number: P030012
>
> Date of Panel Recommendation: February 3, 2004
>
> Date of notice of approval to Applicant: TBD

### B. INDICATIONS FOR USE

The ImageChecker CT CAD Software System is a Computer-Aided Detection
(CAD) system designed to assist radiologists in the detection of solid
pulmonary nodules during review of multi-slice CT (MSCT) scans of the
chest. It is intended to be used as a "second reader," alerting the
radiologist -- after his or her initial reading of the scan -- to
regions of interest (ROIs) that may have been initially overlooked.

In this way, the system reduces the chance of observational lapses by
the reader due to fatigue, distraction, satisfaction of search, or other
recognized causes of a sub-optimal review. Thus, the system increases
the chances that a clinically significant nodule will be not be
overlooked, resulting in improved patient care (e.g., detection at an
earlier, more treatable stage).

## C. CONTRAINDICATIONS

There are no contraindications for the use of this device.

### D. WARNINGS AND PRECAUTIONS

Warnings and Precautions for use of the device are stated in the
attached product labeling.

### 

### 

### E. DEVICE DESCRIPTION

**[System Overview]{.underline}**

The ImageChecker CT System is an image analysis and visualization system
designed to assist radiologists in the review of multi-slice CT (MSCT)
exams of the chest and in the detection of solid pulmonary nodules.

The ImageChecker CT System is a combination of dedicated computer
software and hardware. The system is comprised of the ImageChecker CT
Workstation (K023003) and the ImageChecker CT CAD software that is the
subject of this PMA filing. The two components are related as indicated
in Figure 1.

**Figure 1.** Overview of the ImageChecker CT system illustrating the
two devices that comprise the system and the relevant hardware and
software components.

Developed by R2 Technology, Inc., the ImageChecker CT Computer-Aided
Detection (CAD) System is intended to be used as an adjunctive aid for
radiologists interpreting MSCT scans of the chest -- providing detection
of solid parenchymal densities that may represent clinically significant
nodules.

The ImageChecker CT CAD software does this by applying proprietary
signal processing algorithms to the large digital datasets generated
during scanning. These algorithms, in effect, analyze the complete set
of images and search for findings with features suggestive of a solid
pulmonary nodule. The system conveys information regarding "candidate"
nodules to the workstation for consideration by the reader.

The ImageChecker CT CAD Software System is not intended to be used
principally in a lung-cancer screening environment, where the vast
majority of scans would be expected to be free of clinically significant
disease. Rather, it is intended as a "back-up" nodule identification and
reporting system for radiologists reading everyday diagnostic chest CTs
-- many of which will have other pathologic processes competing for
their attention.

After the initial study acquisition by the CT scanner, copies of the
study are sent automatically to the review workstation for the user to
review as well as to the processing server for segmentation and CAD
analysis. When the processing server finishes, it sends a report to the
workstation. All image and information exchanged between the components
of the system and external devices (such as CT acquisition devices, PACS
systems) are controlled using standard DICOM (Digital Imaging and
Communications in Medicine) protocols.

The following sections give a brief overview of the individual devices
that comprise the ImageChecker CT System.

# 

# ImageChecker CT Workstation

The ImageChecker-CT Workstation is indicated for use as a general
medical imaging workstation, and is used to receive, store, transmit,
and display images from a Multislice CT scanner (MSCT). The FDA cleared
the workstation for marketing on 11/4/2002 (K023003) as ImageChecker CT
Model LN-500. The ImageChecker CT Workstation combined with the
ImageChecker CT CAD Software that is the subject of this PMA submission
will be distributed as the Model LN-1000.

The ImageChecker-CT Workstation LN-1000 is comprised of two
off-the-shelf personal computers, one with a Linux-based operating
system (OS) and one with a Microsoft Windows-based OS, and a display
monitor.

The processing software performs several functions:

-   Receives MSCT exams using the DICOM standards.

-   Takes the CT images and segments different anatomical structures
    into normal structures (e.g. vessels) and other composite features.

-   Stores the location and characteristics of the segmented composite
    features in a DICOM Structured Report object.

The workstation display software provides tools for the radiologist that
aid in the review process. During the review, the radiologist instructs
the display software by means of a standard keyboard and mouse. The
images and findings are communicated to the radiologist by means of a
color ~~Flat Panel~~ display. When the user completes his or her review,
the system makes a summary report available that lists any findings and
measurements associated with the study. The user can also print this
summary.

The workstation is also able to display the findings that are identified
by the CAD software (see next section). The radiologist using the
workstation is able to view the CAD findings -- after a preliminary
review of the study -- using a simple button press on the user
interface. The user can then recheck the areas that the CAD software
points out to determine if a change in his or her interpretation of the
exam is warranted.

# 

# ImageChecker CT CAD software

The ImageChecker CT CAD software is an adjunctive software package that
analyzes the CT images after they have been pre-processed by the
workstation software and identifies regions of interest that may be
solid pulmonary nodules. The regions of interest are identified by means
of the propriety signal processing algorithms that analyze the images
and search for findings with features suggestive of a solid pulmonary
nodule (see Figure 2).

The location information about these identified regions of interest is
sent to the workstation using a DICOM CAD Structured Report (DICOM CAD
SR).

![](media/image1.png){width="2.48125in" height="2.625in"}

![](media/image2.png){width="2.513888888888889in" height="2.625in"}

**Figure 2.** Examples of CAD-marked pulmonary nodules

**F. ALTERNATIVE PRACTICES AND PROCEDURES**

Currently, there is no alternative to assist radiologists in the
detection of lung nodules on multi-slice CT chest scans other than
double reading.

**G. MARKETING HISTORY**

The ImageChecker CT CAD Software System was launched for commercial sale
in the European Union at the European Congress of Radiology in March
2003.

**H. POTENTIAL ADVERSE EFFECTS OF DEVICE ON HEALTH**

There are no known direct risks to safety or health caused by, or
related to, the use of the device. The indirect risks are that the
device may fail to identify and mark some actionable lesions and will
mark some lesions that do not require further action.

### NON-CLINICAL STUDIES

There are no non-clinical studies to report for this device.

### 

### J. CLINICAL STUDIES

R2 Technology, Inc. has conducted two pivotal clinical studies to
evaluate the safety and effectiveness of the ImageChecker CT CAD
Software System. The studies were based on a retrospective case
collection project involving multiple clinical sites in various regions
across the U.S.

[Objectives]{.underline}

The first pivotal study was designed to generate a "truth" set of cases
containing solid pulmonary nodules, as well as cases with no nodules, to
be used as a reference truth for subsequent studies. The second pivotal
study, conducted in two phases, was an Observer/ROC (Receiver Operating
Characteristic) Study, designed to measure the performance enhancement
of radiologists using the System.

[Sites and Cases for Case Collection Project and Subsequent
Studies]{.underline}

Five (5) regionally diverse sites contributed 151 cases to the study; 2
sites in the Northeast, and 1 site each from the South, the Midwest, and
the West. Of these sites, 3 were private imaging centers and 2 were
academic medical centers. These sites are listed in Table 1.

+----------------+------+------------------+---------------+----------+
| **Table 1.     |      |                  |               |          |
| Sites          |      |                  |               |          |
| contributing   |      |                  |               |          |
| nodule and     |      |                  |               |          |
| non-nodule CT  |      |                  |               |          |
| chest cases in |      |                  |               |          |
| the Case       |      |                  |               |          |
| Collection     |      |                  |               |          |
| Project and    |      |                  |               |          |
| subsequent     |      |                  |               |          |
| clinical       |      |                  |               |          |
| studies**      |      |                  |               |          |
+----------------+------+------------------+---------------+----------+
| Name           | S    | Number of        | Number of     | Total    |
|                | tate |                  | Non-nodule    | Cases    |
|                |      | Nodule-present   | Cases         |          |
|                |      | Cases Used in    |               |          |
|                |      | Studies          | Used in       |          |
|                |      |                  | Studies       |          |
+----------------+------+------------------+---------------+----------+
| Atlantic       | NJ   | 11               | 10            | 21       |
| Medical        |      |                  |               |          |
| Imaging        |      |                  |               |          |
+----------------+------+------------------+---------------+----------+
| MRI & CT       | VA   | 15               | 23            | 38       |
| Diagnostics    |      |                  |               |          |
+----------------+------+------------------+---------------+----------+
| South Jersey   | NJ   | 14               | 35            | 49       |
| Radiology      |      |                  |               |          |
+----------------+------+------------------+---------------+----------+
| University of  | IA   | 18               | 9             | 27       |
| Iowa           |      |                  |               |          |
+----------------+------+------------------+---------------+----------+
| UC, San        | CA   | 5                | 11            | 16       |
| Francisco      |      |                  |               |          |
+----------------+------+------------------+---------------+----------+
| Total          |      | 63               | 88            | 151      |
+----------------+------+------------------+---------------+----------+

All cases were culled consecutively from the sites' digital archives
according to the inclusion and exclusion criteria identified in the case
collection protocols.

The nodule-present cases collected included only those in which a
diagnosis of cancer, either primary lung cancer or an extrathoracic
neoplasm, had been documented. Other co-existing disease processes
resulting in the formation of nodules (e.g. TB, histoplasmosis,
rheumatoid lung) were allowed, as were cases containing other
"background" pathology such as lobar pneumonia, emphysema, and heart
failure.

A total of 63 nodule-present cases dating from November 2001 through
December 2002 were included in the studies. The study population
consisted of 56% females and 44% males, with a median age of 66 and a
range of 20-86. The malignancy consisted of primary lung cancer in 24
(38%) of these cases, and documented extra-thoracic primary cancer with
suspected metastatic disease to the lung in the remaining 39 (62%)
cases.

The "normal" cases collected were those in which no nodules were deemed
to be present by the principal investigator at each site. Other disease
processes could be present, including the presence of pulmonary masses
(\>3cm). Histories of cancer, radiation therapy, or even previous
thoracotomy, were allowed. In all, 88 nodule-absent cases dating from
June 2002 through December 2002 were included in this retrospective case
collection.

[STUDY #1 -- IDENTIFICATION OF REFERENCE TRUTH]{.underline}

The objective of this study was to generate a "truth" set of unanimous
actionable nodules, as identified by a panel of 3 experienced
radiologists, to serve as a reference truth for all subsequent studies.

To achieve this objective, multiple panel sessions were scheduled in
which 3 radiologists independently read a variable number of cases (min
= 12, max = 25) until all 151 study cases had been interpreted by all 3
readers.

The three panelists identified a total of 142 findings in the 151 cases
that met the size (4-30mm) and peak density (\> -100HU) requirements,
and which all three panelists agreed were actionable. The presence or
absence of at least one of these findings in a quadrant was used as the
reference truth for study #2 below.

The findings ranged in size from 4-28mm. It is of note that the majority
of these findings were between 4mm and 8mm in diameter (46%, 66/142),
with the largest categories being the 5 -- 6 mm (15%, 21/142) and 6 -- 7
mm (15%, 22/142) findings.

#### STUDY #2 -- EFFECT OF CAD SYSTEM ON IMPROVING ACCURACY OF IDENTIFICATION OF ACTIONABLE NODULES

The objective of this study was to demonstrate that review of CAD output
improves performance of radiologists reviewing MSCT with respect to
their ability to accurately identify actionable nodules. The study
employed a Receiver Operating Characteristic (ROC) methodology that has
become the standard in the radiology community for evaluating imaging
modalities. The study was conducted in two phases, with the first phase
using a smaller case set (n=32 cases) and the second phase using a
larger case set (n=90 cases). The study methods and analysis were the
same for both phases of the study.

For Phase I of the study, thirty-two (32) cases were randomly selected
from the 151 cases evaluated in study #1. These cases were each divided
into 4 quadrants, yielding a total of 128 regions for evaluation. Each
of 15 radiologists independently reviewed the 128 quadrants, first
without computer-aided detection (CAD) and then immediately with CAD.

For Phase II of the study, ninety (90) cases were randomly selected from
the remaining 119 cases (151 -- 32 used in Phase I). Again, the cases
were divided into 4 quadrants, yielding 360 regions for evaluation. Each
of 15 radiologists (9 returning radiologists from Phase I, 6 new
radiologists) independently reviewed the 360 quadrants, first without
computer-aided detection (CAD) and then immediately with CAD

For both phases, each reader rated each quadrant on a 0-100
"actionability scale" as to his or her level of confidence that the
quadrant contained at least one actionable nodule. Ratings were provided
both before and after viewing the CAD marks.

For purposes of measuring reader performance, quadrants were defined as
"actionable" if the reference truth consensus panel described in study
#1 unanimously agreed that at least one of the findings in the quadrant
was actionable.

An area under the receiver operator curve (A~z~) was computed for each
of the 15 radiologists before and after CAD. The average curve is shown
below in Figure 3 for the Phase I study and Figure 4 for the Phase II
study. In both cases the area under the ROC curve increased with the use
of CAD. If the full plot is viewed as a unit square, the area separating
the two curves in the Phase I study is 0.014 and the area separating the
two curves in the Phase II study is 0.024. Although the second study
shows greater separation and a slightly smoother pair of curves, the
basic shape of the curves in the two studies is similar.

![](media/image3.wmf){width="5.497222222222222in"
height="3.9180555555555556in"}

**Figure 3.** Average ROC curve showing pre-CAD (dashed line) and
post-CAD (solid line) for Phase I (32 case, 128 quadrant) study.

![](media/image4.wmf){width="5.497222222222222in"
height="3.935416666666667in"}

**Figure 4.** Average ROC curve showing pre-CAD (dashed line) and
post-CAD (solid line) performance for Phase II (90 case, 360 quadrant)
study.

The primary analysis of statistical significance was based on the
Dorfman-Berbaum-Metz (DBM) ANOVA-after-jackknife approach[^1] (adapted
for the fact that the quadrants are "clustered" data) and the results
are presented in Table 2 below.

In the Phase II study (Figure 4), the average reader improvement in A~Z~
(estimated using the ANOVA-after-jackknife) was 0.0240 0.0077 (p=0.0033)
with a 95% confidence interval of (0.0084, 0.0395). By comparison, in
the Phase I study (Figure 3), the average reader improvement in Az
(estimated using the ANOVA-after-jackknife) was 0.0140 0.0084 (p=0.1083)
with a 95% confidence interval of (-0.0033, 0.0313). The combined 122
case study demonstrated that the average reader improvement in A~Z~
(estimated using the ANOVA-after-jackknife) was 0.0209 0.0062 (p=0.0013)
with a 95% confidence interval of (0.0085, 0.0333). Thus both the Phase
II study by itself and the pooled study results showed a statistically
significant improvement in the area under the ROC curve with the use of
CAD.

The secondary analysis of statistical significance was conducted to
determine the dependence of the study results on the use of the
consensus reference truth from study #1 as the gold standard for the ROC
study. In general, a reference truth based on a consensus panel
assessment of actionability is weaker than one based on a more invariant
gold standard such as biopsy. To examine the effect of variability in
the unanimous three-panelist reference truth, two-panelist reference
truths and single-panelist reference truths were constructed from the
data collected in study #1.

Implementing this variable truth is difficult within the framework of
the ANOVA-after-jackknife analysis, therefore the secondary analysis
employs a bootstrap analysis[^2]. The bootstrap is a
computationally-intensive non-parametric method that allows complex
analyses to be repeated many times using different randomly generated
datasets (all based on the original data) to approximate the variability
that would occur if the entire study were repeated many times. As a test
of the validity of the bootstrap mechanism, the analysis was performed
first using the unanimous reference truth. As shown in Table 2, the
results of the ANOVA-after-jackknife analysis and the bootstrap analysis
using the unanimous reference truth are very similar for the Phase I,
Phase II and pooled data.

+-------------+-------+-----------+-------+--------------+-----------+
| **Table 2.  |       |           |       |              |           |
| Study       |       |           |       |              |           |
| analyses of |       |           |       |              |           |
| the         |       |           |       |              |           |
| s           |       |           |       |              |           |
| ignificance |       |           |       |              |           |
| of the      |       |           |       |              |           |
| Improvement |       |           |       |              |           |
| in Area     |       |           |       |              |           |
| under ROC   |       |           |       |              |           |
| curve for   |       |           |       |              |           |
| the Phase   |       |           |       |              |           |
| I, Phase II |       |           |       |              |           |
| and pooled  |       |           |       |              |           |
| study       |       |           |       |              |           |
| data.**     |       |           |       |              |           |
+-------------+-------+-----------+-------+--------------+-----------+
| Analysis    | Study | Estimated | p-    | 95% CI       | St        |
| Method      |       | Im        | value |              | atistical |
|             |       | provement |       |              |           |
|             |       |           |       |              | Sig       |
|             |       | in Az     |       |              | nificance |
+-------------+-------+-----------+-------+--------------+-----------+
| Primary:    | Phase | 0.0140    | 0.108 | (-0.0033,    | No        |
|             | I     |           |       | .0313)       |           |
| ANOVA-after |       |           |       |              |           |
| -jackknife, |       |           |       |              |           |
| with        |       |           |       |              |           |
| unanimous   |       |           |       |              |           |
| reference   |       |           |       |              |           |
| truth       |       |           |       |              |           |
+-------------+-------+-----------+-------+--------------+-----------+
|             | Phase | 0.0240    | 0.003 | ( 0.0084,    | **Yes**   |
|             | II    |           |       | .0395)       |           |
+-------------+-------+-----------+-------+--------------+-----------+
|             | **Poo | *         | **0.  | **( 0.0085,  | **Yes**   |
|             | led** | *0.0209** | 001** | .0313)**     |           |
+-------------+-------+-----------+-------+--------------+-----------+
| **S         | Phase | 0.0139    | 0.126 | (-0.0041,    | No        |
| econdary:** | I     |           |       | .0354)       |           |
| Bootstrap   |       |           |       |              |           |
| with        |       |           |       |              |           |
| unanimous   |       |           |       |              |           |
| reference   |       |           |       |              |           |
| truth       |       |           |       |              |           |
+-------------+-------+-----------+-------+--------------+-----------+
|             | Phase | 0.0246    | \<    | ( 0.0089,    | **Yes**   |
|             | II    |           | 0.001 | .0446)       |           |
+-------------+-------+-----------+-------+--------------+-----------+
|             | **Poo | *         | *     | **( 0.0074,  | **Yes**   |
|             | led** | *0.0192** | *\<0. | .0345)**     |           |
|             |       |           | 001** |              |           |
+-------------+-------+-----------+-------+--------------+-----------+
| Secondary:  | Phase | 0.0168    | 0.058 | (-0.0004,    | No        |
|             | I     |           |       | .0396)       |           |
| Bootstrap   |       |           |       |              |           |
| with random |       |           |       |              |           |
| 2-panel     |       |           |       |              |           |
| reference   |       |           |       |              |           |
| truth       |       |           |       |              |           |
+-------------+-------+-----------+-------+--------------+-----------+
|             | Phase | 0.0216    | 0.002 | ( 0.0077,    | **Yes**   |
|             | II    |           |       | .0387)       |           |
+-------------+-------+-----------+-------+--------------+-----------+
|             | **Poo | *         | *     | **( 0.0073,  | **Yes**   |
|             | led** | *0.0186** | *\<0. | .0314)**     |           |
|             |       |           | 001** |              |           |
+-------------+-------+-----------+-------+--------------+-----------+

Finally, several approaches were used, based on the bootstrap
re-sampling approach, to incorporate random reference truths for the
random cases against which the random readers' performance could be
estimated. All methods showed similar results; therefore a
representative approach that selected two panelists at random is shown
here. Based on varying the reference truth in this way, in the Phase II
study, the average reader improvement in A~Z~ (estimated using the 1000
bootstrap samples with variability in the reference truth) was 0.0216
(p=0.002) with a 95% confidence interval of (0.0077, 0.0387). By
comparison, in the Phase I study, the average reader improvement in A~Z~
(estimated using the 1000 bootstrap samples with variability in the
reference truth) was 0.0168 (p=0.058) with a 95% confidence interval of
(-0.0004, 0.0396). The combined (pooled) data demonstrated that the
average reader improvement in A~Z~ (estimated using the 1000 bootstrap
samples with variability in the reference truth) was 0.0186 (p\<0.001)
with a 95% confidence interval of (0.0073, 0.0314). Thus again, the
results with the varied reference truth show that both the Phase II
study by itself and the pooled study results showed a statistically
significant improvement in the area under the ROC curve with the use of
CAD. The Phase I study by itself demonstrated a consistent trend, but 32
cases were insufficient to achieve statistical significance.

The primary and secondary analysis from the Phase II study alone or the
two studies combined shows that the ImageChecker CT CAD Software System
significantly improves radiologists' ROC performance for detecting solid
pulmonary nodules between 4 and 30 mm in diameter. Also, this result is
robust when different reference truth definitions are used in the
analysis.

## K. CONCLUSIONS DRAWN FROM STUDIES

For multi-slice CT exams of the chest:

The ImageChecker CT CAD Software System significantly (p=0.003) improves
radiologists' ROC performance for detecting solid pulmonary nodules
between 4 and 30mm in size.

**L. PANEL RECOMMENDATIONS**

(To be added by FDA)

**M. CDRH DECISION**

(To be added by FDA)

**N. APPROVAL SPECIFICATIONS**

(To be added by FDA)

[^1]: Dorfman DD, Berbaum KS and Metz, CE. Receiver Operating
    Characteristic Rating Analysis. Invest Radiol 1992; 27: 723-731.

[^2]: Rutter, C. Bootstrap Estimation of Diagnostic Accuracy with
    Patient-clustered Data. Acad Radiol 2000; 7: 413-419.