Evaluation of Computer-Aided Lung Nodule Detection Software in Thoracic CT for Riverain Technologies LLC
Condition: Lung Cancer
Study type: Interventional

Summary: 
      STUDY DESIGN:

      This is a retrospective, multi-reader, multi-case, (MRMC) randomized reader study.

      OBJECTIVE:

      Primary: The primary objective of this clinical study is to prove that a user aided with
      ClearRead CT InSight (CRCTI) is superior to the unaided reader for detecting actionable lung
      nodules.

      Secondary: The secondary objective of this clinical study is to prove that the reader's
      reading time is not significantly increased when aided with CRCTI.

      NUMBER OF SUBJECTS:

      Retrospective CT studies from approximately 300 patients will be included in the study.
      Approximately 100 true positive cases and 200 normal cases.

      NUMBER OF READERS:

      A reader study with at least ten (10) participating radiologists (US Board Certified) will be
      conducted.

      PRIMARY ENDPOINTS:

      Scores given by the radiologists with and without ClearRead CT Insight will be recorded and
      compared to the true status of the study-cases. The frequency of the scores for each method
      (Unaided, Aided) will be tabulated and LROC curves constructed along with sensitivity,
      specificity, PPV, NPV and clinical actions. Additionally, machine nodule detection rate and
      false positives per patient on normal cases will be measured.

      PATIENT POPULATION :

      The study will target approximately one hundred (100) patients whose CT nodules were shown to
      be cancer and two hundred (200) normal patients. The patient population will be consistent
      with the national lung cancer screening protocols.
    

Detailed description: 
      A reader study with at least ten (10) participating radiologists will be conducted. A
      localized receiver operating characteristic curve will be used to evaluate radiologists'
      diagnostic performance (in terms of the trade-off between the sensitivity and specificity
      when the decision criteria changes) in the detection of lung nodules on lung CTs with and
      without the usage of the ClearRead CT Insight (CRCTI) system. The time needed for the review
      and interpretation of each case will also be recorded.

      An initial (baseline) interpretation will be made by each of the radiologists based on the
      Lung CT in its original form. At a minimum of one month later, each radiologists will again
      interpret the same images viewing the pair of CRCTI CT series: Two sets of CT images
      (standard with CADe marks and processed with vessel suppression will be presented on either
      one large monitor or two adjacent monitors.

      During the baseline reading the radiologist will mark the location of the actionable nodules
      and assign a score. The radiologist will also indicate the recommended method of follow-up
      (Contrast CT, PET-CT, CT Follow-up, Biopsy).

      During the second reading session (concurrent read), the radiologist will be presented with a
      standard appearing CT with computer-aided detection (CADe) marks placed and the vessel
      suppressed same slice with the vessel suppressed view (right image). The second image, vessel
      suppressed, will be grayed out until the radiologist move the mouse to the second panel. The
      radiologist will mark locations. These may or may not correspond to the locations of the CAD
      markers. As before, the radiologist will assign a level of suspicious to each mark and
      indicate the need, if any, of an additional diagnostic action (CT Follow-up, Contrast CT,
      PET-CT, or Biopsy).

      Based on the levels of suspicion for each nodule and the associated likelihood ratings, LROC
      curves will be constructed for both the baseline and the concurrent reads and the
      significance of any difference will be calculated. The recommendations for further action (CT
      Follow-up, Contrast CT, PET-CT, or Biopsy) will be used to calculate sensitivity and
      specificity, PPV and NPV.

      Number and types of cases:

      Retrospective lung CT image series from approximately 300 patients will be included in the
      study. Approximately one hundred (100) of the patients will have pathology confirmed cancers
      and approximately two hundred (200) of the patients will be CTs associated with normal
      patients. Also included as nodule images are those where the actionable nodule was not acted
      on at that time, but was detected and acted on based on a subsequent CT. These are the prior
      images where the nodule can be identified and its location is the same as on the "current"
      image confirmed by the radiologist expert panel (using a majority of three as the decision
      criterion).

      The selected sample, randomly selected from a larger pool of CT cases will be enriched in the
      following way:

        1. Lung nodules (cancers in this study) will be tumor size T1a (20 mm or less). The
           proportion of nodules 20 mm or less may be increased since this is where the
           investigators expect the major impact of this software to be.

        2. Non-Solid (ground glass) nodules will be added to the sample (based on availability) to
           determine the performance of the system on non-solid nodules. For this group, to have
           sufficient cases, the investigators may have to include benign (non-malignant) non-solid
           nodules.

        3. In this project, the investigators will perform a Machine Test of the ClearRead CT
           Insight algorithm followed by a reader performance evaluation study. Riverain will
           provide a system configured with the operating point set to be used for the reader
           studies and a configuration for an "open" system to be used for machine testing and FROC
           generation.

      Arm 1: a baseline read (no secondary content) and Arm 2: concurrent, CAD augmented read.

        1. st Arm: Do baseline (measure time, readers score regions according to action and
           suspiciousness) - mark all locations of concern

        2. nd Arm: Concurrent read (measure time, readers score regions according to action and
           suspiciousness) - mark all locations of concern

      The primary study hypothesis is that the adjunctive use of ClearRead CT Insight is superior
      to use of standard lung CT images alone, as measured by the area under the LROC curve.

      STATISTICAL ANALYSES Accuracy

      To evaluate the hypothesis of superiority in terms of improvement in accuracy for ClearRead
      CT Insight vs. unaided, a mixed effects model (DBM) will be implemented (similar to the model
      outlined in Dorfman, Berbaum and Metz, 1997), where variance components will be included to
      account for reader, case, reader by case, reader by modality, case by modality and reader by
      case by modality. However, it is anticipated that the three-way interaction will be
      inestimable and will subsequently be dropped from the statistical model. Specifically, the
      hypothesis to test for superiority of ClearRead CT Insight vs. unaided is:

      H0: AUCunaided - AUCClearRead CT Insight ≥ 0.0 vs. HA: AUCunaided - AUCClearRead CT Insight <
      0

      The AUC of the LROC is the primary endpoint to evaluate accuracy and the test of interest
      will be a two-sided 95% confidence interval on the effect of modality (i.e. ClearRead CT
      Insight minus unaided). Significance will be concluded if the upper bound of the two-sided
      95% confidence interval does not include zero. If the null hypothesis (H0) is rejected, the
      alternative hypothesis (HA) is accepted and the superiority of using the ClearRead CT Insight
      system will be established.

      Time

      The second co-primary objective is to evaluate reduction in time spent per image for
      ClearRead CT Insight vs. unaided. Specifically, the hypothesis to test for superiority of
      ClearRead CT Insight vs. unaided is:

      H0= Tunaided - TClearRead CT Insight ≤ 0 vs. HA= Tunaided - TClearRead CT Insight > 0

      To evaluate the hypothesis of non-inferiority in terms of improved read time for ClearRead CT
      Insight vs. unaided, a mixed effects model will be implemented (similar to the model outlined
      in Dorfman, Berbaum and Metz, 1992), where variance components will be included to account
      for reader, case, reader by case, reader by modality, case by modality and reader by case by
      modality. However, it is anticipated that the three-way interaction will be inestimable and
      will subsequently be dropped from the statistical model.

      The read times will be tested using a two-sided 95% confidence interval on the effect of
      modality (i.e. ClearRead minus unaided). Significance will be concluded if the upper bound of
      the two-sided 95% confidence interval does not include zero.

      With either analysis, the use of the mixed model could be modified to employ a bootstrap
      sampling approach if the model assumptions of the DBM method have been violated. The upper 95
      % confidence limit for the difference in the areas under the curves would be calculated using
      10,000 bootstraps in the MultiReader MultiCase (MRMC) ROC method.

      POWER AND SAMPLE SIZE

      The power to detect differences in the AUC of the LROC curve for the proposed statistical
      analysis using the current design baseline, i.e. 300 cases, each corresponding image read by
      10 readers, was assessed through a simulation study. Specifically, the model outlined above
      was used to simulate 500 datasets across a range of effect sizes, where power was defined as
      the proportion of datasets that yielded a significant p-value for testing the fixed effect of
      modality. The method of Dorfman, Berbaum and Metz, 1992 (DBM) was implemented utilizing
      statistical mixed model theory with jackknife estimates.

      These simulations required assumptions regarding the magnitude of the variance components
      associated with the different random effects. As a pilot study to obtain estimates of
      variance components was not conducted, variance component estimates from Riverain study
      SoftView 510(k) (Record # BSSI-PR-09-00006) that uses a similar technology as the proposed
      ClearRead CT Insight device were used in all power simulations. Please note that the variance
      components were scaled by the total variance to represent the proportion of total variance
      explained by each component. All power estimates are dependent upon the appropriateness of
      the assumed variance components.
    

Intervention name(s): ClearRead CT Insight
Intervention type(s): Device
Intervention description(s): During the second reading session (concurrent read), the radiologist will be presented with a standard appearing CT with CADe marks placed and the vessel suppressed same slice with the vessel suppressed view as the intervention. The second image, vessel suppressed, will be grayed out until the radiologist move the mouse to the second panel. The radiologist will mark locations. These may or may not correspond to the locations of the CAD markers. As before, the radiologist will assign a level of suspicious to each mark and indicate the need, if any, of an additional diagnostic action (CT Follow-up, Contrast CT, PET-CT, or Biopsy).
Allocation: Randomized
Intervention model: Parallel Assignment
Masking: Double (Participant, Investigator)
Primary purpose: Diagnostic