Computer-Aided Diagnosis of Lung Nodules on CT Scans: . ROC Study of Its Effect on Radiologists' Performance

The use of computed tomographic (CT) imaging for lung cancer screening is an area of active research. CT imaging has been shown to be more sensitive for lung nodule detection than chest x-ray, especially for smaller nodules (1–4). Henschke et al (5) reported a 92% survival rate among patients who underwent surgical resection for detected stage I lung cancers. Sobue et al (6) reported an almost 100% 5-year survival rate for patients with nodules 50,000 enrolled patients, will provide more definitive results as to whether early detection with CT imaging compared to CXR will lead to reduced patient mortality. The higher sensitivity of CT imaging results in an increase in the number of nodules detected and thus an increase in the nodules that need to be followed up and managed. This may require expensive diagnostic tests such as follow-up CT scans and biopsies. Multiple–detector row CT technology has resulted in thinner slices and higher resolution. However, the large number of images that radiologists must interpret greatly increases their workload. Despite higher quality images, Swensen et al (8) reported that as many as 50% of nodules resected at surgery are benign, signifying the difficulty radiologists have in determining whether a lung nodule is malignant or not on the basis of CT and other clinical information. This emphasizes the importance of developing methods that can assist radiologists in characterizing nodules. Computer-aided detection and diagnosis software is being developed to address these issues. Computer-aided detection has been shown to increase the sensitivity of lung nodule detection (9–14). Computer-aided diagnosis (CAD) methods for the classification of lung nodules as malignant or benign have been reported by a number of investigators, with the area under the receiver-operating characteristic (ROC) curve, Az, ranging from 0.79 to 0.92 (15–23). A few observer studies have been performed to evaluate the effects of CAD on radiologists’ assessments of the malignancy of lung nodules on CT images. Matsuki et al (24) performed a study with four radiologists, four fellows, and four residents reading a data set of 25 malignant and 25 benign nodules. They found that the average Az of the entire group increased significantly from 0.831 to 0.959 (P < .001) with CAD. The average Az of each group of observers also improved significantly, and the differences in performance among the three groups were reduced. Shah et al (25) conducted a study with eight radiologists reading 28 nodules (15 malignant, 13 benign) and obtained a significant improvement in the average Az from 0.75 to 0.81 (P = .018) when a computer aid was used. Li et al (21) found that the average Az for 16 radiologists significantly increased from 0.785 to 0.853 (P = .016). In addition, they observed that CAD had a beneficial effect on 68% of their changed recommendations for a data set of 28 malignant and 28 benign nodules (26). Awai et al (27) reported a significant (P = .021) improvement for 19 observers from an average Az of 0.843 ± 0.097 to 0.924 ± 0.043 for 18 malignant and 15 benign nodules. A subgroup analysis showed that the nine radiology residents improved significantly as a group, but the improvement of 10 board-certified radiologists did not achieve statistical significance. Although the previous studies demonstrated a trend of improvement in radiologists’ classification accuracy with CAD, the data sets in those studies were small. In this study, we collected a relatively large data set of 256 (132 benign, 124 malignant) nodules including both primary and metastatic lung cancers and a range of CT parameters to evaluate the effect of our CAD system on radiologists’ estimates of the malignancy of lung nodules. The aim was to reveal the effect of CAD in an environment with a heterogeneous case mix in comparison to that of a homogeneous data set.