## 2 Basics of CAD systems
### Evaluation of CAD systems  
The performance of CAD systems is evaluated by similar test methods to the general pattern recognition systems. The purpose of the test is to assess the generalization and robustness of the systems. The major test methods are resubstitution test, k-fold cross-validation test, and bootstrap method [57]. The resubstitution test would be an initial test for the CAD system, where the training data is the same as the test data. In the k-fold cross-validation test, all lesion candidates or cases are randomly split into k datasets. The k-1 datasets are used for training of the CAD system, and one dataset is used for testing. The training and testing are repeated by k times. When the number of k is equal to the number of candidates or cases, we call the test a leave-one-out test. The leave-one-out 'by patient' test method is often used in the CAD field. All lesion candidates except for the candidates obtained from the same patient were used for training, and the one candidate left out was used for testing with the CAD system. However, either large bias or large variance could be included in the cross-validation test methods. To reduce the bias or variance, the bootstrap method has been used [61], where a number of fake data sets of the same size were generated as the given training set, and the data sets were used for estimation of pattern recognition systems' performances [57]. Prior to the evaluation test, the lesion candidates should be separated into the true positives and false positives by using a criterion based on \"truth\". For instance, in the CAD system for detection of intracranial aneurysms, we used a criterion such that an aneurysm was considered correctly detected if the pixel with the maximum voxel value in the candidate region was located within the diameter of the aneurysm, which was measured by neuroradiologists, for aneurysms smaller than 7.0 mm, and within the diameter of 7.0 mm for aneurysms larger than 7.0 mm [25]. Another example of a criterion is that if the centroid of a candidate region was in a \"truth\" region determined by neuroradiologists, the candidate region would be classified into a true positive [28]. Otherwise, the candidate region was considered as a false positive. The free-response receiver operating characteristic (FROC) and ROC curves are used for evaluation of the overall performance of the CAD systems for various operating points. The FROC curve shows the relationship between the sensitivity and the number of false positives, which can be obtained by thresholding a certain parameter of the CAD system or the output of the classifier. Figure 10 shows the FROC curve for the overall performance of a scheme from [PERSON] _et al._[26] for a database by use of LDA with the leave-one-out-by-patient test method. The SBDI is the shape-based difference image. From Figure 10, we can evaluate that the CAD system with the SBDI features is better than that without the features. It is very important for some CAD systems to evaluate not only the sensitivity of lesions, but also the accuracy of lesion segmentation. For example, MS lesions present temporal changes in the shape, location, and area between patients [3-6]. Therefore, neuroradiologists need to know the differences in the shape, location, and area between two studies, which should be provided by the CAD systems for the MS. For such CAD systems, the segmentation accuracy should be evaluated by using some indicators such as overlap measure [63], which means the degree of coincidence between the candidate region obtained by a method and the gold standard region by a manual method, as follows:  
\[\textit{Overlap measure (\%)}=\frac{n\big{(}T\cap C\big{)}}{n\big{(}T\cup C\big{)}} \times 100 \tag{2}\]  
where \(T\) was the gold standard region manually determined by radiologists, \(C\) was the region automatically determined by using a method, \(n(T\cup C)\) was the number of logical OR pixels between \(T\) and \(C\), and \(n(T\cap C)\) was the number of logical AND pixels between \(T\) and \(C\).