## 7 Assessment of CAD Performance  
The clinical relevance and visual similarity are the two common assessment indices to evaluate performance of CAD schemes using CBIR approaches. The clinical relevance indicates the scheme's capability of correctly detecting and/or classifying suspicious abnormalities (i.e., breast masses). It is commonly evaluated using either ROC method (i.e., comparing the areas under ROC curves) or FROC method (i.e., comparing detection sensitivity levels under a predetermined CAD operating threshold or a fixed false-positive detection rate). When the CAD provides \"visual aid\" to the radiologists, assessing the visual similarity level between the queried ROI and the retrieved similar reference ROIs becomes an important issue, which has great impact on radiologists' confidence of whether to accept or reject CAD-cued results in their decision making [39]. A number of research groups have investigated and applied different methods to assess the improvement of visual similarity using different CBIR approaches. Since visual similarity is a subjective concept, it often has large inter-observer variability [88]. To minimize the impact of the inter-observer variability, two evaluation methods namely, comparing with average subjective rating results provided by a group of observers (i.e., radiologists) and a two-alternative forced-choice (2 AFC) observer preference method, have been reported and applied to assess the visual similarity of CBIR selected similar images in developing CAD schemes. The first method aims to establish a \"ground-truth\" and then compare the \"absolute\" difference or correlation of similarity ratings between the CAD schemes and human observers. One research group has conducted a number of studies to search for better psychophysical similarity measure using this method [89]. In the study, a set of ROIs involving 60 pairs of breast masses was selected and five radiologists subjectively rated the similarity scores for these 60 pairs of masses (using normalized 0 to 1 scale). Then, an artificial neural network (ANN) was employed to learn the relationship between radiologists' average subjective similarity ratings and computer-extracted image features. The study showed that comparing to the commonly used similarity measure based on Euclidean distance, using ANN achieved substantially higher similarity correlation with the radiologists' subjective rating (\(r=0.798\) versus \(r=0.644\)). The second visual similarity evaluation method (2 AFC) only conducts the relative comparison of whether a new (or modified) searching algorithm achieved improved visual similarity [90]. In a 2 AFC observer preference or visual similarity assessment experiment, a panel of observers independently reviews two sets of \"similar\" reference ROIs that are selected by two CBIR algorithms and simultaneously displayed side-by-side on the image screen. Each observer is forced to make a decision on which set is more visually similar to the queried ROI. Using a statistical data analysis method (i.e., one sample test for a binomial proportion with correction for continuity [91]), the average observer preference results are then used to compute and determine whether the reference ROIs selected by two CBIR algorithms have significant difference in visual similarity levels [34; 41]. The researchers have also compared the correlations of subjectively determined similarity ratings obtained by these two assessment methods (the \"absolute\" similarity rating and the 2 AFC based \"relative\" similarity rating). Using a very limited testing dataset with eight pairs of masses and eight pairs of micro-calcification clusters, the study reported that the similarity rating scores of two assessment methods were highly correlated. The Pearson's correlation coefficients were 0.94 and 0.98 for rating similarity levels of masses and micro-calcification clusters, respectively [92]. This preliminary comparison result is very encouraging if it can be validated in future studies with large and/or different testing datasets.