## 5 Reference Databases
The study demonstrated that increasing reference database size and removing \"poorly effective\" references could significantly improve the CAD scheme performance [77]. The contradictory result was also reported in this research topic. One research group focused on developing new methods to recognize and eliminate superfluous and/or detrimental samples (ROIs) from the reference database to improve searching efficiency [78, 79]. The first study reduced size of the reference database from 1820 randomly selected ROIs to 600 ROIs with higher entropy [78]. The second study reported that by applying several intelligent reference ROI searching methods (including genetic algorithm, greedy selection, and a random mutation hill climbing), over 96% to 98% of ROIs stored in the previously optimized and reduced reference database could be further eliminated. The study reported that CAD performance maintained at the same level when using both the initially reduced reference databases with 600 ROIs and the new database including only 10 or 20 intelligently selected ROIs [79]. As a result, some researchers dedicated to build the large and diverse reference databases, while the other researchers suggested that assembling the small databases with intelligently selected cases was the better choice in developing CAD schemes. Besides several incomparable differences including (1) reference databases, (2) CBIR algorithms (multiple morphological features versus one similarity index of mutual information), and (3) overall achieved CAD performance levels ( \(A_{z}=0.87\) versus \(A_{z}=0.76\)), the reported studies in this research topic [77, 79] used different testing methods to test CAD performance changes. For example, one used an independent testing dataset [77], which means that although the reference database size was changed, the testing dataset remains the same; while the other used a leave-one-out testing or ten-fold cross-validation method [79], which means that as the reduction of reference database size, the number of testing ROIs also reduces. Since both testing methods could have potential bias when applying to the limited databases [80], it is clear that this issue needs to be further investigated. Based on the machine learning concepts [76] and results of previous studies [75, 77], one can conclude that in a limited reference database (or a multi-dimensional feature domain) the selected reference ROIs in the database are typically not uniformly distributed. As a result, the CBIR algorithms can retrieve reference ROIs that have very high level of similarity to some queried ROIs but relatively lower level of similarity to the other queried ROIs. All current CAD schemes using CBIR approaches compute the detection or classification score for a queried ROI only based on the relative ratios (scales) of the similarity levels between the selected positive and negative reference ROIs. To test the relationship between CAD performance and the actual similarity level between the queried ROI and the retrieved ROIs, the author and his colleagues recently applied a set of similarity thresholds to systematically remove the queried ROIs whose similarity level to their most \"similar\" \begin{table}
\begin{tabular}{|c|c|c|c|c|c|c|} \hline Reference & \multicolumn{4}{c|}{Originally (randomly) Selected Database} & \multicolumn{2}{c|}{“Optimized” Database} \\ \hline Number of & 630 & 1262 & 1591 & 2523 & 3153 & 2979 \\ ROIs & & & & & & \\ \hline Area under & 0.715 & 0.794 & 0.874 & 0.875 & 0.872 & 0.914 \\ ROC Curve & & & & & & \\ \hline Standard & 0.026 & 0.023 & 0.017 & 0.017 & 0.012 \\ Deviation & & & & & & \\ \hline \end{tabular}
\end{table} 
Table 1: The change of CAD performance as the increase of reference database size and eliminating a fraction of “poorly effective” reference ROIs (the “optimized” database). reference ROIs are less than the threshold. The study used a reference database involving 1500 ROIs depicting verified masses and 1500 ROIs depicting CAD-cued false-positive masses but actually negative breast tissue.