## 5 Reference Databases  
It is well known that performance and robustness of the CAD schemes that use machine learning classifiers also depends on the size of training databases and the case difficulty level [73; 74]. In general, as the increase of training database size and diversity, the performance and the robustness of the CAD schemes improve when they are tested using the independent testing datasets [75]. CAD schemes using CBIR approaches share two common characteristics. First, they use the \"lazy\" machine learning methods in that the decision of how to generalize beyond the training data is deferred until a new query (instance) is observed. Second, the new query is classified by comparing to and analyzing a small set of similar instances while ignoring others that are quite different from the query. One advantage of CAD schemes using CBIR approaches is that increasing (or updating) reference databases is relatively easy without repeating the often complicated process to re-generate the \"global\" optimization function. The new image data collected from the clinical practice may be directly added to the reference database to gradually increase the size and diversity of the reference database at any time. However, CAD schemes using CBIR approaches also have a major disadvantage when using the limited reference databases. Unlike the CAD schemes using \"global\" optimization based machine learning classifiers (i.e., artificial neural networks and support vector machines) that are relatively robust (insensitive) to the local image noise, the CAD schemes using CBIR approaches are much more sensitive to the local image noise and feature selection [76]. Study has shown that when applying to the same training database and the same independent testing dataset, the CAD scheme using a \"global\" data-based machine learning classifier achieved substantially higher performance than the CAD scheme using a CBIR approach (i.e., 75.8% versus 65.9% detection sensitivity at 0.3 false-positives per image) [37]. To investigate the relationship between the CAD performance and the selection of the reference databases, a number of studies has been recently conducted and reported. In one study, the researchers assembled a reference database that includes 3153 ROIs depicting either malignant masses (1592) or CAD-cued false-positive regions (1561) and an independent testing dataset involving 400 ROIs depicting 200 masses and 200 false-positive regions. A CAD scheme using a distance-weighted k-nearest neighbor (KNN) algorithm based CBIR approach was applied to retrieve ROIs from the reference database that are considered the \"most similar\" to each queried ROI of the testing dataset. The area under ROC curve (_A\({}_{z}\)_) was used as a measurement index to evaluate the CAD performance. The study included two experiments to investigate (1) the relationship between CAD performance and size of reference database and (2) whether randomly adding new cases (ROIs) into the reference database could always improve CAD performance. In the first experiment, the researchers systematically increased the size of reference database from 630 ROIs to 3153 ROIs and then tested the change of CAD performance. In the second experiment, based on the hypothesis that an ROI should be removed if it performs poorly compared to a group of similar ROIs in a large and diverse reference database, the researchers applied a strategy to identify and remove the \"poorly effective\" references from the database. The experimental results (Table 1) indicated that scheme performance monotonically improved from _A\({}_{z}=0.715\)_ to 0.874 and then plateau happened when the database size reached approximately half of its maximum capacity. By removing 174 identified \"poorly effective\" ROIs from the reference database (reducing the original 3153 ROIs to 2979 ROIs), CAD performance significantly increases to _A\({}_{z}=0.914\)_ (_p_\(<\) 0.01).