## 5 Performance Evaluation of Classification Algorithms
### How does the Number of Training Instances Influence the Performance of the Learning Machines?  
The dimensionality of the feature space influences the size of the training data to be captured in that the number of training instances per class should be significantly larger than the number of features. A ratio of at least ten times more training instances than features is recommended in the literature [14]. The following experiment aims at finding for each classifier the minimum number of training instances that is needed to produce stable and high average accuracy values. A similar experimental set up as described in Subsection 5.2 is used with the ten unique training datasets for each image type. The size of the feature vector has been adjusted for each classifier and image type individually based on the results of the experiments described in Subsection 5.2. The goal is to test the performance and consistency of the classifiers while the number of training instances per class is iteratively increased.  
Figure 6: Influence of the size of the feature vector on the classification accuracy for MSS. Mean and range of accuracy measures from the ten-fold cross-validation are shown for each iteration.  
Figure 7: ROC curves for the classification of QB, TM and MSS, depicting the performance of different classifiers under varying numbers of training instances.  
From Figure 7, it can be seen that all classifiers show a good overall performance on the different image types while the number of training instances is varied. Besides the slightly poorer performance of KNN on the QB dataset, the classifiers perform comparably well. Figures 8-10 show, however, that for each combination of classifier and image type an operating point exists where classification accuracies are maximized while the number of training instances is minimal. Using a smaller number of training instances than is defined at this operating point negatively affects the classification accuracies, whereas an increase in the number of training instances beyond this operating point flattens out the variations in the accuracy values and leads to generally high average accuracies for all classifiers on all datasets. The position of the optimal operating point and therefore the minimum number of instances needed to train a powerful classifier varies between the different learning algorithms and input image types. SVM, when trained on the QB dataset, needs the least number of training instances to be used.  
Figure 8: Influence of the training set size on the classification accuracy for QB. Mean and range of accuracy measures from the ten-fold cross-validation are shown for each iteration.  
Figure 9: Influence of the training set size on the classification accuracy for TM. Mean and range of accuracy measures from the ten-fold cross-validation are shown for each iteration.  
training instances (60 instances per class) to perform well. RT needs just slightly more instances (80 instances per class), whereas KNN (130 instances per class) and NB (220 instances per class) need significantly more to reach comparable accuracy values. For the TM dataset, SVM needs at least 110, RT 130, KNN 310 and NB 200 instances per class to reach the optimal operating point. When applied to the MSS dataset, SVM should be trained with at least 130, RT with 150, KNN with 230, and NB with 120 instances per class. Averaged over all image types, SVM needs the least number of training instances per class (80), followed by RT (120), KNN (127) and NB (162).