# Label-Free Model Evaluation with Out-of-Distribution Detection
## 4 Experiments
### Experimental Result  
This paper introduces three possible methods to estimate the recognition accuracy, including the prediction-based method, rotation prediction and AutoEval. We report the estimations of these methods in Table 1. For the prediction-based method, two thresholds (i.e., \(\tau=0.8\) and \(0.9\)) are used. For comparison with the prediction-based method, we choose 0.8 and 0.9 as the threshold. The experimental results show that the error of the prediction-based method is higher when \(\tau=0.8\). As \(\tau\) increases to 0.9, the error decreases, but the results are still unsatisfactory. In addition, we find that nearly 60% of OOD images have a large maximumsoftmax score. OOD images should be considered incorrectly classified, but these images are considered correctly classified due to the large maximum softmax scores. Therefore, the results of the prediction-based method are not reliable on datasets containing OOD data. Perhaps we could choose a threshold to achieve better results, but this is like guessing the accuracy by chance rather than by understanding whether each image is correctly classified. For comparison with other label-free model evaluation methods, we compare our methods with the existing work. These methods include rotation prediction and AutoEval without using OOD detection. We conduct experiments using published code and compare our methods on datasets containing OOD data.The experimental results show that the average RMSE = 2.91% for our method on the eight datasets, much smaller than AutoEval without the OOD strategy (RMSE = 4.61%) and rotation prediction (RMSE = 4.18%). This suggests that our method outperforms existing methods in real environments containing OOD data. For the same OOD data, our method performs better on the four datasets with ID data as ImageNet than on the four datasets with ID data as CIFAR-10.1. This is because the distribution shift between CIFAR-10.1 and CIFAR-10.1 is smaller. The OOD classification error rate is smaller on CIFAR-10.1 than on ImageNet. We note that rotation prediction works better than AutoEval using the OOD strategy when the OOD data were SVHN. SVHN is significantly different from the other OOD datasets in that this dataset consists of grey-scale digital images, whereas the other datasets are all natural images. Because the classifier was trained on the natural dataset, the branching task of determining the rotation angle has a low accuracy on SVHN. For other OOD natural images, rotation prediction may maintain a certain level of accuracy with learned information (e.g., the animal's eye is above the mouth). Furthermore, we find that the error in our method comes from two sources: (1) misclassification of OOD detection; (2) the inherent error of the AutoEval method. The error introduced by the former can be reduced by selecting a better threshold. The latter can be reduced by selecting a better representation of the dataset. **Effect of OOD threshold. ** As mentioned earlier, the choice of OOD detection thresholds has a significant impact on the predictive results. Therefore, we analyzed the error of our method in different thresholds. We show the OOD detection error and prediction accuracy error under different thresholds in Figure 3. The following experiments were conducted on datasets with a 9:1 ratio of ID data to OOD data. The experimental results demonstrate that the impact of OOD detection thresholds on the accuracy prediction error comes from two main sources: (1) treating ID data as OOD data, which leads to the exclusion of ID data and directly affects the prediction error; (2) treating OOD data as ID data, resulting in the conflation of OOD data, which indirectly affects the prediction error by affecting the accuracy of AutoEval. Usually, both errors exist simultaneously, but the two errors have different effects on the prediction error at different thresholds. When the threshold \(\tau\) is set to 0.95, OOD detection directly marks nearly 5% of the ID data in the test set as classification errors (the actual impact will be