# Label-Free Model Evaluation with Out-of-Distribution Detection
## 4 Experiments
### Experimental Result
\begin{table}
\begin{tabular}{l c c c c c c c c} \hline \hline
**ID Data** & \multicolumn{4}{c}{**ImageNet**} & \multicolumn{4}{c}{**CIFAR-10**} \\ \hline
**OOD Data** & **ImageNet** & **SVHN** & **ISUN** & **LSUN** & **ImageNet** & **SVHN** & **ISUN** & **LSUN** \\ \hline Pred score (=0.8) & 14.11 & 13.61 & 13.43 & 12.78 & 9.62 & 9.12 & 8.94 & 8.29 \\ Pred score (=0.9) & 8.64 & 7.85 & 7.31 & 6.95 & 4.16 & 3.37 & 2.83 & 2.46 \\ \hline Rotation & 3.75 & 2.95 & 6.24 & 5.88 & 3.29 & 2.08 & 4.99 & 4.26 \\ AutoEval & 4.90 & 4.21 & 3.75 & 3.85 & 5.65 & 4.84 & 4.59 & 5.29 \\ \hline Ours & 2.56 & 3.91 & 3.39 & 3.02 & 3.01 & 3.73 & 2.07 & 1.71 \\ \hline \hline \end{tabular}
\end{table} 
Table 1: Performance of the various methods on eight datasets containing OOD data. These datasets all consist of 90% ID data and 10% OOD data. much larger). This results in a larger error for our method. At this threshold, the first type of error has a greater impact on the prediction error than the second type. As the threshold increases, the first type of the error decreases, while the second type of the error becomes larger. The error of our method decreases and then increases until the AutoEval using OOD detection degenerates into traditional methods due to the large threshold. **Different proportions of OOD data. ** To demonstrate the robustness of our method in datasets with large amounts of OOD data, we conducted experiments using images from CIFAR-10 and ImageNet as ID data, and SVHN and ImageNet as OOD data. We conducted experiments varying the percentage of OOD data between 20% and 40% while keeping all other parameters constant to demonstrate the robustness of our method in datasets with a large amount of OOD data. As shown in Figure 4, our method is almost unaffected by an increase in OOD data, while the error of traditional AutoEval without OOD detection shows a significant increase in error. The rotation prediction method is less affected by the increase in the proportion of OOD data but still performs much worse than our method. However, we observed that our method does not perform as well when the percentage of OOD data is low (<5%). This may be attributed to the incorrect exclusion of some in-distribution (ID) data by the OOD detection mechanism. **Ours method for VGG-16. ** We also conducted experiments on VGG-16. The performance of each method is shown in Table 2. All experiment settings are the same as before. Figure 4: The error changes of our method under different proportions of OOD data. When the proportion of OOD data is low, our method only shows a slight improvement over other methods. However, as the proportion of OOD data increases, our method’s superiority over other methods becomes more prominent. The dataset consists of ID data and OOD data from CIFAR-10.1/ImageNet (**a**) and ImageNet/SVHN (**b**). Figure 3: (**a**): The vertical axis represents the error between the model accuracy predicted by our method and the true accuracy of the classifier, while the horizontal axis represents the OOD detection threshold. This error decreases initially and then increases as the OOD threshold increases. (**b**): The vertical axis represents the error of OOD detection, while the horizontal axis represents the OOD detection threshold. This error decreases as the OOD threshold increases. The ID/OOD data in the dataset comes from CIFAR-10.1/ImageNet and ImageNet/SVHN, respectively. The ratio of ID data to OOD data is 9:1.