# Data Quality Measures and Efficient Evaluation Algorithms for Large-Scale High-Dimensional Data
## 3 Methods
### Methods for Efficient Computation
#### 3.3.2 Bootstrapping
\begin{table}
\begin{tabular}{c c c c c} \hline \hline
**Datasets** & **Accuracy** & \(M\) & **No. Classes** & **Description** \\ \hline MNIST & 92.64\% & 12.5 k & 10 & Hand written digit \\ notMNIST & 89.24\% & 12.5 k & 10 & Fonts and glyphs similar to MNIST \\ Linnaeus & 45.50\% & 4.8 k & 5 & Botany and animal class images \\ CIFAR10 & 42.84\% & 12.5 k & 10 & Object recognition images \\ STL10 & 40.88\% & 12.5 k & 10 & Object recognition images \\ SVHN & 45.60\% & 12.5 k & 10 & House number images \\ ImageNet-1 & 37.40\% & 5 k & 10 & Visual recognition images ( Tiny ImageNet) \\ ImageNet-2 & 40.60\% & 5 k & 10 & Visual recognition images ( Tiny ImageNet) \\ ImageNet-3 & 34.20\% & 5 k & 10 & Visual recognition images ( Tiny ImageNet) \\ ImageNet-4 & 36.20\% & 5 k & 10 & Visual recognition images ( Tiny ImageNet) \\ \hline \hline \end{tabular}
\end{table} 
Table 1: Details of the datasets used in our experiments. The accuracy is from MLP-2, and \(M\) represents the total number of data used for training and evaluation. #### 4.2.1 Correlation with Classifier Accuracy

To demonstrate how well \(M_{sep}\) represents the class separability of given datasets, we compared the absolute value of Pearson correlation and Spearman rank correlation between quality measures \(M_{sep}\), \(N1\), \(N3\), \(F1\), and \(CSG\) to the prediction accuracy of three classification models: perceptron, MLP-1, and MLP-2. Table 2 summarizes the results. In the case of the perceptron, \(M_{sep}\) has a similar Pearson correlation with the shortest computation time to the \(N1\) and \(N3\) which have the highest correlation with the accuracy of classifiers. Furthermore, \(M_{sep}\) and \(F1\) have the highest Spearman rank correlation. This is because \(M_{sep}\) and \(F1\) measure linear separability that is essentially the information captured by the linear classifier, the perceptron in our case. In the case of MLP-1 and MLP-2, \(M_{sep}\) also showed a sufficiently high correlation with classification accuracy although it is slightly lower in Pearson correlation compared to the case of the perceptron. On the other hand, \(CSG\) does not seem to have noticeable benefits considering its computation time. This is because \(CSG\) is affected by an embedding network which requires a large amount of training time. In summary, the result shows that our measure \(M_{sep}\) can capture separability of data as good as the existing data quality measures, while reducing computation time significantly. #### 4.2.2 Correlation with Other Quality Measures

In order to check if our suggested data quality measure \(M_{sep}\) is compatible with the existing ones in quality, and therefore ours can be a faster alternative to the existing data quality measures, we computed the Pearson correlation between \(F1\), \(N1\), \(N3\), \(CSG\), and \(M_{sep}\). The results are summarized in Table 3. Our measure \(M_{sep}\) showed a high correlation with all four existing measures \(F1\), \(N1\), \(N3\), and \(CSG\), indicating that \(M_{sep}\) is able to capture the data quality information represented by \(F1\), \(N1\), \(N3\), and \(CSG\). \begin{table}
\begin{tabular}{c c c c c} \hline \hline
**Classifier** & **Quality Measure** & **Pearson Corr.** & **Spearman Corr.** & **Time (s)** \\ \hline \multirow{8}{*}{Perceptron} & \(F1\) & 0.9386 & 0.7697 & 1253 \\  & \(N1\) & 0.9889 & 0.7333 & 5104 \\  & \(N3\) & 0.9858 & 0.7333 & 9858 \\  & \(CSG\) & 0.9452 & 0.8182 & 23,711 \\  & \(M_{sep}\) (ours) & 0.9693 & 0.8061 & 354 \\ \hline \multirow{8}{*}{MLP-1} & \(F1\) & 0.9039 & 0.3455 & 1253 \\  & \(N1\) & 0.9959 & 0.9758 & 5104 \\  & \(N3\) & 0.9961 & 0.9030 & 9858 \\  & \(CSG\) & 0.9295 & 0.5879 & 23,711 \\  & \(M_{sep}\) (ours) & 0.9261 & 0.3818 & 354 \\ \hline \multirow{8}{*}{MLP-2} & \(F1\) & 0.8855 & 0.3455 & 1253 \\  & \(N1\) & 0.9908 & 0.9273 & 5104 \\ \cline{1-1}  & \(N3\) & 0.9912 & 0.8788 & 9858 \\ \cline{1-1}  & \(CSG\) & 0.9127 & 0.5879 & 23,711 \\ \cline{1-1}  & \(M_{sep}\) (ours) & 0.9117 & 0.4303 & 354 \\ \hline \hline \end{tabular}
\end{table}