# Data Quality Measures and Efficient Evaluation Algorithms for Large-Scale High-Dimensional Data
## 3 Methods
### Methods for Efficient Computation
#### 3.3.2 Bootstrapping
### Class-Wise In-Class Variability Measure, \(M_{Var_{i}}\)

In fact, many of the existing data quality measures are designed to measure the difficulty of classification for a given dataset. However, we believe that the in-class variability of data must be considered as another important factor of data quality. One example to show the importance and usefulness of our in-class variability measure \(M_{var_{i}}\) is the generalization performance of a classifier. Figure 3: Sample size vs. computation time (CIFAR10). \begin{table}
\begin{tabular}{c c c c c c c} \hline
**Sample Size** & \(F1\) & \(N1\) & \(N3\) & \(CSG\) & \(M_{sep}\) (Ours) & Speedup \\ \hline
10,000 & 205 & 939 & 1904 & 2645 & 56 & \(\times 3.7\sim 47.2\) \\
20,000 & 411 & 3719 & 7797 & 5674 & 136 & \(\times 3.0\sim 57.3\) \\
30,000 & 605 & 8572 & 17,177 & 9065 & 220 & \(\times 2.8\sim 78.1\) \\
40,000 & 825 & 14,896 & 29,813 & 11,843 & 293 & \(\times 2.8\sim 101.8\) \\
50,000 & 1006 & 23,235 & 46,541 & 16,208 & 387 & \(\times 2.6\sim 120.3\) \\ \hline \end{tabular}
\end{table} 
Table 5: Sample size vs. computation time (CIFAR10) in detail (the values in the table represent seconds). The generalization performance of the learning model is an important consideration especially in mission-critical AI-augmented systems. There are many possible reasons causing low generalization, and overfitting is one of the troublemakers. Although we have techniques to alleviate overfitting, e.g., checking the learning curve, regularization [26; 27], and ensemble learning [28], it is critical to check if there is an issue in data to begin with which may lead to any inductive bias. For example, a very small value of \(M_{var_{i}}\) in a class compared to the others would indicate a lack of variability in the class, which can lead to low generalization due to, e.g., unchecked input noise, background signal, object occlusion, and angle/brightness/contrast differences during training. On the other hand, the overly large \(M_{var_{i}}\) may indicate outliers or even mislabeled data points likely to incur unwanted inductive bias in training. To show the importance and usefulness of in-class variability, we created a degraded version of CIFAR10 (denoted by degraded-CIFAR10) by reducing the variability of a specific class. The degraded-CIFAR10 is created by the following procedure. First, we chose an image of the deer class in the training data, then selected the nine mostly similar images in the angular distance to the chosen one. Figure 4 shows the total ten images selected by the above procedure that have similar backgrounds and shapes. Next, we created 1000 images by sampling with replacement from the ten images, while adding random Gaussian noise with zero mean and unit variance, and we replaced the original deer class data with sampled degraded deer class data. Table 7 shows that the value of \(M_{var_{i}}\) is significantly small on the degraded deer class compared to the other classes. That is, it can capture small in-class variability. In contrast, Table 8 shows that the existing quality measures \(F1\), \(N1\), \(N3\), and \(CSG\) may not be enough to signify the degradation of the dataset. As we can see, all quality measures indicate that class separability increased in degraded-CIFAR10 compared to the original CIFAR10; however, the test accuracy from MLP-2 decreased. This is because the reduction in in-class variability is very likely to decrease the generalization performance. Therefore, class separability measures can deliver incorrect information regarding data quality in terms of in-class variability, which can be a critical problem for generating a trustworthy dataset or training a trustworthy model.