SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Figure 22 provides the feature shift correction metrics for DF-Correct when using different classifiers as discriminators within the iterative process on both real and simulated datasets. Similar to DF-Locate, tree-based methods provide similar results, surpassing the linear methods in most metrics. We use  $D_{hp}$  as a metric to select our method because it provides an estimate that tightly bounds the total variation distance. Namely, we select CatBoost as our discriminator as it provides competitive performance with the other tree-based methods in the simulated datasets, and clearly outperforms the others in the real datasets.
Figure 22: Mean  $W_2^2$ ,  $D_{hp}$ , and  $D_{skl}$  by DF-Correct using different classifiers on simulated (top) and real (bottom) datasets. Lower is better.
L Variability Analysis for Location and Correction
Conducting the whole evaluation benchmark for both localization and correction multiple times is infeasible and expensive. However, to provide insight into the variation among runs for each method, we present plots showcasing the variability of detection and correction methods with multiple random seeds for a given dataset, namely the Energy dataset.
In Figure 23, we present the variability of detection methods, conveyed by the mean and standard deviation of the F-1 score across five different seeds on the Energy dataset. This evaluation covers scenarios involving  $25\%$  corrupted features and all manipulation types. Similarly, Figure 24 provides insights on the variability of correction methods, conveyed by the mean and standard deviation of  $W_2^2$ ,  $D_{hp}$ , and  $D_{skl}$  computed across five different seeds on the Energy dataset. All detection and correction methods, particularly DF-Locate and DF-Correct, exhibit a low variability across different seeds.
Figure 23: Variability of detection methods on the Energy dataset. Mean and standard deviation of F-1 score across five different seeds, for  $25\%$  corrupted features and all manipulation types. Higher is better.
Figure 24: Variability of correction methods on the Energy dataset. Mean and standard deviation of  $W_{2}^{2}$ ,  $D_{hp}$ , and  $D_{skl}$  across five different seeds. Lower is better.
M Evaluation of Downstream Classification and Regression Tasks after Correction
In this section, we include an evaluation of downstream classification and regression tasks for the Musk2 and Energy datasets, which include categorical and continuous labels, respectively.
We train the downstream models with the corrected query datasets and evaluate their performance on the reference dataset. We perform the experiment using each query dataset version, which includes the original pre-corrupted dataset, and the correct datasets with each correction method. The evaluation of the classifier/regressor is carried out under three different scenarios: when all features are used (100%), when only 50% of the non-corrupted features are included, and when 0% of non-corrupted features are included (only the corrected features are used). The classification results are reported in Table 6, and the regression results are reported in Table 7, using balanced accuracy for classification and root mean square error (RMSE) for regression. Methods are sorted by their performance without non-corrupted features (0%). Notably, the classification/regression downstream performance when using DataFix surpasses the competing methods in most settings, in some cases even surpassing the performance of directly using the original query dataset (ground truth pre-corruption). This observation demonstrates that the class-conditional divergence between the reference and the corrected query with DataFix is small.
Table 6: Downstream classification task with Musk2 dataset, with all corrected features and all non-corrupted features (100%), with half of the non-corrupted features (50%), and without non-corrupted features (0%).
Method	Balanced Acc. (0%)	Balanced Acc. (50%)	Balanced Acc. (100%)
Original Query	0.951	0.960	0.964
DataFix	0.902	0.953	0.957
LR	0.871	0.932	0.938
HyperImpute	0.863	0.932	0.945
ICE	0.848	0.937	0.937
MIRACLE	0.828	0.920	0.937
Sinkhorn	0.818	0.924	0.936
MLP	0.817	0.907	0.911
KNN	0.815	0.925	0.934
INB	0.812	0.950	0.954
MissForest	0.779	0.908	0.944
DD	0.576	0.950	0.956
GAIN	0.574	0.910	0.939
Mean	0.500	0.952	0.961
SoftImpute	0.493	0.620	0.836
Table 7: Downstream regression task with Energy dataset, with all corrected features and all non-corrupted features (100%), with half of the non-corrupted features (50%), and without non-corrupted features (0%).
Method	RMSE (0%)	RMSE (50%)	RMSE (100%)
DataFix	0.089	0.082	0.080
Original Query	0.093	0.084	0.082
KNN	0.105	0.094	0.091
Sinkhorn	0.107	0.090	0.085
INB	0.114	0.094	0.092
Mean	0.122	0.088	0.083
GAIN	0.127	0.200	0.279
HyperImpute	0.129	0.124	0.123
DD	0.137	0.120	0.116
MissForest	0.147	0.155	0.156
MLP	0.175	0.187	0.200
SoftImpute	0.180	0.182	0.252
LR	0.191	0.189	0.205
ICE	0.217	0.225	0.234
MIRACLE	0.311	0.335	0.343
N End-to-end DataFix Analysis
Figure 25 provides five additional examples that illustrate the iterative process of DF-Locate before and after shift correction, in simulated datasets 1, 2, 3, 6, and 10. In each dataset, there are 200 corrupted features out of 1000. Similar to Figure 5, the left column in Figure 25 displays the TVD estimated by the random forest (blue), which provides a lower bound for its ground truth Monte Carlo estimate (black), as the iterative process detects and removes corrupted features. The F-1 detection score progressively increases until all corrupted features are identified, leading to the termination of the iterative process. The right column in Figure 25 showcases the iterative process applied to the corrected query using different methods.
In simulated datasets 1 and 2, the methods DD, INB, MIRACLE and MLP all yield an updated query that results in a lower empirical divergence. Notably, DD stands out by providing a corrected query with no empirical divergence detected by DF-Locate. However, DD produces queries with considerably higher empirical divergence in the case of the other simulated datasets when compared to DF-Correct, and is not able to lower the empirical divergence for simulated dataset 10. Moving to simulated dataset 6, INB and DD yield an updated query that reduces the empirical divergence, with both techniques achieving an empirical divergence that is almost negligible. The other shift correction benchmarking methods generate an updated query that increases the shift instead of reducing it. In simulated dataset 10, none of the benchmarking methods are able to generate an updated query that leads to a lower empirical divergence. Remarkably, DF-Correct (Purple) provides a precisely corrected query with no empirical divergence detected by DF-Locate for all datasets.
O DataFix Limitations
DataFix is specifically tailored for tabular datasets and will not work optimally when applied to other data types such as images, videos, audio, speech, or textual data. DataFix also exhibits limitations due to its computational cost, preventing scalability for online or streaming scenarios, and potentially resulting in reduced speed with very large datasets, although it generally outperforms competing methods in terms of computational efficiency.
Figure 25: DF-Locate iterative process before (left) and after (right) shift correction. Each row corresponds to a different simulated dataset.