SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
\begin{array}{l} \left\| \widehat {\Sigma} - \Sigma \right\| \lesssim \| \Sigma \| \left(\sqrt {\frac {\boldsymbol {r} (\Sigma) \log \boldsymbol {r} (\Sigma)}{\delta^ {2} n}} \vee \sqrt {\frac {t}{\delta^ {2} n}} \vee \frac {\boldsymbol {r} (\Sigma) (t + \log \boldsymbol {r} (\Sigma))}{\delta^ {2} n} \log (n)\right) + \frac {\varepsilon (1 - \delta) \sigma_ {\xi} ^ {2}}{\delta} \\ + \frac {(1 - \delta) \varepsilon}{\delta^ {2} \sqrt {| \log ((1 - \delta) \varepsilon) |}} \sigma_ {\xi} ^ {2} \left(\sqrt {\frac {p}{n}} \vee \frac {p}{n} \vee \sqrt {\frac {t}{n}} \vee \frac {t}{n}\right) \\ + D (\delta , p) \sqrt {\frac {t + \log (p)}{n}} + \sqrt {\delta (1 - \delta) \varepsilon \sigma_ {\xi} ^ {2} p} \sqrt {\operatorname {t r} (\Sigma)} \log (n) \frac {t + \log (p)}{n}, \\ \end{array}
$$
where  $D(\delta, p) = \sqrt{\frac{(1 - \delta)}{\delta^2} \varepsilon(p - 2) \sigma_{\xi}^2 \left[ 2 \| \Sigma \| + \sigma_{\xi}^2 \right] + \frac{(1 - \delta)}{\delta^3} \varepsilon \sigma_{\xi}^4 (|\mathrm{tr}(\Sigma) - \delta(p - 2)| + \| \Sigma \|)}$ .
See App F.3 for the proof. As emphasized in [20], the effective rank  $\mathbf{r}(\Sigma)$  provides a measure of the statistical complexity of the covariance learning problem in the absence of any contamination. However, when cell-wise contamination is present, the statistical complexity of the problem may increase from  $\mathbf{r}(\Sigma)$  to  $\mathbf{r}(\Lambda) = p$ . Fortunately, if the filtering process reduces the proportion of cell-wise contamination  $\varepsilon$  such that  $(1 - \delta)\varepsilon \mathrm{tr}(\Lambda) \leq \delta \mathrm{tr}(\Sigma)$  and  $\varepsilon \| \Lambda \| \leq \delta \| \Sigma \|$ , then we can effectively mitigate the impact of cell-wise contamination. Indeed, we deduce from Theorem 4 that
$$
\begin{array}{l} \left\| \widehat {\Sigma} - \Sigma \right\| \lesssim \| \Sigma \| \left(\sqrt {\frac {\boldsymbol {r} (\Sigma) \log \boldsymbol {r} (\Sigma)}{\delta^ {2} n}} \vee \sqrt {\frac {t}{\delta^ {2} n}} \vee \frac {\boldsymbol {r} (\Sigma) (t + \log \boldsymbol {r} (\Sigma))}{\delta^ {2} n} \log (n)\right) + \frac {\varepsilon (1 - \delta) \sigma_ {\xi} ^ {2}}{\delta} \\ + \frac {1}{\delta} \sqrt {(1 - \delta) \operatorname {t r} (\Sigma)} \sqrt {\frac {t + \log (p)}{n}} \left(\sqrt {\delta \sigma_ {\xi} ^ {2}} + \sqrt {\operatorname {t r} (\Sigma)} \log (n) \sqrt {\frac {t + \log (p)}{n}}\right), \tag {12} \\ \end{array}
$$
where we considered for convenience the reasonable scenario where  $\delta (p - 2)\geq \mathrm{tr}(\Sigma)$  and  $\sigma_{\xi}^{2}\geq \| \Sigma \|$ . The combination of the upper bound (12) with the lower bound in Theorem (3) provides the first insights into the impact of cell-wise contamination on covariance estimation.
In our experiments, MV refers either to the debiased MCAR covariance estimator (6) or to its MAR extension (8). The synthetic data generation is described in App. A. We also performed experiments on real life datasets described in App. B. All experiments were conducted on a 2020 MacBook Air with a M1 processor (8 cores, 3.4 GHz).
$^{1}$ Code available at https://github.com/klounici/COVARIANCE_contaminated_data
Table 1: Execution time of the covariance estimation procedures (in milliseconds) with  $n = 300$  averaged over all values of the contamination rate  $\delta$  and 20 repetitions.
method	p = 50	p = 100	p = 500
MV (ours)	0.29 ± 0.03	0.49 ± 0.08	9.7 ± 4.5
KNNImputer (KNN)	26 ± 9.8	45 ± 17	470 ± 190
IterativeImputer (II)	940 ± 350	2,800 ± 900	3.7 × 10^5 ± 1.1 × 10^5
Gain	6,900 ± 480	1.1 × 10^4 ± 250	8.8 × 10^4 ± 1.1 × 10^3
MIWAE	5.1 × 10^4 ± 2.8 × 10^3	6.7 × 10^4 ± 550	1.77 × 10^5 ± 5.8 × 10^3
Figure 2: Estimation error on a synthetic dataset with  $p = 50$ ,  $n = 300$ ,  $r(\Sigma) = 5$ .
Figure 3: Estimation error on a synthetic dataset with  $p = 500$ ,  $n = 300$ ,  $r(\Sigma) = 5$ .
5.1 Missing Values
We compared our method to popular imputations methods: KNNImputer (KNNI), which imputes the missing values based on the k-nearest neighbours [39], and IterativeImputer (II), which is inspired by the R package MICE [42], as coded in sklearnn [30]; and two recent GANs-based imputation methods MIWAE [28] and GAIN [47] as found in the package hyperimpute [15]. The deep methods were tested using the same architectures, hyperparameters and early stopping rules as their respective papers.
In Figures 2, 3 and Table 1 we compare our estimator MV defined in (6) to these imputation methods combined with the usual covariance estimator on synthetic data (see App. A for details of data generation) in terms of statistical accuracy and execution time. First, MV beats all other methods in low-dimensional scenarios and maintains a competitive edge with II in high-dimensional situations when the missing data rate remains below  $30\%$ . Furthermore, it stands as the second-best choice when dealing with missing data rates exceeding  $35\%$ . Next, MV has by far the smallest execution time down several orders of magnitude while the execution time of II increases very quickly with the dimension and can become impractical (see Figure 9 for a dataset too large for II). Overall, the procedures MV and II perform better than MIwAE and GAIN in this experiment. Our understanding is that MIwAE and GAIN use training metrics designed to minimize the entrywise error of imputation. We suspect this may be why their performances for the estimation of covariance with operator norm are not on par with other minimax methods. An interesting direction would be to investigate whether training MIwAE and GAIN with different metrics may improve the operator norm performance.
We refer to App. E for more experiments in the MAR setting of [28, Annex 3] which led to similar conclusions. These results confirm that imputation of missing values is not mandatory for accurate estimation of the covariance operator. Another viable option is to apply a debiasing correction to the empirical covariance computed on the original data containing missing values. The advantage of this approach is its low computational cost even in high-dimension.
Table 2: We consider contaminated data following model (2) contaminated with a Dirac contamination of high intensity with  $\varepsilon = 1$  and for several values of  $\delta$  in a grid. For each  $\delta$ , we average the proportion of real data  $\hat{\delta}$  and contaminated data  $\hat{\varepsilon}$  after filtering over 20 repetitions. Values are displayed in percentages ( $\hat{\delta}$  must be high,  $\hat{\varepsilon}$  low)). STD stands for standard deviation.
CONTAMINATION RATE (1 - δ)	TAIL CUT	DDC 99%	DDC 90%
δ	STD	δ̂	STD	δ̂	STD	δ̂	STD	δ̂	STD	δ̂	STD
0.1 %	99.6	0.023	0.000	0.000	99.1	0.029	0.000	0.000	94.8	0.054	0.00	0.00
1%	98.8	0.027	0.000	0.000	98.2	0.037	0.000	0.00	94.3	0.102	0.00	0.00
5%	94.9	0.013	0.000	0.000	94.6	0.018	0.000	0.000	91.8	0.060	0.00	0.000
10%	90.0	0.004	0.000	0.000	89.9	0.016	0.00	0.000	88.2	0.109	0.000	0.000
20%	80.0	0.000	20.0	0.000	80.0	0.003	0.017	0.035	79.4	0.035	0.009	0.022
30%	70.0	0.000	30.0	0.000	70.0	0.001	3.48	2.19	69.9	0.015	2.930	2.31
5.2 Cell-wise contamination
Methods tested. Our baselines are the empirical covariance estimator applied without care for contamination and an oracle which knows the position of every outlier, deletes them and then computes the MV bias correction procedure (6). In view of Theorems [1] and [1], this oracle procedure is the best possible in the setting of cell-wise contamination. Hence, we have a practical framework to assess the performance of any procedure designed to handle cell-wise contamination.
The SOTA methods in the cell-wise contamination setting are the DI (Detection-Inputation) method [32] and the TSGS method (Two Step Generalised S-estimator) [2]. Both these methods were designed to work in the standard setting  $n > p$  but cannot handle the high-dimensional setting as we already mentioned. Nevertheless, we included comparisons of our methods to them in the standard setting  $n > p$ . The code for DI and TSGS are from the R packages cellwise and GSE respectively.
We combine the DDC detection procedure [34] to first detect and remove outliers with several estimators developed to handle missing values. Our main estimators are DDCMV (short for Detecting Deviating Cells Missing Values), which uses first DDC and then computes the debiaised covariance estimator [6] on the filtered data, and tailMV, which detects outliers through thresholding and then uses again [6]. But we also proposed to combine the DDC procedure with imputation methods KNNI, II, GAIN and MIWAE and finally compute the standard covariance estimator on the completed data. Hence we define four additional novel robust procedures which we call DDCKNN, DDCII, DDCGAIN and DDCMIWAE. To the best of our knowledge, neither the first approach combining filtering with debiasing nor the second alternative approach combining filtering with missing values imputation have never been tested to deal with cell-wise contamination. A detailed description of each method is provided in App. C
Outlier detection and estimation error under cell-wise contamination on synthetic data. We showed that the error of a covariance estimator under cell-wise contamination depends on the proportion of remaining outliers after a filtration. In Table 2 we investigate the filtering power of the Tail Cut and DDC methods in presence of Dirac contamination. We consider the cell-wise contamination setting ② in the most difficult case  $\varepsilon = 1$  which means that an entry is either correctly observed or replaced by an outlier (in other words, the dataset does not contain any missing value). For each values of  $\delta$  in a grid, the quantities  $\hat{\delta}$  and  $\hat{\varepsilon}$  are the proportions of true entries and remaining contaminations after filtering averaged over 20 repetitions. The DDC based methods are particularly efficient since the proportion of Dirac contamination drops from  $1 - \delta$  to virtually 0 for any  $\delta \geq 0.74$ . In Figures 1 and 4, we see that the performance of our method is virtually the same as the oracle OracleMV as long as the filtering procedure correctly eliminates the Dirac contaminations. As soon as the filtering procedure fails, the statistical accuracy brutally collapses and our DDC based estimators no longer do better than the usual empirical covariance. In Table 8 and Figure 5, we repeated the same experiment but with a centered Gaussian contamination. Contrarily to the Dirac contamination scenario, we see in Figure 5 that the statistical accuracy of our DDC based methods slowly degrades as the contamination rate increases but their performance remains significantly better than that of the usual empirical covariance.
Figure 4: Estimation error as a function of the contamination rate for  $n = 500$ ,  $p = 400$ ,  $\mathbf{r}(\Sigma) = 5$  and Dirac contamination.
Figure 5: Estimation error as a function of the contamination rate for  $n = 500$ ,  $p = 400$ ,  $\mathbf{r}(\Sigma) = 5$  and Gaussian contamination.
Figure 6: DI fails on ATTEND since the covariance matrix is approximately low rank. The dataset has only 8 features and the effective rank of its covariance matrix is below 2.
Figure 7: Woolridge's CEOsAL dataset fails both TSGS and DI with its dimension of 13 and effective rank of around 2.5.
5.3 The effect of cell-wise contamination on real-life datasets
We tested the methods on 8 datasets from sklearn and Woolridge's book on econometrics [46]. These are low dimensional datasets (less than 20 features) representing various medical, social and economic phenomena. We also included 2 high-dimensional datasets. See App. [B] for the list of the datasets.
One interesting observation is that the instability of Mahalanobis distance-based algorithms is not limited to high-dimensional datasets. Even datasets with a relatively small number of features can exhibit instability. This can be seen in the performance of DI on the Attend dataset, as depicted in Figure 6, where it fails to provide accurate results. Similarly, both TSGS and DI fail to perform well on the CEOSAL2 dataset, as shown in Figure 7 despite both datasets having fewer than 15 features.
On the Abalone dataset, once we have removed 4 obvious outliers (which are detected by both DDC and the tail procedure), all estimators reached a consensus with the non-robust classical estimator, meaning that this dataset provides a ground truth against which we can evaluate and compare the performance of robust procedures in our study. To this end, we artificially contaminate  $5\%$  of the cells at random in the dataset with a Dirac contamination and compare the spectral error of the different robust estimators. As expected, TSGS and all our new procedures succeed at correcting the error, however DI becomes unstable (see Table 3). DDC MIWAE is close to SOTA TSGS for cellwise contamination and DDC II performs better. We also performed experiments on two high-dimensional datasets, where our methods return stable estimates of the covariance (DDCMV99 and DDCMV95 are within  $\approx 3\%$  of each other) and farther away from the classical estimator (See Figures 8 and 9). Note also that DDCII's computation time explodes and even returns out-of-memory errors due to the high computation cost of II that we already highlighted in Table 1.
Figure 8: Relative spectral difference (in %) between covariance estimators on SP500 stock returns over 2021 and 2022. On high-dimensional data, DDCII becomes inconsistent with the other procedures, maybe because II does not scale well with dimension.
Figure 9: Relative spectral difference (in %) between between covariance estimators on NASDAQ stock returns over 2021 and 2022. Here, DDCII fails due to out-of-memory errors.
Table 3: Relative spectral difference (in %) between estimated covariance matrices on Abalone with  $5\%$  synthetic contamination  $(\delta = 0.95, \varepsilon = 1)$ . On the cleaned dataset, all the robust estimators are very close to the empirical covariance (relative differences  $< 5\%$ ), so we consider the empirical covariance matrix as the truth. Here the D I procedure fails probably due to numerical errors.
relative error to	Classical estimator	DDCMV99	DDCMV95	DDC II	DDC KNN	DDC MIWAE	DDC GAIN	TSGS	DI
Truth	12.8	4.12	6.81	1.70	2.06	3.46	5.06	3.06	8.85
std	0.45	0.29	0.26	0.10	0.092	0.18	0.35	0.21	1.48
Classical	-	13.1	14.3	13.0	13.0	12.9	13.1	13.4	14.9
DDCMV99	-	-	2.99	2.52	2.22	1.87	2.66	5.44	8.79
DDCMV95	-	-	-	5.27	5.03	4.04	3.71	8.28	9.99
DDC II	-	-	-	-	0.465	1.88	3.49	3.27	8.28
DDC KNN	-	-	-	-	-	1.58	3.19	3.46	8.15
DDC MIWAE	-	-	-	-	-	-	1.70	4.50	7.22
DDC GAIN	-	-	-	-	-	-	-	5.97	6.71
TSGS	-	-	-	-	-	-	-	-	6.94
In this paper, we have extended theoretical guarantees on the spectral error of our covariance estimators robust to missing data to the missing at random setting. We have also derived the first theoretical guarantees in the cell-wise contamination setting. We highlighted in our numerical experimental study that in the missing value setting, our debiased estimator designed to tackle missing values without imputation offers statistical accuracy similar to the SOTA IterativeImputer for a dramatic computational gain. We also found that SOTA algorithms in the cell-wise contamination setting often fail in the standard setting  $p < n$  for dataset with fast decreasing eigenvalues (resulting in approximately low rank covariance), a setting which is commonly encountered in many real life applications. This is due to the fact that these methods use matrix inversion which is unstable to small eigenvalues in the covariance structure and can even fail to return any estimate. In contrast, we showed that our strategy combining filtering with estimation procedures designed to tackle missing values produce far more stable and reliable results. In future work, we plan to improve our theoretical upper and lower bounds in the cell-wise contamination setting to fully clarify the impact of this type of contamination in covariance estimation.
Acknowledgements. This paper is based upon work partially supported by the Chaire Business Analytic for Future Banking and EU Project ELIAS under grant agreement No. 101120237.
[39] Olga Troyanskaya, Michael Cantor, Gavin Sherlock, Pat Brown, Trevor Hastie, Robert Tibshirani, David Botstein, and Russ B. Altman. Missing value estimation methods for DNA microarrays. Bioinformatics, 17(6):520-525, June 2001.
[40] Alexandre B. Tsybakov. Nonparametric estimators. In Alexandre B. Tsybakov, editor, Introduction to Nonparametric Estimation, Springer Series in Statistics, pages 1-76. Springer, New York, NY, 2009.
[41] John W. Tukey. The Ninther, a Technique for Low-Effort Robust (Resistant) Location in Large Samples. In H. A. David, editor, Contributions to Survey Sampling and Applied Statistics, pages 251-257. Academic Press, January 1978.
[42] Stef van Buuren and Karin Groothuis-Oudshoorn. Mice: Multivariate Imputation by Chained Equations in R. Journal of Statistical Software, 45:1-67, December 2011.
[43] Roman Vershynin. Introduction to the non-asymptotic analysis of random matrices, November 2011.
[44] Roman Vershynin. High-Dimensional Probability: An Introduction with Applications in Data Science. Cambridge Series in Statistical and Probabilistic Mathematics. Cambridge University Press, Cambridge, 2018.
[45] Vincent Q. Vu and Jing Lei. Minimax sparse principal subspace estimation in high dimensions. The Annals of Statistics, 41(6):2905-2947, December 2013.
[46] Jeffrey M. Wooldridge. Introductory econometrics: a modern approach / Jeffrey M. Wooldridge,... Cengage learning, 2016.
[47] Jinsung Yoon, James Jordan, and Mihaela van der Schaar. GAIN: Missing Data Imputation using Generative Adversarial Nets, June 2018.
[48] Shuhan Zheng and Nontawat Charoenphakdee. Diffusion models for missing value imputation in tabular data, March 2023.