Title: Robust covariance estimation with missing values and cell-wise contamination
Venue: NeurIPS 2023
Accepted: yes
Keywords: robust statistics

missing values

cell-wise contamination
Abstract: Large datasets are often affected by cell-wise outliers in the form of missing or erroneous data. However, discarding any samples containing outliers may result in a dataset that is too small to accurately estimate the covariance matrix. Moreover, the robust procedures designed to address this problem require the invertibility of the covariance operator and thus are not effective on high-dimensional data. In this paper, we propose an unbiased estimator for the covariance in the presence of missing values that does not require any imputation step and still achieves near minimax statistical accuracy with the operator norm. We also advocate for its use in combination with cell-wise outlier detection methods to tackle cell-wise contamination in a high-dimensional and low-rank setting, where state-of-the-art methods may suffer from numerical instability and long computation times. To complement our theoretical findings, we conducted an experimental study which demonstrates the superiority of our approach over the state of the art both in low and high dimension settings.

Robust covariance estimation with missing values and cell-wise contamination

Abstract

Large datasets are often affected by cell-wise outliers in the form of missing or erroneous data. However, discarding any samples containing outliers may result in a dataset that is too small to accurately estimate the covariance matrix. Moreover, the robust procedures designed to address this problem require the invertibility of the covariance operator and thus are not effective on high-dimensional data. In this paper, we propose an unbiased estimator for the covariance in the presence of missing values that does not require any imputation step and still achieves near minimax statistical accuracy with the operator norm. We also advocate for its use in combination with cell-wise outlier detection methods to tackle cell-wise contamination in a high-dimensional and low-rank setting, where state-of-the-art methods may suffer from numerical instability and long computation times. To complement our theoretical findings, we conducted an experimental study which demonstrates the superiority of our approach over the state of the art both in low and high dimension settings.

1 Introduction

Outliers are a common occurrence in datasets, and they can significantly affect the accuracy of data analysis. While research on outlier detection and treatment has been ongoing since the 1960s, much of it has focused on cases where entire samples are outliers (Huber's contamination model) [11, 41, 13]. While sample-wise contamination is a common issue in many datasets, modern data analysis often involves combining data from multiple sources. For example, data may be collected from an array of sensors, each with an independent probability of failure, or financial data may come from multiple companies, where reporting errors from one source do not necessarily impact the validity of the information from the other sources. Discarding an entire sample as an outlier when only a few features are contaminated can result in the loss of valuable information, especially in high-dimensional datasets where samples are already scarce. It is important to identify and address the specific contaminated features, rather than simply treating the entire sample as an outlier. In fact, if each dimension of a sample has a contamination probability of ε , then the probability of that sample containing at least one outlier is given by 1 -(1 -ε ) p , where p is the dimensionality of the sample. In high dimension, this probability can quickly exceed 50% , surpassing the breakdown point of many robust estimators designed for the Huber sample-wise contamination setting. Hence, it is crucial to develop robust methods that can handle cell-wise contaminations and still provide accurate results.

The issue of cell-wise contamination, where individual cells in a dataset may be contaminated, was first introduced in [3]. However, the issue of missing data due to outliers was studied much earlier, dating back to the work of [35]. Although missing values in a dataset are much easier to detect than outliers, they can lead to errors in estimating the location and scale of the underlying distribution [25]

0

.

00

0

.

05

0

.

10

0

.

15

0

.

20

0

.

25

0

.

30

Dataset contamination rate (1

δ

)

2

.

5

5

.

0

7

.

5

10

.

0

12

.

5

15

.

0

17

.

5

Operator error of covariance estimator

DI

TSGS

classical

tailMV

Estimator

Computation time

Matrix

p=50

p=100

inversion

tailMV

10

-

3

±

10

-

4

10

-

2

±

10

-

3

no

DDCMV

0

.

6

±

10

-

3

0

.

7

±

0

.

007

no

DI

6

±

0

.

5

74

±

4

yes

TSGS

20

±

0

.

2

200

±

10

yes

-

and can negatively affect the performance of supervised learning algorithms [17]. This motivated the development of the field of data imputation. Several robust estimation methods have been proposed to handle missing data, including Expectation Maximization (EM)-based algorithms [7], maximum likelihood estimation [14] and Multiple Imputation [25], among which we can find k-nearest neighbor imputation [39] and iterative imputation [42]. Recently, sophisticated solutions based on deep learning, GANs [47, 28, 9], VAE [27] or Diffusion schemes [48] have been proposed to perform complex tasks like artificial data generation or image inpainting. The aforementioned references focus solely on minimising the entrywise error for imputed entries. Noticeably, our practical findings reveal that applying state-of-the-art imputation methods to complete the dataset, followed by covariance estimation on the completed dataset, does not yield satisfactory results when evaluating the covariance estimation error using the operator norm.

In comparison to data missingness or its sample-wise counterpart, the cell-wise contamination problem is less studied. The Detection Imputation (DI) algorithm of [32] is an EM type procedure combining a robust covariance estimation method with an outlier detection method to iteratively update the covariance estimation. Other methods include adapting methodology created for Huber contamination for the cell-wise problem, such as in [6] or [2]. In high dimensional statistics, however, most of these methods fail due to high computation time and numerical instability. Or they are simply not designed to work in this regime since they are based on the Mahalanobis distance, which requires an inversion of the estimated covariance matrix. This is a major issue since classical covariance matrix estimators have many eigenvalues close to zero or even exactly equal to zero in high-dimension. To the best of our knowledge, no theoretical result exists concerning the statistical accuracy of these methods in the cell-wise contamination setting contrarily to the extensive literature on Huber's contamination [1].

Contributions. In this paper we address the problem of high-dimensional covariance estimation in the presence of missing observations and cell-wise contamination. To formalize this problem, we adopt and generalize the setting introduced in [10]. We propose and investigate two different strategies, the first based on filtering outliers and debiasing and the second based on filtering outliers followed by imputation and standard covariance estimation. We propose novel computationally efficient and numerically stable procedures that avoid matrix inversion, making them well-suited for high-dimensional data. We derive non-asymptotic estimation bounds of the covariance with the operator norm and minimax lower bounds, which clarify the impact of the missing value rate and outlier contamination rate. Our theoretical results also improve over [26] in the MCAR and no contamination. Next, we conduct an experimental study on synthetic data, comparing our proposed methods to the state-of-the-art (SOTA) methods. Our results demonstrate that SOTA methods fail in the high-dimensional regime due to matrix inversions, while our proposed methods perform well in this regime, highlighting their effectiveness. Then we demonstrate the practical utility of our approach by applying it to real-life datasets, which highlights that the use of existing estimation methods significantly alters the spectral properties of the estimated covariance matrices. This implies

that cell-wise contamination can significantly impact the results of dimension reduction techniques like PCA by completely altering the computed principal directions. Our experiments demonstrate that our methods are more robust to cell-wise contamination than SOTA methods and produce reliable estimates of the covariance.

2 Missing values and cell-wise contamination setting

Let X 1 , . . . , X n be n i.i.d. copies of a zero mean random vector X admitting unknown covariance operator Σ = E [ X ⊗ X ] , where ⊗ is the outer product. Denote by X ( j ) i the j th component of vector X i for any j ∈ [ p ] . All our results are non-asymptotic and cover a wide range of configurations for n and p including the high-dimensional setting p /greatermuch n . In this paper, we consider the following two realistic scenarios where the measurements are potentially corrupted.

Missing values. We assume that each component X ( j ) i is observed independently from the others with probability δ ∈ (0 , 1] . Formally, we observe the random vector Y ∈ R p defined as follows:

where d ij are independent realisations of a bernoulli random variable of parameter δ . This corresponds the Missing Completely at Random (MCAR) setting of [35]. Our theory also covers the more general Missing at Random (MAR) setting in Theorem 2.

Cell-wise contamination. Here we assume that some missing components X ( j ) i can be replaced with probability ε by some independent noise variables, representing either a poisoning of the data or random mistakes in measurements. The observation vector Y then satisfies:

where ξ ( j ) i are independent erroneous measurements and e i,j are i.i.d. bernoulli random variables with parameter ε . We also assume that all the variables X i , ξ ( j ) i , d i,j , e i,j are mutually independent. In this scenario, a component X ( j ) i is either perfectly observed with probability δ , replaced by a random noise with probability ε ′ = ε (1 -δ ) or missing with probability (1 -δ )(1 -ε ) . Cell-wise contamination as introduced in [3] corresponds to the case where ε = 1 , and thus ε ′ = 1 -δ .

In both of these settings, the task of estimating the mean of the random vectors X i is well-understood, as it reduces to the classical Huber setting for component-wise mean estimation. One could for instance apply the Tuker median on each component separately [3]. However, the problem becomes more complex when we consider non-linear functions of the data, such as the covariance operator. Robust covariance estimators originally designed for the Huber setting may not be suitable when applied in the presence of missing values or cell-wise contaminations.

We study a simple estimator based on a correction of the classical covariance estimator on Y 1 , . . . , Y n as introduced in [26] for the missing values scenario. The procedure is based on the following observation, linking Σ Y the covariance of the data with missing values and Σ the true covariance:

Note that this formula assumes the knowledge of δ . In the missing values scenario, δ can be efficiently estimated by a simple count of the values exactly set to 0 or equal to NaN (not a number). In the contamination setting (2), the operator Σ Y = E ( Y ⊗ Y ) satisfies, for Λ = E [ ξ ⊗ ξ ] :

In this setting, as one does not know the exact location and number of outliers we propose to estimate δ by the proportion of data remaining after the application of a filtering procedure.

Notations. We denote by /circledot the Hadamard (or term by term) product of two matrices and by ⊗ the outer product of vectors, i.e. ∀ x, y ∈ R d , x ⊗ y = xy /latticetop . We denote by ‖ . ‖ and ‖ . ‖ F the operator and Frobenius norms of a matrix respectively. We denote by ‖ · ‖ 2 the vector l 2 -norm.

3 Estimation of covariance matrices with missing values

We consider the scenario outlined in (1) where the matrix Σ is of approximately low rank. To quantify this, we use the concept of effective rank, which provides a useful measure of the inherent complexity of a matrix. Specifically, the effective rank of Σ is defined as follows

We note that 0 ≤ r ( Σ ) ≤ rank ( Σ ) . Furthermore, for approximately low rank matrices with rapidly decaying eigenvalues, we have r ( Σ ) /lessmuch rank ( Σ ) . This section presents a novel analysis of the estimator defined in equation (3), which yields a non-asymptotic minimax optimal estimation bound in the operator norm. Our findings represent a substantial enhancement over the suboptimal guarantees reported in [26, 18]. Similar results could be established for the Frobenius norm using more straightforward arguments, as those in [4] or [31]. We give priority to the operator norm since it aligns naturally with learning tasks such as PCA. See [19, 21, 22] and the references cited therein.

We need the notion of Orlicz norms. For any α ≥ 1 , the ψ α -norms of a real-valued random variable V are defined as: ‖ V ‖ ψ α = inf { u > 0 , E exp( | V | α /u α ) ≤ 2 } . A random vector X ∈ R p is sub-Gaussian if and only if ∀ x ∈ R p , ‖〈 X,x 〉‖ ψ 2 /lessorsimilar ‖〈 X,x 〉‖ L 2 .

Minimax lower-bound. We now provide a minimax lower bound for the covariance estimation with missing values problem. Let S p the set of p × p symmetric semi-positive matrices. Then, define C r = { S ∈ S p : r ( S ) ≤ r } the set of matrices of S p with effective rank at most r .

Theorem 1. Let p, n, r be strictly positive integers such that p ≥ max { n, 2 r } . Let X 1 , . . . , X n be i.i.d. random vectors in R p with covariance matrix Σ ∈ C r . Let ( d i,j ) 1 ≤ i ≤ n, 1 ≤ j ≤ p be an i.i.d. sequence of Bernoulli random variables with probability of success δ ∈ (0 , 1] , independent from the X 1 , . . . , X n . We observe n i.i.d. vectors Y 1 , . . . , Y n ∈ R p such that Y ( j ) i = d i,j X ( j ) i , i ∈ [ n ] , j ∈ [ p ] . Then there exists two absolute constants C > 0 and β ∈ (0 , 1) such that:

where inf ̂ Σ represents the infimum over all estimators ̂ Σ of matrix Σ based on Y 1 , . . . , Y n .

Sketch of proof. We first build a sufficiently large test set of hard-to-learn covariance operators exploiting entropy properties of the Grassmann manifold such that the distance between any two distinct covariance operator is at least of the order ‖ Σ ‖ δ √ r ( Σ ) n . Next, in order to control the KullbackLeibler divergence of the observations with missing values, we exploit in particular interlacing properties of the eigenvalues of the perturbed covariance operators [37].

This lower bound result improves upon [26, Theorem 2] as it relaxes the hypotheses on n and r . More specifically, the lower bound in [26] requires n ≥ 2 r 2 / δ 2 while we only need the mild assumption p ≥ max { n, 2 r } . Our proof leverages the properties of the Grassmann manifold, which has been previously utilized in different settings such as sparse PCA without missing values or contamination [45] and low-rank covariance estimation without missing values or contamination [23]. However, tackling missing values in the Grassmann approach adds a technical challenge to these proofs as they modify the distribution of observations. Our proof requires several additional nontrivial arguments to control the distribution divergences, which is a crucial step in deriving the minimax lower bound.

Non-asymptotic upper-bound in the operator norm. We provide an upper bound of the estimation error in operator norm. We write Y i = d i /circledot X i . Let ̂ Σ Y = n -1 ∑ n i =1 Y i ⊗ Y i be the classical covariance estimator of the covariance of Y . When the dataset contains missing values and corruptions, ̂ Σ Y is a biased estimator of Σ . Exploiting Equation (3), [26] proposed the following unbiased estimator of the covariance matrix Σ :

The following result is from [18, Theorem 4.2].

Lemma 1. Let X 1 , . . . , X n be i.i.d. sub-Gaussian random variables in R p , with covariance matrix Σ , and let d ij , i ∈ [1 , n ] , j ∈ [1 , p ] be i.i.d bernoulli random variables with probability of success δ > 0 . Then there exists an absolute constant C such that, for t > 0 , with probability at least 1 -e -t :

This result uses a recent unbounded version of the non-commutative Bernstein inequality, thus yielding some improvement upon the previous best known bound of [26]. Theorem 1 and Lemma 1 provide some important insights on the minimax rate of estimation in the missing values setting. In the high-dimensional regime p ≥ max { n, 2 r } and n ≥ δ -2 r ( Σ )(log r ( Σ )) log 2 n , we observe that the two bounds coincide up to a logarithmic factor in r ( Σ ) , hence clarifying the impact of missing data on the estimation rate via the parameter δ .

Heterogeneous missingness. We can extend the correction to the more general case where each feature has a different missing value rate known as the Missing at Random (MAR) setting in [35]. We denote by δ j ∈ (0 , 1] the probability to observe feature X ( j ) , 1 ≤ j ≤ p and we set δ := ( δ j ) j ∈ [ p ] . As in the MCAR setting, the probabilities ( δ j ) j ∈ [ p ] can be readily estimated by tallying the number of missing entries for each feature. Hence they will be assumed to be known for the sake of brevity. Let δ inv = ( δ -1 j ) j ∈ [ p ] be the vector containing the inverse of the observing probabilities and ∆ inv = δ inv ⊗ δ inv. In this case, the corrected estimator becomes :

¯

Let δ = max j { δ j } and δ ¯ = min j { δ j } be the largest and smallest probabilities to observe a feature. Theorem 2. (i) Let X 1 , . . . , X n be i.i.d. sub-Gaussian random variables in R p , with covariance matrix Σ . We consider the MAR setting described above. Then the estimator (8) satisfies, for any t > 0 , with probability at least 1 -e -t

(ii) Let p, n, r be strictly positive integers such that p ≥ max { n, 2 r } . Let X 1 , . . . , X n be i.i.d. random vectors in R p with covariance matrix Σ ∈ C r . Then,

If ¯ δ /equivasymptotic δ ¯ then the rates for the MCAR and MAR settings match. The proof is a straightforward adaptation of the proof in the MCAR setting.

4 Optimal estimation of covariance matrices with cell-wise contamination

In this section, we consider the cell-wise contamination setting (2).We derive both an upper bound on the operator norm error of the estimator (6) and a minimax lower bound for this specific setting. Let us assume that the ξ 1 , . . . ξ n are sub-Gaussian r.v. Note also that Λ := E [ ξ 1 ⊗ ξ 1 ] is diagonal in the cell-wise contamination setting (2).

Minimax lower-bound. The lower bound for missing values still applies to the contaminated case as missing values are a particular case of cell-wise contamination. But we want a more general lower bound that also covers the case of adversarial contaminations.

Theorem 3. Let p, n, r be strictly positive integers such that p ≥ max { n, 2 r } . Let X 1 , . . . , X n be i.i.d. random vectors in R p with covariance matrix Σ ∈ C r . Let ( d i,j ) 1 ≤ i ≤ n, 1 ≤ j ≤ p be i.i.d. sequence of bernoulli random variables of probability of success δ ∈ (0 , 1] , independent to the X 1 , . . . , X n . We observe n i.i.d. vectors Y 1 , . . . , Y n ∈ R p satisfying (2) where ξ i are i.i.d. of arbitrary distribution Q . Then there exists two absolute constants C > 0 and β ∈ (0 , 1) such that:

where inf ̂ Σ represents the infimum over all estimators of matrix Σ and max Q is the maximum over all contamination Q .

The proof of this theorem adapts an argument developed to derive minimax lower bounds in the Huber contamination setting. See App. G.3 for the full proof.

Non-asymptotic upper-bound in the operator norm. Note that the term ε (1 -δ ) Λ in the cell-wise contamination setting is negligible when δ ≈ 1 or ε ≈ 0 . Using the DDC detection procedure of [32], we can detect the contaminations and make ε smaller without decreasing δ too much. For simplicity, we assume from now on that the ξ ( j ) i are i.i.d. with common variance σ 2 ξ . Hence Λ = σ 2 ξ I p . We further assume that the ξ ( j ) i are sub-Gaussian since we observed in our experiments that filtering removed all the large-valued contaminated cells and only a few inconspicuous contaminated cells remained. Our procedure (6) satisfies the following result.

Theorem 4. Let the assumptions of Theorem 1 be satisfied. We assume in addition that the observations Y 1 , . . . , Y n satisfy (2) with ε ∈ [0 , 1) and δ ∈ (0 , 1] and i.i.d. sub-Gaussian ξ ( j ) i 's. Then, for any t > 0 , with probability at least 1 -e -t :

See App F.3 for the proof. As emphasized in [20], the effective rank r ( Σ ) provides a measure of the statistical complexity of the covariance learning problem in the absence of any contamination. However, when cell-wise contamination is present, the statistical complexity of the problem may increase from r ( Σ ) to r ( Λ ) = p . Fortunately, if the filtering process reduces the proportion of cell-wise contamination ε such that (1 -δ ) ε tr( Λ ) ≤ δ tr( Σ ) and ε ‖ Λ ‖ ≤ δ ‖ Σ ‖ . Then we can effectively mitigate the impact of cell-wise contamination. Indeed, we deduce from Theorem 4 that

where we considered for convenience the reasonable scenario where δ ( p -2) ≥ tr( Σ ) and σ 2 ξ ≥ ‖ Σ ‖ . The combination of the upper bound (12) with the lower bound in Theorem (3) provides the first insights into the impact of cell-wise contamination on covariance estimation.

5 Experiments

In our experiments, MV refers either to the debiased MCAR covariance estimator (6) or to its MAR extension (8). The synthetic data generation is described in App. A. We also performed experiments on real life datasets described in App. B. All experiments were conducted on a 2020 MacBook Air with a M1 processor (8 cores, 3.4 GHz). 1

0

.

0

0

.

1

0

.

2

0

.

3

0

.

4

0

.

5

0

.

6

0

.

7

0

.

8

Missing values rate

0

.

2

0

.

3

0

.

4

0

.

5

0

.

6

0

.

7

0

.

8

0

.

9

Spectral error

Classical

Mean Imputation

Median Imputation

II

KNN

MIWAE

GAIN

MV (ours)

5.1 Missing Values

We compared our method to popular imputations methods: KNNImputer ( KNNI ), which imputes the missing values based on the k-nearest neighbours [39], and IterativeImputer ( II ), which is inspired by the R package MICE [42], as coded in sklearn [30]; and two recent GANs-based imputation methods MIWAE [28] and GAIN [47] as found in the package hyperimpute [15]. The deep methods were tested using the same architectures, hyperparameters and early stopping rules as their respective papers.

In Figures 2, 3 and Table 1, we compare our estimator MV defined in (6) to these imputation methods combined with the usual covariance estimator on synthetic data (see App. A for details of data generation) in terms of statistical accuracy and execution time. First, MV beats all other methods in low-dimensional scenarios and maintains a competitive edge with II in high-dimensional situations when the missing data rate remains below 30% . Furthermore, it stands as the second-best choice when dealing with missing data rates exceeding 35% . Next, MV has by far the smallest execution time down several orders of magnitude while the execution time of II increases very quickly with the dimension and can become impractical (see Figure 9 for a dataset too large for II ). Overall, the procedures MV and II perform better than MIWAE and GAIN in this experiment. Our understanding is that MIWAE and GAIN use training metrics designed to minimize the entrywise error of imputation. We suspect this may be why their performances for the estimation of covariance with operator norm are not on par with other minimax methods. An interesting direction would be to investigate whether training MIWAE and GAIN with different metrics may improve the operator norm performance.

We refer to App. E for more experiments in the MAR setting of [28, Annex 3] which led to similar conclusions. These results confirm that imputation of missing values is not mandatory for accurate estimation of the covariance operator. Another viable option is to apply a debiasing correction to the empirical covariance computed on the original data containing missing values. The advantage of this approach is its low computational cost even in high-dimension.

0

.

0

0

.

1

0

.

2

0

.

3

0

.

4

0

.

5

0

.

6

0

.

7

0

.

8

Missing values rate

0

.

2

0

.

3

0

.

4

0

.

5

0

.

6

0

.

7

0

.

8

0

.

9

1

.

0

Spectral error

Classical

Mean Imputation

Median Imputation

II

KNN

MIWAE

GAIN

MV (ours)

5.2 Cell-wise contamination

Methods tested. Our baselines are the empirical covariance estimator applied without care for contamination and an oracle which knows the position of every outlier, deletes them and then computes the MV bias correction procedure (6). In view of Theorems 1 and 1, this oracle procedure is the best possible in the setting of cell-wise contamination. Hence, we have a practical framework to assess the performance of any procedure designed to handle cell-wise contamination.

The SOTA methods in the cell-wise contamination setting are the DI (Detection-Inputation) method [32] and the TSGS method (Two Step Generalised S-estimator) [2]. Both these methods were designed to work in the standard setting n > p but cannot handle the high-dimensional setting as we already mentioned. Nevertheless, we included comparisons of our methods to them in the standard setting n > p . The code for DI and TSGS are from the R packages cellwise and GSE respectively.

We combine the DDC detection procedure [34] to first detect and remove outliers with several estimators developed to handle missing values. Our main estimators are DDCMV (short for Detecting Deviating Cells Missing Values), which uses first DDC and then computes the debiaised covariance estimator (6) on the filtered data, and tailMV , which detects outliers through thresholding and then uses again (6). But we also proposed to combine the DDC procedure with imputation methods KNNI , II , GAIN and MIWAE and finally compute the standard covariance estimator on the completed data. Hence we define four additional novel robust procedures which we call DDCKNN , DDCII , DDCGAIN and DDCMIWAE . To the best of our knowledge, neither the first approach combining filtering with debiasing nor the second alternative approach combining filtering with missing values imputation have never been tested to deal with cell-wise contamination. A detailed description of each method is provided in App. C.

Outlier detection and estimation error under cell-wise contamination on synthetic data. We showed that the error of a covariance estimator under cell-wise contamination depends on the proportion of remaining outliers after a filtration. In Table 2 we investigate the filtering power of the Tail Cut and DDC methods in presence of Dirac contamination. We consider the cell-wise contamination setting (2) in the most difficult case ε = 1 which means that an entry is either correctly observed or replaced by an outlier (in other words, the dataset does not contain any missing value). For each values of δ in a grid, the quantities ˆ δ and ˆ ε are the proportions of true entries and remaining contaminations after filtering averaged over 20 repetitions. The DDC based methods are particularly efficient since the proportion of Dirac contamination drops from 1 -δ to virtually 0 for any δ ≥ 0 . 74 . In Figures 1 and 4, we see that the performance of our method is virtually the same as the oracle OracleMV as long as the filtering procedure correctly eliminates the Dirac contaminations. As soon as the filtering procedure fails, the statistical accuracy brutally collapses and our DDC based estimators no longer do better than the usual empirical covariance. In Table 8 and Figure 5, we repeated the same experiment but with a centered Gaussian contamination. Contrarily to the Dirac contamination scenario, we see in Figure 5 that the statistical accuracy of our DDC based methods slowly degrades as the contamination rate increases but their performance remains significantly better than that of the usual empirical covariance.

0

.

00

0

.

05

0

.

10

0

.

15

0

.

20

0

.

25

0

.

30

0

.

35

0

.

40

Dataset contamination rate (1

δ

)

4

6

8

10

12

14

Operator error of covariance estimator

classical

DDCMV95

tailMV

oracleMV

-

DDC MV 99

DDC MV 95

DDC KNN 99

DDC II 99

DDC GAIN

TSGS

DI

classical

DDC MV 99

DDC MV 95

DDC KNN 99

DDC II 99

DDC GAIN

TSGS

12

13

4.5

5.3

14

4.3

53

3.3

9.8

17

2.8

9.4

49

10

18

4.3

9.8

50

7.8

11

0.71

51

18

8.1

54

11

47

51

10

20

30

40

50

0

.

00

0

.

05

0

.

10

0

.

15

0

.

20

0

.

25

0

.

30

0

.

35

0

.

40

Dataset contamination rate (1

δ

)

2

.

5

5

.

0

7

.

5

10

.

0

12

.

5

15

.

0

17

.

5

20

.

0

22

.

5

Operator error of covariance estimator

classical

DDCMV95

tailMV

oracleMV

-

DDC MV 99

DDC MV 95

DDC KNN 99

DDC II 99

DDC GAIN

TSGS

DI

classical

DDC MV 99

DDC MV 95

DDC KNN 99

DDC II 99

DDC GAIN

TSGS

13

15

12

67

14

8

4.1

5.1

67

4.5

6.3

7.6

66

5.6

7.9

65

8.5

4.7

67

65

9.3

10

20

30

40

50

60

5.3 The effect of cell-wise contamination on real-life datasets

We tested the methods on 8 datasets from sklearn and Woolridge's book on econometrics [46]. These are low dimensional datasets (less than 20 features) representing various medical, social and economic phenomena. We also included 2 high-dimensional datasets. See App. B for the list of the datasets.

One interesting observation is that the instability of Mahalanobis distance-based algorithms is not limited to high-dimensional datasets. Even datasets with a relatively small number of features can exhibit instability. This can be seen in the performance of DI on the Attend dataset, as depicted in Figure 6, where it fails to provide accurate results. Similarly, both TSGS and DI fail to perform well on the CEOSAL2 dataset, as shown in Figure 7, despite both datasets having fewer than 15 features.

On the Abalone dataset, once we have removed 4 obvious outliers (which are detected by both DDC and the tail procedure), all estimators reached a consensus with the non-robust classical estimator, meaning that this dataset provides a ground truth against which we can evaluate and compare the performance of robust procedures in our study. To this end, we artificially contaminate 5% of the cells at random in the dataset with a Dirac contamination and compare the spectral error of the different robust estimators. As expected, TSGS and all our new procedures succeed at correcting the error, however DI becomes unstable (see Table 3). DDC MIWAE is close to SOTA TSGS for cellwise contamination and DDC II performs better. We also performed experiments on two highdimensional datasets, where our methods return stable estimates of the covariance ( DDCMV99 and DDCMV95 are within ≈ 3% of each other) and farther away from the classical estimator (See Figures 8 and 9 ). Note also that DDCII 's computation time explodes and even returns out-of-memory errors due to the high computation cost of II that we already highlighted in Table 1.

DDC MV 99

DDC MV 95

DDC KNN 99

DDC II 99

DDC GAIN

TSGS

DI

classical

DDC MV 99

DDC MV 95

DDC KNN 99

DDC II 99

DDC GAIN

TSGS

5.9

6

4.8

3.4

6.7

6

2.2

2.3

3.6

2.1

6.1

3

4

3.1

6.3

1.9

3

6.2

4.3

6.2

6.2

2

3

4

5

6

6 Conclusion and future work

In this paper, we have extended theoretical guarantees on the spectral error of our covariance estimators robust to missing data to the missing at random setting. We have also derived the first theoretical guarantees in the cell-wise contamination setting. We highlighted in our numerical experimental study that in the missing value setting, our debiased estimator designed to tackle missing values without imputation offers statistical accuracy similar to the SOTA IterativeImputer for a dramatic computational gain. We also found that SOTA algorithms in the cell-wise contamination setting often fail in the standard setting p < n for dataset with fast decreasing eigenvalues (resulting in approximately low rank covariance), a setting which is commonly encountered in many real life applications. This is due to the fact that these methods use matrix inversion which is unstable to small eigenvalues in the covariance structure and can even fail to return any estimate. In contrast, we showed that our strategy combining filtering with estimation procedures designed to tackle missing values produce far more stable and reliable results. In future work, we plan to improve our theoretical upper and lower bounds in the cell-wise contamination setting to fully clarify the impact of this type of contamination in covariance estimation.

Acknowledgements. This paper is based upon work partially supported by the Chaire Business Analytic for Future Banking and EU Project ELIAS under grant agreement No. 101120237.

References

Pedro Abdalla and Nikita Zhivotovskiy. Covariance Estimation: Optimal Dimension-free Guarantees for Adversarial Corruption and Heavy Tails, July 2023.

Claudio Agostinelli, Andy Leung, Victor J. Yohai, and Ruben H. Zamar. Robust estimation of multivariate location and scatter in the presence of cellwise and casewise contamination, June 2014.

Fatemah Alqallaf, Stefan Van Aelst, Victor J. Yohai, and Ruben H. Zamar. Propagation of outliers in multivariate data. The Annals of Statistics , 37(1):311-331, February 2009.

Florentina Bunea and Luo Xiao. On the sample covariance matrix estimator of reduced effective rank population matrices, with applications to fPCA. Bernoulli , 21(2), May 2015.

Mengjie Chen, Chao Gao, and Zhao Ren. Robust Covariance and Scatter Matrix Estimation under Huber's Contamination Model. arXiv:1506.00691 [math, stat] , June 2017.

Mike Danilov, Victor Yohai, and Ruben Zamar. Robust Estimation of Multivariate Location and Scatter in the Presence of Missing Data. JASA. Journal of the American Statistical Association , 107, September 2012.

A. P. Dempster, N. M. Laird, and D. B. Rubin. Maximum Likelihood from Incomplete Data via the EM Algorithm. Journal of the Royal Statistical Society. Series B (Methodological) , 39(1):1-38, 1977.

Sheela Devadas, Peter J Haine, and Keaton Stubis. The Schur-Horn Theorem. 2015.

Weinan Dong, Daniel Yee Tak Fong, Jin-sun Yoon, Eric Yuk Fai Wan, Laura Elizabeth Bedford, Eric Ho Man Tang, and Cindy Lo Kuen Lam. Generative adversarial networks for imputing missing data for big data clinical research. BMC Medical Research Methodology , 21(1):78, April 2021.

Alessio Farcomeni. Robust Constrained Clustering in Presence of Entry-Wise Outliers. Technometrics , 56, February 2014.

Peter J. Huber. Robust Estimation of a Location Parameter. The Annals of Mathematical Statistics , 35(1):73-101, 1964.

Peter J. Huber and Elvezio M. Ronchetti. Robust Statistics, 2nd Edition | Wiley . Wiley Series in Probability and Statistics. John Wiley and Sons, Inc., 2009.

Mia Hubert, Michiel Debruyne, and Peter J. Rousseeuw. Minimum Covariance Determinant and Extensions. WIREs Computational Statistics , 10(3), May 2018.

Mortaza Jamshidian and Peter M. Bentler. ML Estimation of Mean and Covariance Structures with Missing Data Using Complete Data Routines. Journal of Educational and Behavioral Statistics , 24(1):21-41, 1999.

Daniel Jarrett, Bogdan Cebere, Tennison Liu, Alicia Curth, and Mihaela van der Schaar. Hyperimpute: Generalized iterative imputation with automatic model selection. 2022.

Charles R. Johnson. Matrix theory and applications (charles r. johnson, ed.). Proceedings of symposia in applied mathematics , 40, 1989.

Julie Josse, Nicolas Prost, Erwan Scornet, and Gaël Varoquaux. On the consistency of supervised learning with missing values. arXiv:1902.06931 [cs, math, stat] , July 2020.

Yegor Klochkov and Nikita Zhivotovskiy. Uniform Hanson-Wright type concentration inequalities for unbounded entries via the entropy method, August 2019.

Vladimir Koltchinskii and Karim Lounici. Asymptotics and concentration bounds for bilinear forms of spectral projectors of sample covariance. Annales de l'Institut Henri Poincaré, Probabilités et Statistiques , 52(4):1976 - 2013, 2016.

Vladimir Koltchinskii and Karim Lounici. Concentration inequalities and moment bounds for sample covariance operators. Bernoulli , 23(1):110-133, February 2017.

Vladimir Koltchinskii and Karim Lounici. New asymptotic results in principal component analysis. Sankhya A , 79(2):254-297, Aug 2017.

Vladimir Koltchinskii and Karim Lounici. Normal approximation and concentration of spectral projectors of sample covariance. The Annals of Statistics , 45(1):121 - 157, 2017.

Vladimir Koltchinskii, Karim Lounici, and Alexander B. Tsybakov. Estimation of Low-Rank Covariance Function, April 2015.

Andy Leung, Hongyang Zhang, and Ruben H. Zamar. Robust regression estimation and inference in the presence of cellwise and casewise contamination. Computational Statistics & Data Analysis , 99:1-11, July 2016.

Roderick Little and Donald Rubin. Statistical Analysis with Missing Data, Second Edition. Wiley Series in Probability and Mathematical Statistics. Probability and Mathematical Statistics. Wiley edition, 2002.

Karim Lounici. High-dimensional covariance matrix estimation with missing observations. Bernoulli , 20(3):1029-1058, August 2014.

Chao Ma, Sebastian Tschiatschek, José Miguel Hernández-Lobato, Richard Turner, and Cheng Zhang. VAEM: A Deep Generative Model for Heterogeneous Mixed Type Data, June 2020.

Pierre-Alexandre Mattei and Jes Frellsen. MIWAE: Deep Generative Modelling and Imputation of Incomplete Data Sets. In Kamalika Chaudhuri and Ruslan Salakhutdinov, editors, Proceedings of the 36th International Conference on Machine Learning, ICML 2019, 9-15 June 2019, Long Beach, California, USA , volume 97 of Proceedings of Machine Learning Research , pages 4413-4423. PMLR, 2019.

Alain Pajor. Metric Entropy of the Grassmann Manifold. Complex Geometry Analysis , 34:181188, 1998.

F. Pedregosa, G. Varoquaux, A. Gramfort, V. Michel, B. Thirion, O. Grisel, M. Blondel, P. Prettenhofer, R. Weiss, V. Dubourg, J. Vanderplas, A. Passos, D. Cournapeau, M. Brucher, M. Perrot, and E. Duchesnay. Scikit-learn: Machine learning in Python. Journal of Machine Learning Research , 12:2825-2830, 2011.

Nikita Puchkin, Fedor Noskov, and Vladimir Spokoiny. Sharper dimension-free bounds on the Frobenius distance between sample covariance and its expectation, August 2023.

Jakob Raymaekers and Peter J. Rousseeuw. Handling cellwise outliers by sparse regression and robust covariance. arXiv:1912.12446 [stat] , December 2020.

Jakob Raymaekers and Peter J. Rousseeuw. Fast robust correlation for high-dimensional data. Technometrics , 63(2):184-198, April 2021.

Peter J. Rousseeuw and Wannes Van den Bossche. Detecting deviating data cells. Technometrics , 60(2):135-145, April 2018.

Donald B. Rubin. Inference and Missing Data. Biometrika , 63(3):581-592, 1976.

Eckhard Schlemm. The kearns-saul inequality for bernoulli and poisson-binomial distributions. Journal of theoretical probability , 29:48-62, 2016.

R. C. Thompson. Principal submatrices of normal and Hermitian matrices. Illinois Journal of Mathematics , 10(2):296-308, June 1966.

Joel A. Tropp. An introduction to matrix concentration inequalities. Foundations and Trends® in Machine Learning , 8(1-2):1-230, 2015.

Olga Troyanskaya, Michael Cantor, Gavin Sherlock, Pat Brown, Trevor Hastie, Robert Tibshirani, David Botstein, and Russ B. Altman. Missing value estimation methods for DNA microarrays. Bioinformatics , 17(6):520-525, June 2001.

Alexandre B. Tsybakov. Nonparametric estimators. In Alexandre B. Tsybakov, editor, Introduction to Nonparametric Estimation , Springer Series in Statistics, pages 1-76. Springer, New York, NY, 2009.

John W. Tukey. The Ninther, a Technique for Low-Effort Robust (Resistant) Location in Large Samples. In H. A. David, editor, Contributions to Survey Sampling and Applied Statistics , pages 251-257. Academic Press, January 1978.

Stef van Buuren and Karin Groothuis-Oudshoorn. Mice: Multivariate Imputation by Chained Equations in R. Journal of Statistical Software , 45:1-67, December 2011.

Roman Vershynin. Introduction to the non-asymptotic analysis of random matrices, November 2011.

Roman Vershynin. High-Dimensional Probability: An Introduction with Applications in Data Science . Cambridge Series in Statistical and Probabilistic Mathematics. Cambridge University Press, Cambridge, 2018.

Vincent Q. Vu and Jing Lei. Minimax sparse principal subspace estimation in high dimensions. The Annals of Statistics , 41(6):2905-2947, December 2013.

Jeffrey M. Wooldridge. Introductory econometrics : a modern approach / Jeffrey M. Wooldridge,... Cengage learning, 2016.

Jinsung Yoon, James Jordon, and Mihaela van der Schaar. GAIN: Missing Data Imputation using Generative Adversarial Nets, June 2018.

Shuhan Zheng and Nontawat Charoenphakdee. Diffusion models for missing value imputation in tabular data, March 2023.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper “Robust covariance estimation with missing values and cell-wise contamination”  presents an unbiased estimator for covariance in the presence of missing values that do not require any imputation step, and the authors claim they achieve competitive results compared to other state-of-the-art methods. The authors check their findings in synthetical and real-world datasets and compare them to other methods such as imputation using KNN or the MICE algorithm\n\nSTRENGTHS:\nThe notion of rigorously establishing and deriving bounds for covariance matrix estimation in the presence of missing data is intriguing, as it suggests that the bounds established for covariance estimation in complete data scenarios should extend to account for missing data. Hence, this contribution from the paper opens up new avenues of thought for researchers in the field of \"missing data.”\n\nWEAKNESSES:\nHowever, there are some substantial flaws in the paper that I would like to point out. First and probably most important, I consider the paper is not well structured and it is challenging to follow, even for experienced readers in the field of missing data handling. There is no evident thread from which the reader can follow the ideas presented throughout the paper. The connection between sections, paragraphs and ideas is often misleading: the reader might be introduced to new notations, and new concepts, or expected to remember previous concepts that were not actually introduced (Line 108 you comment $\\Sigma$ is low rank with no extra justification). Specially, after reading the paper the ultimate goal of the paper still remains unclear: is it providing a novel method for covariance estimation under missing data, imputing missing data directly? \n\nThe paper also lacks a clear section presenting the related work. These sections are often included in papers because: 1) they permit newcomers to the field to get some background, 2) they set the presented paper in the context of other historical or concurrent works, and outline the clear differences w.r.t others and 3) they allow experienced readers to corroborate if the proposed paper has properly undergone a prior deep literature review. I feel this paper lacks this important aspect: authors claim they compare to SOTA methods, but the most recent paper dates from 2021, then 2020, and after that 2018. Although MICE and KNN imputation are widely used in the missing data literature, the missing data is a rather hot topic in the machine learning field, and many works have proposed novel methods in the context of VAEs([1], [2], [3], [4]), GANs([5]) or recently diffusion models ([6]). I consider claiming this paper can outperform SOTA methods might be a strong assertion.\n\nThis, and the lack of comparison with standard datasets used in the missing data literature (tabular datasets from UCI or high-dimensional datasets such as MNIST, and CelebA, for example) makes the experimental section difficult to follow. The synthetic and real-world datasets are not described at all in the main manuscript and are referred to the Appendix, but the authors make comments about these datasets in the main text. From Figures 4 and 5 it is still not evident to me which method they propose, because it seems like they mix already proposed methods with their new covariance estimation. What’s more, the authors refer to the Figures and Tables in the Appendix and make comments about the results in the main text. I believe if comments about Figures are to be made, then these figures should be added in the main manuscript; or conversely, comments should be made in the Appendix. Something in between might make the reading a bit cumbersome. \n\n[1] Nazabal, A., Olmos, P. M., Ghahramani, Z., & Valera, I. (2020). Handling incomplete heterogeneous data using vaes. *Pattern Recognition*, *107*, 107501.\n\n[2] Mattei, P. & Frellsen, J.. (2019). MIWAE: Deep Generative Modelling and Imputation of Incomplete Data Sets. Proceedings of the 36th International Conference on Machine Learning, in Proceedings of Machine Learning Research 97:4413-4423 Available from https://proceedings.mlr.press/v97/mattei19a.html.\n\n[3] Ma, C., Tschiatschek, S., Palla, K., Hernández-Lobato, J. M., Nowozin, S., & Zhang, C. (2018). Eddi: Efficient dynamic discovery of high-value information with partial vae. *arXiv preprint arXiv:1809.11142*.Chicago\n\n[4] Ma, C., Tschiatschek, S., Turner, R., Hernández-Lobato, J. M., & Zhang, C. (2020). VAEM: a deep generative model for heterogeneous mixed type data. *Advances in Neural Information Processing Systems*, *33*, 11237-11247.\n\n[5] Yoon, J., Jordon, J., & Schaar, M. (2018, July). Gain: Missing data imputation using generative adversarial nets. In *International conference on machine learning* (pp. 5689-5698). PMLR.\n\n[6] Zheng, S., & Charoenphakdee, N. (2022). Diffusion models for missing value imputation in tabular data. Representation Learning Workshop at NeurIPS.\n\nQUESTIONS:\nI still have some questions that could maybe help to make the paper more easy to understand:\n\n1. In Equation 2, why $X_i, \\psi_i, d_{i,j}, e_{i,j}$ are mutually independent? Isn’t this a rather strong assumption? \n2. The oracle using the MV method seems to always perform perfectly. More comments on this would be helpful to understand why. My first believe is that there might not be a fair comparison w.r.t the other methods. \n3. In Table 2: What are DDCMV90, DDCM95? I cannot find where these are defined. \n4. What is the so called “classical” covariance estimator? There is no definition of such method.\n\nLIMITATIONS:\nThe authors don’t make any comment about the limitations of their method. However, I found a rather strong limitation in Appendix B. The method is tested on real-world datasets from sklearn, but the authors remove the categorical variables and variables that might be multimodal. This is a strong limitation since the method is tested on very smooth data distributions. I believe the authors should perform experiments in these datasets, assuming the natural artefacts that might appear.\n\nThe lack of a clear thread that makes the narrative of the paper easy to follow, the difficulty of understanding the final results from the experimental section because some figures do not appear and the need for further comparison of the proposed method with more SOTA models and other datasets drive me to believe that this paper lies below the acceptance threshold for NeurIPS conference at this point. I also hope some of my concerns can be addressed during the rebuttal period.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-06-24"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe objective of this paper is to mitigate the influence of outliers and missing values in covariance estimation without resorting to imputations.  A particular type of missing data that satisfies assumption (1) is considered and an unbiased estimator of the true covariance  that achieves the minimax bound with the operator norm is proposed.  For the outliers, they are further assumed to be cell-wise contaminated satisfying assumption (2) and two further assumptions (see lines 169 and  173).  Under these assumptions, an unbiased estimator of the true covariance is proposed, and its upper and lower bounds in the operator norm are established.  Numerical experiments demonstrate the advantages of the proposed approaches over existing approaches in a high-dimension and low rank setting.\n\nSTRENGTHS:\nThis is a paper with both new methodology and theory for an interesting problem. The primary contribution is its strong theoretical results, which significantly  improve over existing results.\n\nWEAKNESSES:\nThe assumptions on missing values seem strong.  Usually, measurements for a subject are collected at the same time.  So, forcing the missing mechanism to be independent across the components seems to limit its applications. Also, why would all the components have the same missing probability? Some components might be harder to measure, hence more likely to be missing.  This assumption is much stronger than MCAR. \n\nThe definition of cell-wise contamination in (2) is restrictive as an outlier for a vector data does not have to be an outlier in any of the component. Also, it is further assumed in line 169 that components of cell-wise contaminations are uncorrelated.\n\nQUESTIONS:\nDoes equation (3) only involve missing values with no cell-wise contaminations as it does not involve any of the parameters in (2)?   \n\nThe \\xi_k in line 86 are random vectors with diagonal covariance matrix (see line 169).    Is this diagonal assumption a key to the results in Section 4? What would happen if they were not diagonal?\n\nLIMITATIONS:\nThe proposed approach involves several assumptions that will limit its applicability in practice.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-05"}

comments: SUMMARY:
This work proposes an unbiased estimator for the covariance matrix under missing entries. Unlike most existing approaches, the proposed estimator does not require an imputation step to discard missing values before computing the covariance matrix. Interestingly, the proposed method employs an approach that avoids matrix inversion in high dimensions, thereby overcoming large complexity issues. 
Moreover, along with relevant outlier detection methods, the proposed estimator can be used used under the presence of cell-wise outliers to tackle such outliers in a high dimensional setting with inherent low rank structure. The effectiveness of the proposed method is demonstrated via experimental studies.

STRENGTHS:
Derivation of non-asymptotic estimation bounds of the covariance matrix with the operator norm and matching minimax lower bounds, enabling the quantification of impact of rates of missing values rate outliers.

Traditional methods rely on matrix inversions resulting in failure in the high-dimensional setting, while the proposed avoids matrix multiplication and therefore, performs well. 

Experiments demonstrate that the proposed method is more robust to cell-wise contamination than comparing state-of-the-art methods, while producing reliable estimates of the covariance in low computation duration.

WEAKNESSES:
Missing definitions of metrics -- Please see below.

QUESTIONS:
What is STD in Table 2?

Please provide an expression for the relative spectral difference used in Table 3.

LIMITATIONS:
NA
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-06

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper studies the problem of estimating a covariance matrix in the presence of missing observations and/or outliers.\nThe authors first consider the missing values framework and theoretically analyse the unbiased covariance estimator introduced in [Karim Lounici. High-dimensional covariance matrix estimation with missing observations]. Then, in the framework (missing values + outliers), they also recommend its use in combination with cell-wise outlier detection methods such as DDC.\nFinally, to support their theoretical results, they carried out some experiments to demonstrate the advantage of their approach over existing methods.\n\nSTRENGTHS:\n1\\  It is a real pleasure to read this paper. Indeed, It is very well written, which makes it easy to understand\n\n2\\ The contributions are clear and well-supported. In addition, the theoretical results (if true - see next section) are theoretically sound because there are probability bounds on the difference between the estimated covariance and the true covariance, but more importantly there are minimax bounds.\n\n3\\ The experimental part is again very clear and seems to compare well to existing methods with the error bars and calculation time indicated.\n\nWEAKNESSES:\n1\\ The contamination is assumed to be sub-Gaussian. This contrasts with the recent robust literature, which attempts to take into account any type of contamination.\n\n2\\ The probability of missing values d and the outlier epsilon are assumed to be known in the theoretical results (although I understand that a full analysis of the estimator when d and epsilon are estimated is certainly difficult). \n\n3\\  3. There are possible problems in the theorems which should be clarified (see Questions).\n\n================\nAdditional remarks:\n\nFigure 1 is never cited and explained.\n\nI think the formal definition of the operator norm should be given in the notation part.\n\nline 101: \"The The\"\n\nline 171: \"procedure procedure\"\n\nQUESTIONS:\nMy first question is more related to Theorem 1. However, as the proofs of the other theorems use sometimes the same techniques, they are also related to them.\n\n1\\ In the proof of Theorem 1 (Appendix E.2.3), why there is no absolute value on (1/d - 1/d^2) when this scalar moves outside of the norm? This could change the proof because then |(1/d - 1/d^2)| is no longer always less than 1/d\n\n2\\ In the section \"Experiments\", it is stated that the best oracle is the one that knows \"the position of every outlier, deletes them and then computes the MV bias correction procedure\". Isn't an oracle that knows \"the position of every outlier\" but instead of deleting them, performs a good imputation and then uses a standard estimator better?\n\n3\\ It seems that only the estimator of equation (5) is used in the experiments. Why not also use the estimator from equation (10)?\n\n================\n\nOne final remark: My score reflects my concerns about the first question. Of course, I'm willing to raise the score if there's something I'm missing.\n\nLIMITATIONS:\n-", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-08"}