Title: Conformalized matrix completion
Venue: NeurIPS 2023
Accepted: yes
Keywords: matrix completion

conformal inference

uncertainty quantification
Abstract: Matrix completion aims to estimate missing entries in a data matrix, using the assumption of a low-complexity structure (e.g., low-rankness) so that imputation is possible. While many effective estimation algorithms exist in the literature, uncertainty quantification for this problem has proved to be challenging, and existing methods are extremely sensitive to model misspecification. In this work, we propose a distribution-free method for predictive inference in the matrix completion problem. Our method adapts the framework of conformal prediction, which provides prediction intervals with guaranteed distribution-free validity in the setting of regression, to the problem of matrix completion. Our resulting method, conformalized matrix completion (cmc), offers provable predictive coverage regardless of the accuracy of the low-rank model. Empirical results on simulated and real data demonstrate that cmc is robust to model misspecification while matching the performance of existing model-based methods when the model is correct.

Conformalized matrix completion

Abstract

Matrix completion aims to estimate missing entries in a data matrix, using the assumption of a low-complexity structure (e.g., low rank) so that imputation is possible. While many effective estimation algorithms exist in the literature, uncertainty quantification for this problem has proved to be challenging, and existing methods are extremely sensitive to model misspecification. In this work, we propose a distribution-free method for predictive inference in the matrix completion problem. Our method adapts the framework of conformal prediction, which provides confidence intervals with guaranteed distribution-free validity in the setting of regression, to the problem of matrix completion. Our resulting method, conformalized matrix completion ( cmc ), offers provable predictive coverage regardless of the accuracy of the low-rank model. Empirical results on simulated and real data demonstrate that cmc is robust to model misspecification while matching the performance of existing model-based methods when the model is correct.

1 Introduction

Matrix completion, the task of filling in missing entries in a large data matrix, has a wide range of applications such as gene-disease association analysis in bioinformatics (Natarajan and Dhillon, 2014), collaborative filtering in recommender systems (Rennie and Srebro, 2005), and panel data prediction and inference in econometrics (Amjad et al., 2018; Bai and Ng, 2021; Athey et al., 2021). If the underlying signal is assumed to be low-rank, a range of estimation algorithms have been proposed in the literature, including approaches based on convex relaxations of rank (Candes and Plan, 2010; Candès and Tao, 2010; Koltchinskii et al., 2011; Foygel and Srebro, 2011; Negahban and Wainwright, 2012; Yang and Ma, 2022) and nonconvex optimization over the space of low-rank matrices (Keshavan et al., 2010; Jain et al., 2013; Sun and Luo, 2016; Ma et al., 2020).

While the problem of estimation has been explored through many different approaches in the literature, the question of uncertainty quantity for matrix completion remains challenging and under-explored. Cai et al. (2016) and Carpentier et al. (2018) construct confidence intervals for the missing entries using order-wise concentration inequalities, which could be extremely loose. Chen et al. (2019) proposes a de-biased estimator for matrix completion and characterizes its asymptotic distribution, which then yields entrywise confidence intervals are then derived for the underlying incoherent matrix with i.i.d. Gaussian noise. See also Xia and Yuan (2021); Farias et al. (2022) for related approaches.

The effectiveness of the aforementioned uncertainty quantification methods rely heavily on the exact low-rank structure as well as assumptions on the tail of the noise. In practice, the low-rank assumption can only be met approximately, and heavy-tailed noise is quite common in areas such as macroeconomics and genomics (Nair et al., 2022). Without exact model specifications, the solution to the matrix completion algorithm is hard to interpret and the asymptotic distribution of the obtained estimator is no longer valid. The dependence of model-based approaches on exact model assumptions motivates us to answer the question: can we construct valid and short confidence intervals for the

missing entries that are free of both model assumptions and of constraints on which estimation algorithms we may use?

1.1 Conformal prediction for distribution-free uncertainty quantification

Our proposed method is built on the idea of conformal prediction. The conformal prediction framework, developed by Vovk et al. (2005) and Shafer and Vovk (2008), has drawn significant attention in recent years in that it enables the construction of distribution-free confidence intervals that are valid with exchangeable data from any underlying distribution and with any 'black-box' algorithm. To reduce computation in full conformal prediction, variants exist based on data splitting (Vovk et al., 2005; Lei et al., 2018), leave-one-out (Barber et al., 2021), cross-validation (V ovk, 2015; Barber et al., 2021), etc.

When distribution shift is present, with covariate shift as a special case, the exchangeability is violated, and Tibshirani et al. (2019) proposes a more general procedure called weighted conformal prediction that guarantees the validity under the weighted exchangeability of data. Barber et al. (2022) further relaxes the exchangeability assumption on the data and characterizes the robustness of a generalized weighted approach via an upper bound for the coverage gap. Other related works (Lei and Candès, 2020; Candès et al., 2021; Jin et al., 2023) adapt the conformal framework to applications in causal inference, survival analysis, etc., and study the robustness of weighted conformal prediction with estimated weights.

1.2 Main contributions

In this paper, we adapt the framework of conformal prediction for the problem of matrix completion, and make the following contributions:

We construct distribution-free confidence intervals for the missing entries in matrix completion via conformal prediction. The validity is free of any assumption on the underlying matrix and holds regardless of the choice of estimation algorithms practitioners use. To achieve this, we prove the (weighted) exchangeability of unobserved and observed units when they are sampled (possibly nonuniformly) without replacement from a finite population.

When the sampling mechanism is unknown, we develop a provable lower bound for the coverage rate which degrades gracefully as the estimation error of the sampling probability increases.

In addition, to improve computational efficiency when faced with a large number of missing entries, we propose a one-shot conformalized matrix completion approach that only computes the weighted quantile once for all missing entries.

2 Problem setup

In this section, we formulate the matrix completion problem and contrast our distribution-free setting with the model-based settings that are more common in the existing literature. For a partially-observed d 1 × d 2 matrix, the subset S ⊆ [ d 1 ] × [ d 2 ] denotes the set of indices where data is observed-that is, our observations consist of ( M ij ) ( i,j ) ∈S . In much of the matrix completion literature, it is common to assume a signal-plus-noise model for the observations,

where M ∗ = ( M ∗ ij ) ( i,j ) ∈ [ d 1 ] × [ d 2 ] is the 'true' underlying signal while ( E ij ) ( i,j ) ∈S denotes the (typically zero-mean) noise. Frequently, it is assumed that M ∗ is low-rank, or follows some other latent low-dimensional structure, so that recovering M ∗ from the observed data is an identifiable problem. In works of this type, the goal is to construct an estimator ̂ M that accurately recovers the underlying M ∗ . In contrast, in a more model-free setting, we may no longer wish to hypothesize a signal-plus-noise model, and can instead assume that we are observing a random subset of entries of a deterministic matrix M ; in this setting, estimating M itself becomes the goal.

For both frameworks, many existing results focus on the problem of estimation (either of M ∗ or of M ), with results establishing bounds on estimation errors under various assumptions. For instance, see Candes and Plan (2010); Candès and Tao (2010); Negahban and Wainwright (2012); Chen et al. (2020) for results on estimating M ∗ (under stronger conditions, e.g., a low-rank and incoherent signal,

and zero-mean sub-Gaussian noise), or Srebro and Shraibman (2005); Foygel and Srebro (2011) for results on estimating M (under milder conditions); note that in the latter case, it is common to instead refer to predicting the entries of M , e.g., temperature or stock market returns, since we often think of these as noisy observations of some underlying signal. On the other hand, relatively little is known about the problem of uncertainty quantification for these estimates: given some estimator ̂ M , can we produce a confidence interval around each ̂ M ij that is (asymptotically) guaranteed to contain the target with some minimum probability?

The results of Chen et al. (2019) address this question under strong assumptions, namely, a low-rank and incoherent signal M ∗ , plus i.i.d. Gaussian noise ( E ij ) ( i,j ) ∈ [ d 1 ] × [ d 2 ] . Zhao and Udell (2020) proposes another inferential procedure under the Gaussian copula assumption for the data. In this work, we provide a complementary answer that instead addresses the model-free setting: without relying on assumptions, we aim to produce a provably valid confidence interval for the entries M ij of M (not of M ∗ , because without assuming a low-rank model, there is no meaningful notion of an 'underlying signal').

To do so, we will treat the matrix M as deterministic, while the randomness arises purely from the random subset of entries that are observed. More specifically, we assume

Z ij = ✶ { ( i, j ) is observed } ∼ Bern( p ij ) , independently for all ( i, j ) ∈ [ d 1 ] × [ d 2 ] , (1) and let S = { ( i, j ) ∈ [ d 1 ] × [ d 2 ] : Z ij = 1 } be the subset of observed locations. We consider the setting where the sample probabilities p ij 's are nonzero. After observing M S = ( M ij ) ( i,j ) ∈S , our goal is to provide confidence intervals { ̂ C ( i, j ) } ( i,j ) ∈S c for all unobserved entries, with 1 -α coverage over the unobserved portion of the matrix-that is, we would like our confidence intervals to satisfy

where

measures the 'average coverage rate' of the confidence intervals. 1 The goal of this work is to provide an algorithm for constructing confidence intervals { ̂ C ( i, j ) } ( i,j ) ∈S c based on the observed entries M S , satisfying (2) with no assumptions on M . In particular, given the strong success of various matrix completion algorithms for the problem of estimation, we would like to be able to provide uncertainty quantification around any base estimator-i.e., given any algorithm for producing an estimate ̂ M , our method constructs confidence intervals { ̂ C ( i, j ) } ( i,j ) ∈S c around this initial estimate.

Notation. For a matrix A = ( A ij ) ∈ R m 1 × m 2 , we define ∥ A ∥ ∞ := max ( i,j ) ∈ [ m 1 ] × [ m 2 ] | A ij | , and ∥ A ∥ 1 := ∑ ( i,j ) ∈ [ m 1 ] × [ m 2 ] | A ij | . We use A i, · and A · ,j to refer to the i th row and j th column of A , respectively. For any t ∈ R , δ t denotes the distribution given by a point mass at t .

3 Conformalized matrix completion

We next present our method, conformalized matrix completion ( cmc ). Our procedure adapts the split conformal prediction method (Vovk et al., 2005) to the problem at hand. As is standard in the conformal prediction framework, the goal of cmc is to provide uncertainty quantification (i.e., confidence intervals) around the output of any existing estimation procedure that the analyst chooses to use-this is a core strength of the conformal methodology, as it allows the analyst to use state-ofthe-art estimation procedures without compromising on the validity of inference. At a high level, the cmc procedure will output a confidence interval of the form

for the matrix value M ij at each unobserved entry ( i, j ) . Here, after splitting the observed entries into a training set and a calibration set, the training set is used to produce ̂ M ij (a point estimate for the

target value M ij ) and ̂ s ij (a scaling parameter that estimates our uncertainty for this point estimate); then, the calibration set is used to tune the scalar ̂ q , to adjust the width of these confidence intervals and ensure coverage at the desired level 1 -α . 2

Algorithm 1 Conformalized matrix completion ( cmc )

1: Input : target coverage level 1 -α ; data splitting proportion q ∈ (0 , 1) ; observed entries M S .

2: Split the data: draw W ij i.i.d. ∼ Bern( q ) , and define training and calibration sets

3: Using the training data M S tr indexed by S tr ⊆ [ d 1 ] × [ d 2 ] , compute:

An initial estimate ̂ M using any matrix completion algorithm (with ̂ M ij estimating the target M ij );

Optionally, a local uncertainty estimate ̂ s (with ̂ s ij estimating our relative uncertainty in the estimate ̂ M ij ), or otherwise set ̂ s ij ≡ 1 ;

An estimate ̂ P of the observation probabilities (with ̂ p ij estimating p ij , the probability of entry ( i, j ) being observed).

4: Compute normalized residuals on the calibration set, R ij = | M ij -̂ M ij | ̂ s ij , ( i, j ) ∈ S cal .

5: Compute estimated odds ratios for the calibration set and test set, ̂ h ij = 1 -̂ p ij ̂ p ij , ( i, j ) ∈ S cal ∪S c , and then compute weights for the calibration set and test point,

6: Compute threshold ̂ q = Quantile 1 -α ( ∑ ( i,j ) ∈S cal ̂ w ij · δ R ij + ̂ w test · δ + ∞ ) .

7: Output : confidence intervals ̂ C ( i, j ) = ̂ M ij ± ̂ q · ̂ s ij for each unobserved entry ( i, j ) ∈ S c .

3.1 Exchangeability and weighted exchangeability

We split the set of observed indices S into a training and a calibration set S = S tr ∪ S cal as is shown in Algorithm 1. We should notice that the sampling without replacement may introduce implicit dependence as well as a distribution shift from the i.i.d. sampling. Thus the two sets are dependent, which is different from the split conformal methods in regression problems. Before we present the coverage guarantees for our method, we first examine the role of (weighted) exchangeability in this method, to build intuition for how the method is constructed.

3.1.1 Intuition: the uniform sampling case

First, for intuition, consider the simple case where the entries are sampled with constant probability, p ij ≡ p . If this is the case, then the set of calibration samples and the test point are exchangeablethat is, for ( i ∗ , j ∗ ) denoting the test point that is drawn uniformly from S c , if we are told that the combined set S cal ∪ { ( i ∗ , j ∗ ) } is equal to { ( i 1 , j 1 ) , . . . , ( i n cal +1 , j n cal +1 ) } in no particular order (where n cal = |S cal | ), then the test point location ( i ∗ , j ∗ ) is equally likely to be at any one of these n cal +1 locations. Consequently, by exchangeability, the calibration residuals { R ij : ( i, j ) ∈ S cal } , defined in Algorithm 1 above, are exchangeable with the test residual R i ∗ j ∗ = | M i ∗ j ∗ -̂ M i ∗ j ∗ | / ̂ s i ∗ j ∗ ; as a result, we can construct our confidence interval ̂ C ( i ∗ , j ∗ ) with ̂ q determined by the (1 -α ) -quantile of the calibration residuals { R ij : ( i, j ) ∈ S cal } .

Indeed, this is exactly what cmc does in this case: if we are aware that sampling is uniform, it would naturally follow that we estimate ̂ p ij ≡ ̂ p 0 by some (potentially data-dependent) scalar ̂ p 0 ; consequently, all the weights are given by ̂ w ij ≡ 1 n cal +1 and ̂ w test = 1 n cal +1 , and thus ̂ q is the (unweighted) (1 -α ) -quantile of the calibration residuals (with a small correction term, i.e., the term + ̂ w test · δ + ∞ appearing in the definition of ̂ q , to ensure the correct probability of coverage at finite sample sizes).

3.1.2 Weighted exchangeability for the matrix completion problem

Next, we move to the general case, where now the sampling may be highly nonuniform. First suppose the sampling probabilities p ij are known, and suppose again that we are told that the combined set S cal ∪ { ( i ∗ , j ∗ ) } is equal to { ( i 1 , j 1 ) , . . . , ( i n cal +1 , j n cal +1 ) } in no particular order. In this case, the test point ( i ∗ , j ∗ ) is not equally likely to be at any one of these n cal +1 locations. Instead, we have the following result:

Lemma 3.1. Following the notation defined above, if ( i ∗ , j ∗ ) | S ∼ Unif( S c ) , it holds that

Consequently, while the test point's residual R i ∗ j ∗ = | M i ∗ j ∗ -̂ M i ∗ j ∗ | / ̂ s i ∗ j ∗ is not exchangeable with the calibration set residuals { R ij : ( i, j ) ∈ S cal } , the framework of weighted exchangeability (Tibshirani et al., 2019) tells us that we can view R i ∗ j ∗ as a draw from the weighted distribution that places weight w i k j k on each residual R i k j k -and in particular,

where q ∗ i ∗ ,j ∗ = Quantile 1 -α ( ∑ n cal +1 k =1 w i k j k · δ R i k j k ) . Noting that this threshold q ∗ i ∗ ,j ∗ may depend on the test location ( i ∗ , j ∗ ) , to accelerate the computation of prediction sets for all missing entries, we instead propose a one-shot weighted conformal approach by upper bounding the threshold

Now the threshold q ∗ no longer depends on the location ( i ∗ , j ∗ ) of the test point 3 . If the true probabilities p ij were known, then, we could define 'oracle' conformal confidence intervals

As we will see in the proofs below, weighted exchangeability ensures that, if we do have oracle knowledge of the p ij 's, then these confidence intervals satisfy the goal (2) of 1 -α average coverage. Since the p ij 's are not known in general, our algorithm simply replaces the oracle weights w ∗ in (5) with their estimates ̂ w defined in (4), using ̂ p ij as a plug-in estimate of the unknown p ij .

3.2 Theoretical guarantee

In the setting where the p ij 's are not known but instead are estimated, our cmc procedure simply repeats the procedure described above but with weights ̂ w calculated using ̂ p ij 's in place of p ij 's. Our theoretical results therefore need to account for the errors in our estimates ̂ p ij , to quantify the coverage gap of conformal prediction with the presence of these estimation errors. To do this, we define an estimation gap term

where ( i ∗ , j ∗ ) denotes the test point. Effectively, ∆ is quantifying the difference between the 'oracle' weights, w ∗ ij , defined in (5), and the estimated weights, ̂ w ij , defined in (4), except that ∆ is defined relative to a specific test point, while the weights ̂ w (and w ∗ , for the oracle version) provide a 'oneshot' procedure that is universal across all possible test points, and is thus slightly more conservative.

Theorem 3.2. Let ̂ M , ̂ s , and ̂ P be estimates constructed using any algorithms, which depend on the data only through the training samples M S tr at locations S tr . Then, under the notation and definitions above, conformalized matrix completion ( cmc ) satisfies

In the homogeneous case, where we are aware that sampling is uniform (i.e., p ij ≡ p for some potentially unknown p ∈ (0 , 1) ), our error in estimating the weights is given by ∆ ≡ 0 (since h ij is constant across all ( i, j ) , and same for ̂ h ij -and therefore, the true and estimated weights are all equal to 1 n cal +1 ). In this setting, therefore, we would achieve the exact coverage goal (2), with E [ AvgCov ( ̂ C ; M , S ) ] guaranteed to be at least 1 -α .

3.3 Examples for missingness

To characterize the coverage gap explicitly, we present the following concrete examples for P and show how the coverage gap ∆ can be controlled. Technical details are shown in the Supplementary Material.

3.3.1 Logistic missingness

Suppose that the missingness follows a logistic model, with log( p ij / (1 -p ij )) = -log ( h ij ) = u i + v j for some u ∈ R d 1 and v ∈ R d 2 , where we assume u ⊤ 1 = 0 for identifiability. This model is closely related to logistic regression with a diverging number of covariates (Portnoy, 1988; He and Shao, 2000; Wang, 2011). Following Chen et al. (2023), we estimate u , v via maximum likelihood,

Here the log-likelihood is defined by

Consequently, we define the estimate of p ij as ̂ p ij = (1 + exp( -̂ u i -̂ v j )) -1 , for which one can show that E [∆] ≲ √ log(max { d 1 ,d 2 } ) min { d 1 ,d 2 } . The proof of this upper bound is shown in Section A.3.1.

3.3.2 Missingness with a general link function

More broadly, we can consider a general link function, where we assume log( p ij / (1 -p ij )) = -log ( h ij ) = ϕ ( A ij ) , where ϕ is a link function and A ∈ R d 1 × d 2 is the model parameter. The logistic model we introduced above corresponds to the case when ϕ is the identity function and A = u 1 ⊤ + 1 v ⊤ is a rank-2 matrix. In this general setup, if rank( A ) = k ∗ and ∥ A ∥ ∞ ≤ τ , we can apply one-bit matrix completion (Davenport et al., 2014) to estimate { p ij } . More precisely, we define the log-likelihood

where ψ ( t ) = q (1 + e -ϕ ( t ) ) -1 (here rescaling by the constant q is necessary to account for subsampling the training set S tr from the observed data indices S ), and solve the following optimization problem

where ∥ B ∥ ∗ is the nuclear norm of B . Consequently, we let ̂ h ij = exp( -ϕ ( ̂ A ij )) , which leads to E [∆] ≲ min { d 1 , d 2 } -1 / 4 . The proof of this upper bound is shown in Section A.3.2.

4 Simulation studies

In this section, we conduct numerical experiments to verify the coverage guarantee of conformalized matrix completion ( cmc ) using both synthetic and real datasets. As the validity of cmc is independent of the choice of base algorithm, we choose alternating least squares ( als ) (Jain et al., 2013) as the base algorithm (results using a convex relaxation (Fazel et al., 2004; Candes and Recht, 2012; Chen et al., 2020) are shown in the Supplementary Material). We use cmc-als to refer to this combination. We use AvgCov ( ̂ C ) (the average coverage, defined in (3)), and average confidence interval length, AvgLength ( ̂ C ) = 1 |S c | ∑ ( i,j ) ∈S c length ( ̂ C ( i, j ) ) , to evaluate the performance of different methods. All the results in this section can be replicated with the code available at https: //github.com/yugjerry/conf-mc .

4.1 Synthetic dataset

Throughout the experiments, we set the true rank r ∗ = 8 , the desired coverage rate 1 -α = 0 . 90 , and report the average results over 100 random trials.

Data generation. In the synthetic setting, we generate the data matrix by M = M ∗ + E , where M ∗ is a rank r ∗ matrix, and E is a noise matrix (with distribution specified below). The low-rank component M ∗ is generated by M ∗ = κ U ∗ V ∗⊤ , where U ∗ ∈ R d 1 × r ∗ is the orthonormal basis of a random d 1 × r ∗ matrix consisting of i.i.d. entries from a certain distribution P u,v (specified below) and V ∗ ∈ R d 2 × r ∗ is independently generated in the same manner. In addition, the constant κ is chosen such that the average magnitude of the entries | M ∗ ij | is 2 . Following the observation model (1), we only observe the entries in the random set S .

Implementation. First, when implementing als , we adopt r as the hypothesized rank of the underlying matrix. For model-based methods (Chen et al., 2019), by the asymptotic distributional characterization ̂ M ij -M ∗ ij d →N (0 , θ 2 ij ) and E ij ⊥ ( ̂ M ij -M ∗ ij ) , one has ̂ M ij -M ij d →N (0 , θ 2 ij + σ 2 ) . Consequently, the confidence interval for the model-based methods can be constructed as

where q β is the β th quantile of N (0 , 1) . The estimates are obtained via ̂ σ 2 = ∥ M S tr -̂ M S tr ∥ 2 F / |S tr | and ̂ θ 2 ij = ( ̂ σ 2 / ̂ p )( ∥ ̂ U i, · ∥ 2 + ∥ ̂ V j, · ∥ 2 ) with the SVD ̂ M = ̂ U ̂ Σ ̂ V ⊤ . For cmc-als , we obtain ̂ M , ̂ s from M S tr by running als on this training set (and taking ̂ s ij = ( ̂ θ 2 ij + ̂ σ 2 ) 1 / 2 ). The probabilities ̂ p ij 's are estimated via ̂ p ij ≡ ( d 1 d 2 q ) -1 |S tr | for this setting where the p ij 's are homogeneous.

4.1.1 Homogeneous missingness

In the homogeneous case, we consider the following four settings with d 1 = d 2 = 500 :

Setting 1: large sample size + Gaussian noise . p = 0 . 8 , P u,v = N (0 , 1) , and E ij ∼ N (0 , 1) .

Setting 2: small sample size + Gaussian noise . p = 0 . 2 , P u,v = N (0 , 1) , and E ij ∼ N (0 , 1) .

Setting 3: large sample size + heavy-tailed noise . p = 0 . 8 , P u,v = N (0 , 1) , and E ij ∼ 0 . 2 t 1 . 2 , where t ν denotes the t distribution with ν degrees of freedom.

Setting 4: violation of incoherence . p = 0 . 8 , P u,v = t 1 . 2 , and E ij ∼ N (0 , 1) .

Figure 1 displays the results (i.e., the coverage rate and the interval length) for both cmc-als and als when we vary the hypothesized rank r from 2 to 40 . The oracle length is the difference between the (1 -α/ 2) -th and ( α/ 2) -th quantiles of the underlying distribution for E ij . As can be seen, cmc-als achieves nearly exact coverage regardless of the choice of r across all four settings. When the hypothesized rank r is chosen well, cmc-als is often able to achieve an average length similar to that of the oracle.

For the model-based method als , when we underestimate the true rank ( r < r ∗ ), we may still see adequate coverage since ̂ σ tends to over-estimate σ by including the remaining r ∗ -r factors as noise. However, if we overestimate the rank ( r > r ∗ ), then als tends to overfit the noise with additional factors and under-estimate σ regardless of the choice of r , leading to significant undercoverage

5

10

15

20

25

30

35

40

hypothesized rank

0.7

0.8

0.9

1.0

AvgCov

5

10

15

20

25

30

35

40

hypothesized rank

0

5

10

15

AvgLength

cmc-als

als

oracle

(a) Setting 1: large sample size + Gaussian noise

5

10

15

20

25

30

35

40

hypothesized rank

0.25

0.50

0.75

1.00

AvgCov

5

10

15

20

25

30

35

40

hypothesized rank

0

5

10

15

AvgLength

cmc-als

als

oracle

5

10

15

20

25

30

35

40

hypothesized rank

0.7

0.8

0.9

1.0

AvgCov

5

10

15

20

25

30

35

40

hypothesized rank

0

10

20

30

AvgLength

cmc-als

als

oracle

(c) Setting 3: large sample size + heavy-tailed noise

5

10

15

20

25

30

35

40

hypothesized rank

0.7

0.8

0.9

1.0

AvgCov

5

10

15

20

25

30

35

40

hypothesized rank

0

20

40

AvgLength

cmc-als

als

oracle

(d) Setting 4: violation of incoherence

5

0

5

0.0

0.1

0.2

0.3

0.4

0.5

0.6

Density

Alternative least squares

empirical scores

Gaussian(0,1)

0

20

40

index

5.0

2.5

0.0

2.5

5.0

7.5

unobserved entries

Lower and upper bounds

true entry

cmc-als

als

(a) Large sample size + Gaussian noise: r = r ∗ = 8

0

5

0.0

0.1

0.2

0.3

0.4

0.5

0.6

Density

Alternative least squares

empirical scores

Gaussian(0,1)

0

20

40

index

10

5

0

5

unobserved entries

Lower and upper bounds

true entry

cmc-als

als

(b) Large sample size + Gaussian

noise:

r

∗

= 8

<

50 =

r

5

0

5

0.0

0.5

1.0

1.5

2.0

Density

Alternative least squares

empirical scores

Gaussian(0,1)

0

20

40

index

10

0

10

20

unobserved entries

Lower and upper bounds

true entry

cmc-als

als

(c) Violation of incoherence: r = 6 < 8 = r ∗

in Settings 1, 2 and 4. In Setting 3 with heavy-tailed noise, als tends to be conservative due to overestimating σ . Overall, Figure 1 confirms our expectation that the validity of the model-based estimates relies crucially on a well specified model, and it fails to hold when sample size is small (cf. Figure 1b), when noise is heavy tailed (cf. Figure 1c), and when the underlying matrix is not incoherent (cf. Figure 1d).

In Figure 2, we present the histogram of standardized scores ( ̂ M ij -M ij ) / √ ̂ θ 2 ij + ̂ σ 2 and the plot of the upper and lower bounds for three settings. In Figure 2a, when the model assumptions are met and r = r ∗ , the scores match well with the standard Gaussian and the prediction bounds produced by als and cmc-als are similar. With the same data generating process, when the rank is overparametrized, the distribution of scores cannot be captured by the standard Gaussian, thus the quantiles are misspecified. As we can see from the confidence intervals, als tends to have smaller intervals which lead to the undercoverage. In the last setting, the underlying matrix is no longer incoherent. When the rank is underestimated, the r ∗ -r factors will be captured by the noise term and the high heterogeneity in the entries will further lead to overestimated noise level. As a result, the intervals by als are much larger while the conformalized intervals are more adaptive to the magnitude of entries.

4.1.2 Heterogeneous missingness

Now we move on to the case with heterogeneous missingness, where the observed entries are no longer sampled uniformly. To simulate this setting, we draw { a il : i ≤ d 1 , l ≤ k ∗ } i.i.d. from Unif(0 , 1) and { b lj : l ≤ k ∗ , j ≤ d 2 } i.i.d. from Unif( -0 . 5 , 0 . 5) , and define sampling probabilities by

5

10

15

20

25

30

35

40

hypothesized rank

0.85

0.90

0.95

1.00

AvgCov

5

10

15

20

25

30

35

40

hypothesized rank

0

5

10

AvgLength

cmc*-als

cmc-als

k

∗

= 1

.

5

10

15

20

25

30

35

40

hypothesized rank

0.85

0.90

0.95

1.00

AvgCov

5

10

15

20

25

30

35

40

hypothesized rank

0

5

10

AvgLength

cmc*-als

cmc-als

k

∗

= 5

.

5

10

15

20

25

30

35

40

hypothesized rank

0.85

0.90

0.95

1.00

AvgCov

5

10

15

20

25

30

35

40

hypothesized rank

0

5

10

AvgLength

cmc*-als

cmc-als

(c) Adversarial heterogeneous noise:

k

∗

= 1

.

5

10

15

20

25

30

35

40

hypothesized rank

0.85

0.90

0.95

1.00

AvgCov

5

10

15

20

25

30

35

40

hypothesized rank

0

5

10

AvgLength

cmc*-als

cmc-als

(d) Adversarial heterogeneous noise:

k

∗

= 5

.

5

10

15

20

25

30

35

40

hypothesized rank

0.85

0.90

0.95

1.00

AvgCov

5

10

15

20

25

30

35

40

hypothesized rank

0

5

10

AvgLength

cmc*-als

cmc-als

(e) Random heterogeneous noise:

k

∗

= 1

.

5

10

15

20

25

30

35

40

hypothesized rank

0.85

0.90

0.95

1.00

AvgCov

5

10

15

20

25

30

35

40

hypothesized rank

0

5

10

AvgLength

cmc*-als

cmc-als

(f) Random heterogeneous noise:

k

∗

= 5

.

We consider two settings with k ∗ = 1 and k ∗ = 5 , respectively. In both settings, ̂ p ij is constructed by estimating ̂ a il and ̂ b lj via constrained maximum likelihood estimation as is shown in Example 3.3.2.

To analyze the effect of estimation error predicted by Theorem 3.2, we consider three matrix generation processes for each choice of k ∗ , where M ∗ is generated in the same way as in Section 4.1.1 with d 1 = d 2 = d = 500 , P u,v = N (0 , 1) :

Gaussian homogeneous noise : E ij ∼ N (0 , 1) independently.

Adversarial heterogeneous noise : E ij ∼ N (0 , σ 2 ij ) independently, where we take σ ij = 1 / 2 p ij . For this setting, note that the high-noise entries (i.e., those entries that are hardest to predict) occur in locations ( i, j ) that are least likely to be observed during training.

Random heterogeneous noise : E ij ∼ N (0 , σ 2 ij ) independently, where ( σ ij ) ( i,j ) ∈ [ d 1 ] × [ d 2 ] is drawn from the same distribution as (1 / 2 p ij ) ( i,j ) ∈ [ d 1 ] × [ d 2 ] .

We write cmc ∗ -als to denote the conformalized method (with als as the base algorithm) where we use oracle knowledge of the true p ij 's for weighting, while cmc-als uses estimates ̂ p ij . The results are shown in Figures 3, for the settings k ∗ = 1 and k ∗ = 5 (i.e., the rank of P ). Under both homogeneous noise and random heterogeneous noise, both cmc ∗ -als and cmc-als achieve the correct coverage level, with nearly identical interval length. For adversarial heterogeneous noise, on the other hand, cmc ∗ -als achieves the correct coverage level as guaranteed by the theory, but cmc-als shows some undercoverage due to the errors in the estimates ̂ p ij (since now the variance of the noise is adversarially aligned with these errors); nonetheless, the undercoverage is mild. In Section B.2 in the appendix, the local coverage of cmc-als conditioning p ij = p 0 for varying p 0 is presented.

4.2 Real data application

We also compare conformalized approaches with model-based approaches using the Rossmann sales dataset 4 (Farias et al., 2022). This real data provides the underlying fixed matrix M ; the missingness pattern is generated artificially, as detailed below. We focus on daily sales (the unit is 1K dollar) of 1115 drug stores on workdays from Jan 1, 2013 to July 31, 2015 and hence the underlying matrix has dimension 1115 × 780 . The hypothesized rank r is varied from 5 to 60 with step size 5 and we set

5

10 15 20 25 30 35 40 45 50 55 60

hypothesized rank

0.75

0.80

0.85

0.90

0.95

1.00

AvgCov

5

10 15 20 25 30 35 40 45 50 55 60

hypothesized rank

0

1

2

3

4

AvgLength

cmc-als

als

5

10 15 20 25 30 35 40 45 50 55 60

hypothesized rank

0.75

0.80

0.85

0.90

0.95

1.00

AvgCov

5

10 15 20 25 30 35 40 45 50 55 60

hypothesized rank

0

1

2

3

4

AvgLength

cmc-als

als

(a) als vs cmc-als : homogeneous missingness.

(b) als vs cmc-als : logistic missingness.

the target level to be 1 -α = 0 . 9 . We apply random masking for 100 independent trials and report the average for AvgCov and AvgLength in Figure 4.

Homogeneous p ij . For each entry, we first apply random masking 1 -Z ij ∼ Bern(0 . 2) independently. In Figure 4a, conformalized approach has exact coverage at 1 -α but als tends to be conservative when r is small and loses coverage when r increases to 50 .

Heterogeneous p ij . After drawing p ij 's from log( p ij / (1 -p ij )) = -log ( h ij ) = ϕ ( A ij ) in Section 4.1.2 with k ∗ = 1 , ̂ P is obtained via the constrained maximum likelihood estimator in Example 3.3.2. In Figure 4b, cmc-als has coverage around 1 -α and in comparison, when r is greater than 40 , als fails to guarantee the coverage and AvgCov decreases to 0 . 75 .

5 Discussion

In this paper, we introduce the conformalized matrix completion method cmc , which offers a finitesample coverage guarantee for all missing entries on average and requires no assumptions on the underlying matrix or any specification of the algorithm adopted, relying only on an estimate of the entrywise sampling probabilities. Given an estimate of the entrywise sampling probability, we provide an upper bound for the coverage gap and give the explicit form with examples of the logistic as well as the general one-bit low-rank missingness. In the implementation, an efficient one-shot weighted conformal approach is proposed with the provable guarantee and achieves nearly exact coverage.

We can compare our findings to a few related results in the literature. The work of Chernozhukov et al. (2021) applies conformal prediction to counterfactual and synthetic control methods, where they include matrix completion as an example with a regression-type formulation. However, their approach relies on a test statistic that is a function of estimated residuals. Consequently, their method requires the assumption of stationarity and weak dependence of errors. Furthermore, the validity of their approach is contingent upon the estimator being either consistent or stable. Additionally, Wieczorek (2023) studies conformal prediction with samples from a deterministic and finite population, but the validity under the sampling without replacement remains an open question in their work.

Our results suggest a number of questions for further inquiry. In the setting of matrix completion, while the results are assumption-free with regard to the matrix M itself, estimating the sampling probabilities P that determine which entries are observed remains a key step in the procedure; while our empirical results suggest the method is robust to estimation error in this step, studying the robustness properties more formally is an important open question. More broadly, our algorithm provides an example of how the conformal prediction framework can be applied in a setting that is very different from the usual i.i.d. sampling regime, and may lend insights for how to develop conformal methodologies in other applications with non-i.i.d. sampling structure.

References

Amjad, M., Shah, D., and Shen, D. (2018). Robust synthetic control. The Journal of Machine Learning Research , 19(1):802-852.

Angelopoulos, A., Bates, S., Malik, J., and Jordan, M. I. (2020). Uncertainty sets for image classifiers using conformal prediction. arXiv preprint arXiv:2009.14193 .

Athey, S., Bayati, M., Doudchenko, N., Imbens, G., and Khosravi, K. (2021). Matrix completion methods for causal panel data models. Journal of the American Statistical Association , 116(536):1716-1730.

Bai, J. and Ng, S. (2021). Matrix completion, counterfactuals, and factor analysis of missing data. Journal of the American Statistical Association , 116(536):1746-1763.

Barber, R. F., Candès, E. J., Ramdas, A., and Tibshirani, R. J. (2021). Predictive inference with the jackknife+. The Annals of Statistics , 49(1):486 - 507.

Barber, R. F., Candes, E. J., Ramdas, A., and Tibshirani, R. J. (2022). Conformal prediction beyond exchangeability. arXiv preprint arXiv:2202.13415 .

Cai, T. T., Liang, T., and Rakhlin, A. (2016). Geometric inference for general high-dimensional linear inverse problems. The Annals of Statistics , 44(4):1536 - 1563.

Candes, E. and Recht, B. (2012). Exact matrix completion via convex optimization. Communications of the ACM , 55(6):111-119.

Candès, E. J., Lei, L., and Ren, Z. (2021). Conformalized survival analysis. arXiv preprint arXiv:2103.09763 .

Candes, E. J. and Plan, Y. (2010). Matrix completion with noise. Proceedings of the IEEE , 98(6):925936.

Candès, E. J. and Tao, T. (2010). The power of convex relaxation: Near-optimal matrix completion. IEEE Transactions on Information Theory , 56(5):2053-2080.

Cao, Y. and Xie, Y. (2015). Categorical matrix completion. In 2015 IEEE 6th International Workshop on Computational Advances in Multi-Sensor Adaptive Processing (CAMSAP) , pages 369-372. IEEE.

Carpentier, A., Klopp, O., Löffler, M., and Nickl, R. (2018). Adaptive confidence sets for matrix completion. Bernoulli , 24(4A):2429 - 2460.

Chen, W., Chun, K.-J., and Barber, R. F. (2018). Discretized conformal prediction for efficient distribution-free inference. Stat , 7(1):e173.

Chen, W., Wang, Z., Ha, W., and Barber, R. F. (2016). Trimmed conformal prediction for highdimensional models. arXiv preprint arXiv:1611.09933 .

Chen, Y., Chi, Y., Fan, J., Ma, C., and Yan, Y . (2020). Noisy matrix completion: Understanding statistical guarantees for convex relaxation via nonconvex optimization. SIAM journal on optimization , 30(4):3098-3121.

Chen, Y., Fan, J., Ma, C., and Yan, Y. (2019). Inference and uncertainty quantification for noisy matrix completion. Proceedings of the National Academy of Sciences , 116(46):22931-22937.

Chen, Y., Li, C., Ouyang, J., and Xu, G. (2023). Statistical inference for noisy incomplete binary matrix. Journal of Machine Learning Research , 24(95):1-66.

Chernozhukov, V., Wüthrich, K., and Zhu, Y. (2021). An exact and robust conformal inference method for counterfactual and synthetic controls. Journal of the American Statistical Association , 116:1849 - 1864.

Davenport, M. A., Plan, Y., Van Den Berg, E., and Wootters, M. (2014). 1-bit matrix completion. Information and Inference: A Journal of the IMA , 3(3):189-223.

Farias, V., Li, A. A., and Peng, T. (2022). Uncertainty quantification for low-rank matrix completion with heterogeneous and sub-exponential noise. In International Conference on Artificial Intelligence and Statistics , pages 1179-1189. PMLR.

Fazel, M., Hindi, H., and Boyd, S. (2004). Rank minimization and applications in system theory. In Proceedings of the 2004 American control conference , volume 4, pages 3273-3278. IEEE.

Foygel, R. and Srebro, N. (2011). Concentration-based guarantees for low-rank matrix reconstruction. In Kakade, S. M. and von Luxburg, U., editors, Proceedings of the 24th Annual Conference on Learning Theory , volume 19 of Proceedings of Machine Learning Research , pages 315-340, Budapest, Hungary. PMLR.

He, X. and Shao, Q.-M. (2000). On parameters of increasing dimensions. Journal of multivariate analysis , 73(1):120-135.

Jain, P., Netrapalli, P., and Sanghavi, S. (2013). Low-rank matrix completion using alternating minimization. In Proceedings of the forty-fifth annual ACM symposium on Theory of computing , pages 665-674.

Jin, Y., Ren, Z., and Candès, E. J. (2023). Sensitivity analysis of individual treatment effects: A robust conformal inference approach. Proceedings of the National Academy of Sciences , 120(6):e2214889120.

Keshavan, R. H., Montanari, A., and Oh, S. (2010). Matrix completion from a few entries. IEEE transactions on information theory , 56(6):2980-2998.

Koltchinskii, V., Lounici, K., and Tsybakov, A. B. (2011). Nuclear-norm penalization and optimal rates for noisy low-rank matrix completion. The Annals of Statistics , 39(5):2302 - 2329.

Lei, J., G'Sell, M., Rinaldo, A., Tibshirani, R. J., and Wasserman, L. (2018). Distribution-free predictive inference for regression. Journal of the American Statistical Association , 113(523):10941111.

Lei, L. and Candès, E. J. (2020). Conformal inference of counterfactuals and individual treatment effects. arXiv preprint arXiv:2006.06138 .

Ma, C., Wang, K., Chi, Y., and Chen, Y. (2020). Implicit regularization in nonconvex statistical estimation: Gradient descent converges linearly for phase retrieval, matrix completion, and blind deconvolution. Foundations of Computational Mathematics , 20:451-632.

Nair, J., Wierman, A., and Zwart, B. (2022). The fundamentals of heavy tails: Properties, emergence, and estimation , volume 53. Cambridge University Press.

Natarajan, N. and Dhillon, I. S. (2014). Inductive matrix completion for predicting gene-disease associations. Bioinformatics , 30(12):i60-i68.

Negahban, S. and Wainwright, M. J. (2012). Restricted strong convexity and weighted matrix completion: Optimal bounds with noise. The Journal of Machine Learning Research , 13(1):16651697.

Portnoy, S. (1988). Asymptotic behavior of likelihood methods for exponential families when the number of parameters tends to infinity. The Annals of Statistics , pages 356-366.

Rennie, J. D. and Srebro, N. (2005). Fast maximum margin matrix factorization for collaborative prediction. In Proceedings of the 22nd international conference on Machine learning , pages 713-719.

Romano, Y., Sesia, M., and Candes, E. (2020). Classification with valid and adaptive coverage. Advances in Neural Information Processing Systems , 33:3581-3591.

Shafer, G. and Vovk, V. (2008). A tutorial on conformal prediction. Journal of Machine Learning Research , 9(3).

Srebro, N. and Shraibman, A. (2005). Rank, trace-norm and max-norm. In Learning Theory: 18th Annual Conference on Learning Theory, COLT 2005, Bertinoro, Italy, June 27-30, 2005. Proceedings 18 , pages 545-560. Springer.

Sun, R. and Luo, Z.-Q. (2016). Guaranteed matrix completion via non-convex factorization. IEEE Transactions on Information Theory , 62(11):6535-6579.

Tibshirani, R. J., Barber, R. F., Candès, E. J., and Ramdas, A. (2019). Conformal prediction under covariate shift. In NeurIPS .

Vovk, V. (2015). Cross-conformal predictors. Annals of Mathematics and Artificial Intelligence , 74:9-28.

Vovk, V., Gammerman, A., and Shafer, G. (2005). Algorithmic learning in a random world , volume 29. Springer.

Wang, L. (2011). Gee analysis of clustered binary data with diverging number of covariates. The Annals of Statistics .

Wieczorek, J. (2023). Design-based conformal prediction. arXiv preprint arXiv:2303.01422 .

Xia, D. and Yuan, M. (2021). Statistical inferences of linear forms for noisy matrix completion. Journal of the Royal Statistical Society Series B: Statistical Methodology , 83(1):58-77.

Yang, Y. and Ma, C. (2022). Optimal tuning-free convex relaxation for noisy matrix completion. arXiv preprint arXiv:2207.05802 .

Zhao, Y. and Udell, M. (2020). Matrix completion with quantified uncertainty through low rank gaussian copula. Advances in Neural Information Processing Systems , 33:20977-20988.

Contents

A Proofs

The section collects the proofs for the technical results in the main text. Section A.1 and A.2 are devoted to the proof of Lemma 3.1 and Theorem 3.2, respectively. In the end, Section A.3 provides proofs for the coverage gap bounds in the two examples in Section 3.3.

A.1 Proof of Lemma 3.1

Fix a set of locations B := { ( i 1 , j 1 ) , . . . , ( i n cal +1 , j n cal +1 ) } , and an index 1 ≤ m ≤ n cal +1 . By the definition of conditional probability, we have

It then boils down to computing P ( S cal = S 1 , ( i ∗ , j ∗ ) = ( i l , j l ) | S tr = S 0 , |S| = n ) for any fixed set S 1 and fixed location ( i l , j l ) / ∈ S 0 ∪ S 1 . To this end, we have the following claim (we defer its

proof to the end of this section)

where h ij = (1 -p ij ) /p ij . This finishes the proof.

Proof of Equation (8) . Denote W the matrix consisting of entries W ij defined in Algorithm 1. The matrix Z consists of Z ij which are the indicators of non-missingness, i.e. Z ij = ✶ { ( i, j ) is observed } . Fix any two disjoint subsets S 0 , S 1 ⊆ [ d 1 ] × [ d 2 ] with |S 0 | + |S 1 | = n . We have

Since W and Z are independent, one further has

where the last two identities are based on direct computations.

Based on (10), one can further compute

where the last identity uses Equation (10).

Now we are ready to prove (8). Recall that the new data point ( i ∗ , j ∗ ) | S is drawn uniformly from Unif( S c ) . Therefore one has

where the last line follows from (10) and (11).

A.2 Proof of Theorem 3.2

First, fix any a ∈ [0 , 1] . Lemma 3.1, together with the weighted conformal prediction framework of Tibshirani et al. (2019), implies that where

and

Indeed, here a can be any function of the random variables we are conditioning on-that is, a may depend on n , on S tr , and on S cal ∪ { ( i ∗ , j ∗ ) } .

Next define a = α +∆ where ∆ is defined as in (6). We observe that, since each ̂ h ij is a function of S tr , then ∆ (and thus also a ) can therefore be expressed as a function of n , S tr , and S cal ∪{ ( i ∗ , j ∗ ) } . Therefore, applying the work above, we have

and thus, after marginalizing,

Next, we verify that

holds almost surely-if this is indeed the case, then we have shown that

which establishes the desired result. Thus we only need to show that ̂ q ≥ q ∗ i ∗ j ∗ ( α +∆) , or equivalently,

Quantile

for all ( i, j ) ∈ S ∪ { ( i ∗ , j ∗ ) } . Then by definition of ̂ w , we see that w ′ ij ≥ ̂ w ij for ( i, j ) ∈ S , and therefore,

Quantile

holds almost surely. Therefore it suffices to show that

holds almost surely. Indeed, we have

where d TV denotes the total variation distance. This completes the proof.

A.3 Proofs for examples in Section 3.3

Recall the definition of ∆ :

Note that ∆ ≤ 1 by definition. We start with stating a useful lemma to bound the coverage gap ∆ using the estimation error of ̂ h ij .

Lemma A.1. By the definition of the coverage gap ∆ , we have

A.3.1 Proof for Example 3.3.1

Under the logistic model, one has for any ( i, j )

In this case, if ̂ u and ̂ v are the constrained maximum likelihood estimators in Example 3.3.1, Theorem 6 in Chen et al. (2023) implies that

with the proviso that ∥ u ∥ ∞ + ∥ v ∥ ∞ ≤ τ < ∞ , d 2 ≫ √ d 1 log d 1 and d 1 ≫ (log d 2 ) 2 . Then for h ij = exp( -u i -v j ) and ̂ h ij = exp( -̂ u i -̂ v j ) , we have

Further, as min i,j h i,j = min i,j exp( -u i -v j ) ≥ e -τ , then with probability approaching one, for every ( i, j ) ∈ [ d 1 ] × [ d 2 ] , one has ̂ h i,j ≥ h i,j -| h ij -̂ h ij | ≥ e -τ / 2 =: h 0 . By the upper bound (14), we have

Further, as ∆ ≤ 1 , we have E [∆] ≲ √ log max { d 1 ,d 2 } min { d 1 ,d 2 } .

A.3.2 Proof of Example 3.3.2

Define the link function ψ ( t ) = q (1 + e -ϕ ( t ) ) , where ϕ is monotonic. Applying Theorem 1 in the paper (Davenport et al., 2014), we obtain that with probability at least 1 -C 1 / ( d 1 + d 2 )

with the proviso that d 1 d 2 ≥ ( d 1 + d 2 ) log( d 1 d 2 ) . Here ˜ C τ = 2 39 / 4 e 9 / 4 (1 + √ 6) τL τ β τ with

Denote this high probability event to be E 0 . Since ∥ A ∥ 1 ≤ √ d 1 d 2 ∥ A ∥ F , on this event E 0 , we further have

where C τ = ζ √ L τ β τ and ζ is a universal constant.

Recall that h ij = exp( -ϕ ( A ij )) and ∥ A ∥ ∞ ≤ τ . On the same event E 0 , we further have

where C ′ τ = 2 e ϕ ( τ ) √ ˜ C τ

By the feasibility of the minimizer ̂ A and the fact that ̂ h ij = exp( -ϕ ( ̂ A ij )) , we have ̂ h ij ≥ h 0 = e -ϕ ( τ ) for all ( i, j ) . This together with the upper bound (14) implies that

Define a second high probability event E 1 = { n cal ≥ (1 -c )(1 -q ) ∥ P ∥ 1 } . Using the Chernoff bound, we have P ( E 1 ) ≥ 1 -C 0 ( d 1 d 2 ) -5 . Therefore, on the event E 0 ∩ E 1 , we have

Under the assumptions that p ij = 1 / (1 + e -ϕ ( A ij ) ) , and that ∥ A ∥ ∞ ≤ τ , we know that p ij ≥ C 2 for some constant that only depends on τ . As a result, we have ∥ P ∥ 1 ≥ C 2 d 1 d 2 , which further leads to the conclusion that

on the event E 0 ∩ E 1 . In addition, on the small probability event ( E 0 ∩ E 1 ) , one trivially has ∆ ≤ 1 . Therefore simple combinations of the cases yields the desired bound on E [∆] .

A.3.3 Proof of Lemma A.1

Reusing the definition of w ′ (13), one has

This completes the proof.

B Additional numerical experiments

In this section, we provide additional simulation results. In Section B.1, we compare the proposed one-shot weighted approach with the exact weighted conformal prediction. In Section B.2, the

approach

0.88

0.89

0.90

0.91

0.92

coverage

approach

3.5

3.6

3.7

3.8

3.9

4.0

length

0.0

0.2

0.4

0.6

0.8

1.0

pij

=

p

0

0.88

0.90

0.92

Conditional coverage

pij

=

p

0

0.0

0.2

0.4

0.6

0.8

1.0

pij

=

p

0

3.0

3.2

3.4

3.6

3.8

4.0

Length

local coverage of cmc-als conditioning on p ij is evaluated. We present the simulation results for the convex relaxation ( cvx ) and the conformalized convex method ( cmc-cvx ) in both settings with homogeneous and heterogeneous missingness in Section B.3 and B.5. We further evaluate the performance of cmc-als when the sampling probability is misspecified in Section B.6.

B.1 Comparison between one-shot and exact weighted approaches

We conduct comparison in the setting with heterogeneous missingness and Gaussian noise as shown in Section 4.1.2 with k ∗ = 12 . From the result in Figure 5a, the performance of two aproaches are essentially the same while the one-shot cmc-als is more computationally efficient.

B.2 Local coverage of cmc

In Figure 5b, we evaluate the local performance of cmc-als when conditioning on { p ij = p 0 } , i.e. a subpopulation determined by the value of sampling probability. We use the setting with heterogeneous missingness and Gaussian noise as shown in Section 4.1.2 with k ∗ = 12 and p 0 ranging from 0 . 1 to 0 . 9 with a step size of 0 . 1 . The conditional coverage is approximated via kernel smoothing: with the indicators A ij for coverage, we use the weight K ij = ϕ p 0 ,h ( p ij ) and calculate the conditional coverage by ( ∑ A ij K ij ) / ( ∑ K ij ) . Here ϕ µ,σ is the density function of N ( µ, σ 2 ) and h = 0 . 05 . When p 0 increases from 0 . 2 to 0 . 8 , the conditional coverage increases and stays around 0 . 9 . At p 0 = 0 . 1 , 0 . 9 , the coverage varies much, which is due to the small effective sample size for the edge values of p ij . Moreover, for uniform sampling, since the rows and columns are generated i.i.d., there are no meaningful subgroups of the data to condition on.

B.3 Additional results for homogeneous misingness

In this section, we present the results for synthetic simulation with the convex relaxation cvx and the conformalized convex matrix completion method cmc-cvx . Setting 1, 2, 3, 4 are the same as introduced in Section4.1. The true rank is r ∗ = 8 and the hypothesized rank varies from 4 to 24 with the stepsize 4 .

The conformalized methods, regardless of the based algorithm adopted, have nearly exact coverage around 1 -α . But we can observe different behaviors between als and cvx since the convex relaxation is free of the choice of r until the projection of ̂ M cvx onto the rankr subspace (Chen et al., 2020). As a result, when r > r ∗ , cvx tends to overestimate the strength of the noise. In Figure 6a, 6b and 6d, when r > r ∗ , cvx has coverage rate higher than the target level and the confidence interval is more conservative than conformalized methods. Since the accuracy of cvx is based on the large sample size, in Figure 6b, when the effective sample size is insufficient with small p ij , the residuals from cvx have a large deviation from the standard distribution and the intervals are much larger than the oracle ones. Besides, in Figure 6c, when the noise has heavy tails than Gaussian variables, cvx overestimates the noise strength similar to als and is conservative in coverage. When the incoherence condition is violated in Figure 6d, if r < r ∗ , both cvx and als fit the missed factor by overestimating the noise strength and produce extremely large intervals.

B.4 Estimation error for one-bit matrix estimation

The estimation error in ̂ p ij can be visualized from the following heatmaps comparing P and ̂ P . Here the entries are sorted by the order of p ij for each row and each column.

5

10

15

20

hypothesized rank

0.7

0.8

0.9

1.0

AvgCov

5

10

15

20

hypothesized rank

0

5

10

15

AvgLength

cmc-als

als

cmc-cvx

cvx

oracle

(a) Setting 1: Large sample size + Gaussian noise

5

10

15

20

hypothesized rank

0.4

0.6

0.8

1.0

AvgCov

5

10

15

20

hypothesized rank

0

10

20

AvgLength

cmc-als

als

cmc-cvx

cvx

oracle

(b) Setting 2: Small sample size+Gaussian noise

5

10

15

20

hypothesized rank

0.7

0.8

0.9

1.0

AvgCov

5

10

15

20

hypothesized rank

0

20

40

60

AvgLength

cmc-als

als

cmc-cvx

cvx

oracle

5

10

15

20

hypothesized rank

0.7

0.8

0.9

1.0

AvgCov

5

10

15

20

hypothesized rank

0

25

50

75

AvgLength

cmc-als

als

cmc-cvx

cvx

oracle

(c) Setting 3: Large sample size + heavy-tailed noise

(d) Setting 4: Violation of incoherence

(a)

k

∗

= 1

:

d

1

=

d

2

= 500

.

(b)

k

∗

= 1

:

d

1

=

d

2

= 2000

.

(c)

k

∗

= 5

:

d

1

=

d

2

= 500

.

(d)

k

∗

= 5

:

d

1

=

d

2

= 2000

.

True P

0.65

Estimated P

0.65

True P

0.65

Estimated P

0.65

0.60

0.60

0.60

0.60

125

125

500

500

0.55

0.55

0.55

0.55

250

- 0.50250

- 0.50

1000

- 0.50 1000

- 0.50

0.45

0.45

0.45

0.45

375

375

1500

1500

0.40

0.40

0.40

0.40

500

0.35500

0.35

2000

0.352000

0.35

125

250

375

500

125

250

375

0OS

500

1000

1500

2000

500

1000

1500

2000

True P

Estimated P

True P

Estimated P

0.8

0.8

0.8

0.8

125

125

500

500

0.6

0.6

0.6

0.6

250

250

1000

1000

0.4

0.4

0.4

0.4

375

375

1500

1500

0.2

0.2

0.2

0.2

500

500

2000

2000

125

250

375

500

125

250

375

500

500

1000

1500

2000

500

1000

1500

2000

B.5 Additional results for heterogeneous missingness

In Figure 8, we present the results for synthetic simulation with the convex relaxation cvx as the base algorithm, where we denote cmc-cvx and cmc ∗ -cvx as the conformalized matrix completion method with estimated weights and true weights, respectively. Three settings with heterogeneous missingness are the same as Figure 3.

B.6 Misspecified sampling probability

We consider the following four settings:

(a) The underlying missingness follows the rank-one model where p ij = a i b j , both a i and b j are generated i.i.d. from Unif(0 . 2 , 1) . The noise is adversarial. But we estimate p ij via the one-bit matrix completion based on the logistic model (working model with hypothesized rank k = 5 ).

(b) The underlying missingness follows the logistic model as in Example 3.3.2 with k ∗ = 5 and we adopt the adversarial noise. But p ij is estimated under the assumption of uniform sampling (working model), i.e. ̂ p ij = ̂ p .

5

10

15

20

hypothesized rank

0.85

0.90

0.95

1.00

AvgCov

5

10

15

20

hypothesized rank

0

5

10

AvgLength

cmc*-als

cmc-als

cmc*-cvx

cmc-cvx

k

∗

= 1

.

5

10

15

20

hypothesized rank

0.85

0.90

0.95

1.00

AvgCov

5

10

15

20

hypothesized rank

0

5

10

AvgLength

cmc*-als

cmc-als

cmc*-cvx

cmc-cvx

k

∗

= 5

.

5

10

15

20

hypothesized rank

0.85

0.90

0.95

1.00

AvgCov

5

10

15

20

hypothesized rank

0

5

10

AvgLength

cmc*-als

cmc-als

cmc*-cvx

cmc-cvx

(c) Adversarial heterogeneous noise:

k

∗

= 1

.

5

10

15

20

hypothesized rank

0.85

0.90

0.95

1.00

AvgCov

5

10

15

20

hypothesized rank

0

5

10

AvgLength

cmc*-als

cmc-als

cmc*-cvx

cmc-cvx

(d) Adversarial heterogeneous noise:

k

∗

= 5

.

5

10

15

20

hypothesized rank

0.85

0.90

0.95

1.00

AvgCov

5

10

15

20

hypothesized rank

0

5

10

AvgLength

cmc*-als

cmc-als

cmc*-cvx

cmc-cvx

(e) Random heterogeneous noise:

k

∗

= 1

.

5

10

15

20

hypothesized rank

0.85

0.90

0.95

1.00

AvgCov

5

10

15

20

hypothesized rank

0

5

10

AvgLength

cmc*-als

cmc-als

cmc*-cvx

cmc-cvx

(f) Random heterogeneous noise:

k

∗

= 5

.

5

10

15

20

hypothesized rank

0.80

0.85

0.90

0.95

1.00

AvgCov

5

10

15

20

hypothesized rank

0

10

20

30

40

AvgLength

cmc-als-mis

cmc-als

5

10

15

20

hypothesized rank

0.80

0.85

0.90

0.95

1.00

AvgCov

5

10

15

20

hypothesized rank

0.0

2.5

5.0

7.5

10.0

12.5

AvgLength

cmc-als-mis

cmc-als

(b) Estimating logistic model with uniform sampling (adversarial noise).

5

10

15

20

hypothesized rank

0.80

0.85

0.90

0.95

1.00

AvgCov

5

10

15

20

hypothesized rank

0

5

10

15

20

25

AvgLength

cmc-als-mis

cmc-als

5

10

15

20

hypothesized rank

0.80

0.85

0.90

0.95

1.00

AvgCov

5

10

15

20

hypothesized rank

0.0

2.5

5.0

7.5

10.0

12.5

AvgLength

cmc-als-mis

cmc-als

(c) Estimating rank-1 model with logistic model (random noise).

(d) Estimating logistic model with uniform sampling (random noise).

(c) Same as (a) except that we use the random heterogeneous noise.

(d) Same as (b) except that we use the random heterogeneous noise.

From the results, we can see that when the noise is generated following the random heterogeneous model, where the values of entries are independent of the sampling probabilities, the misspecification of the sampling model only slightly affects the coverage. Moreover, when the noise is generated in the adversarial way, where the values of entries depend on p ij 's, we can see that the coverage with a misspecified sampling model is lower than the target level, but is above 0 . 85 in practice, which depends on the divergence between the true and the working sampling models.

B.7 Additional results for the sales dataset

Denote M the underlying matrix in the sales dataset. In Figure 10a, we plot singular values of M and top5 singular values contain a large proportion of the information. In Figure 10b, we plot the histogram of entries M ij 's of the underlying matrix, and the sales dataset has the range from 0 to over 20 thousand with a heavy tail.

10

10

10

10

4

3

2

1

400

0

200

600

800

(a) Singular values of the underlying matrix M .

0

10

20

30

40

0.00

0.05

0.10

0.15

(b) Histogram of entries in

M

.

B.7.1 Comparison under misspecified missingness

We consider the sales dataset where the missingness occurs in the way that: for weekdays, each entry is observed with p = 0 . 8 . On weekends, as the stores are likely to be operated by less experienced interns or are less frequent to report the sales data, each entry is observed with a lower probability, e.g. 0 . 8 / 3 . Moreover, as there could be a subgroup of stores that are less frequent in sales reporting, 200 stores are randomly sampled, for which p = 0 . 8 / 3 . We use the logistic model with k = 5 to estimate p ij . Comparison between als and cmc-als is shown in Figure 11.

B.7.2 Results with convex optimization

In Figure 12, cmc-cvx has nearly exact coverage at 1 -α , but cvx tends to have higher coverage than the target level. Besides, the convex approach has much larger intervals when r is large, which can be caused by the overfitting of the observed entries. As conformalized approach leaves out a proportion of observed entries as the training set, intervals produced by cmc-cvx are less accurate than cvx due to the poorly behaved base algorithm.

5

15

25

35

45

55

hypothesized rank

0.80

0.85

0.90

0.95

1.00

AveCov

5

15

25

35

45

55

hypothesized rank

0

1

2

3

4

AveLen

cmc-als

als

5

15

25

35

45

55

65

hypothesized rank

0.75

0.80

0.85

0.90

0.95

1.00

AvgCov

5

15

25

35

45

55

65

hypothesized rank

0

2

4

6

8

AvgLength

cmc-als

als

cmc-cvx

cvx

(a) Homogeneous missingness.

5

15

25

35

45

55

65

hypothesized rank

0.75

0.80

0.85

0.90

0.95

1.00

AvgCov

5

15

25

35

45

55

65

hypothesized rank

0

5

10

15

20

AvgLength

cmc-als

als

cmc-cvx

cvx

(b) Heterogeneous missingness with

k

∗

= 1

.

C Additional details of algorithms and extensions

C.1 Extension to likelihood-based scores and categorical matrices

We will show in this section that cmc can also be applied to categorical matrix completion (Cao and Xie, 2015) or a more general setting in (19), the validity of which is also guaranteed by the presented theorem.

Setup To formulate the problem, consider an underlying parameter matrix M ∗ ∈ [ d 1 ] × [ d 2 ] and the observations { M ij : i ∈ [ d 1 ] , j ∈ [ d 2 ] } are drawn from the distribution

where {P θ } θ ∈ Θ can be a family of parametric distributions with probability density p θ . The categorical matrix completion is a specific example where the support of M ij is finite or countable. For example, a Poisson matrix is generated by M ij ∼ Pois( M ∗ ij ) , where M ∗ ij is the Poisson mean. Similar to the previous setup, we treat M as deterministic, and entries in the subset S ⊆ [ d 1 ] × [ d 2 ] are available. Here S is sampled in the same manner as before with the matrix P ∈ [ d 1 ] × [ d 2 ] .

Split conformal approach Consider the split approach with the partition S = S tr ∪ S cal as in Algorithm 1. With the training set M tr , we obtain an estimated likelihood function ̂ π ( m ; i, j ) such that

which is an estimate for the true likelihood of M ij at m . The estimation can be feasible given certain low-complexity structures. For example, if a hypothesized distribution family {Q θ } θ ∈ Θ with probability density q θ is given and the underlying mean matrix M ∗ is assumed to be low-rank. Then, M can be viewed as a perturbation of M ∗ and we can estimate M ∗ via matrix completion algorithms with entries in M tr . Denote ̂ M as the estimate for M ∗ , then we have the estimated likelihood

The odds ratios are also estimated from the training set, i.e. ̂ h ij , from which we compute the weights

For each ( i, j ) ∈ S cal , calculate the likelihood-based nonconformity score

Then, for any test point ( i ∗ , j ∗ ) ∈ S c , we can construct the confidence interval

where ̂ q is the weighted quantile

More examples of conformal methods for classification are shown in Romano et al. (2020), Angelopoulos et al. (2020), etc.

C.2 Full conformalized matrix completion

In Algorithm 2, the procedure of the full conformalized matrix completion ( full-cmc ) is presented. This full conformal version of cmc offers the same coverage guarantee as given in Theorem 3.2 for the split version of cmc (except with the entire observed set S in place of S cal , when defining ∆( ̂ w ) ; the formal proof of this bound for full conformal is very similar to the proof of Theorem 3.2, using an analogous weighted exchangeability argument as in Lemma 3.1, and so we omit it here.

To define this algorithm, we need some notation: given the observed data M S , plus a test point location ( i ∗ , j ∗ ) and a hypothesized value m for the test point M i ∗ j ∗ , define a matrix M ( m ) with entries

where, abusing notation, ' M ij = ∅ ' denotes that no information is observed in this entry.

Algorithm 2 full-cmc : full conformalized matrix completion

1: Input : target level 1 -α ; partially observed matrix M S .

2: Using the training data M S , compute an estimate ̂ P of the observation probabilities (with ̂ p ij estimating p ij , the probability of entry ( i, j ) being observed).

3: for ( i ∗ , j ∗ ) in S c do

4: for m ∈ M do

5: Augment M S with one additional hypothesized entry, { M i ∗ ,j ∗ = m } , to obtain M ( m ) defined as in (20).

6: Using the imputed matrix M ( m ) S , compute:

An initial estimate ̂ M ( m ) using any matrix completion algorithm (with ̂ M ( m ) ij estimating the target M ij );

Optionally, a local uncertainty estimate ̂ s ( m ) (with ̂ s ( m ) ij estimating our relative uncertainty in the estimate ̂ M ( m ) ij ), or otherwise set ̂ s ( m ) ij ≡ 1 ;

An estimate ̂ P of the observation probabilities (with ̂ p ij estimating p ij , the probability of entry ( i, j ) being observed).

7: Compute normalized residuals for ( i, j ) ∈ S ∪ { ( i ∗ , j ∗ ) } ,

8: Compute weights

9: Compute the weighted quantile

10: end for

11: end for

We note that, when M = R (or an infinite subset of R ), the computation of the prediction set is impossible in most of the cases. In that case, our algorithm can be modified via a trimmed or discretized approximation; these extensions are presented for the regression setting in the work of

Chen et al. (2016, 2018), and can be extended to the matrix completion setting in a straightforward way.

C.3 Exact split conformalized matrix completion

In Algorithm 3, we present the exact split approach, which is less conservative than our one-shot approach given in Algorithm 1, but may be less computationally efficient. In this version of the algorithm, the quantile ̂ q = ̂ q ( i ∗ , j ∗ ) needs to be computed for each missing entry since the weight vector ̂ w depends on the value of ̂ p i ∗ ,j ∗ .

Algorithm 3 split-cmc : split conformalized matrix completion

1: Input : target coverage level 1 -α ; data splitting proportion q ∈ (0 , 1) ; observed entries M S .

2: Split the data: draw W ij i.i.d. ∼ Bern( q ) , and define training and calibration sets,

3: Using the training data M S tr indexed by S tr ⊆ [ d 1 ] × [ d 2 ] , compute:

An initial estimate ̂ M using any matrix completion algorithm (with ̂ M ij estimating the target M ij );

Optionally, a local uncertainty estimate ̂ s (with ̂ s ij estimating our relative uncertainty in the estimate ̂ M ij ), or otherwise set ̂ s ij ≡ 1 ;

An estimate ̂ P of the observation probabilities (with ̂ p ij estimating p ij , the probability of entry ( i, j ) being observed).

4: Compute normalized residuals on the calibration set,

5: Compute estimated odds ratios for the calibration set and test set,

6: for ( i ∗ , j ∗ ) ∈ S c do

7: Compute weights for the calibration set and test point,

8: Compute threshold

where δ t denotes the point mass at t .

9: end for

10: Output : confidence intervals

for each unobserved entry ( i ∗ , j ∗ ) ∈ S c .

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper addresses the problem of uncertainty quantification in matrix completion by developing a distribution-free method for predictive inference.  The authors propose a novel approach based on the conformal prediction framework, aiming to overcome the limitations imposed by stringent model assumptions such as low-rankness of the underlying matrix and the light-tailed noise.  Their method, referred to as Conformalized Matrix Completion (CMC; Algorithm 1), can be combined with any arbitrary matrix completion algorithm to provide confidence intervals for the imputed matrix entries.  Here, the “confidence” is measured with respect to the randomness inherent in the measurements of the matrix entries, assuming a randomized probabilistic measurement model.  \n\nTo support their proposed method, the authors also present an intuitive explanation of the method and a theoretical guarantee.  They start by giving an intuitive exposition of how the (weighted) exchangeability plays a crucial role in constructing confidence intervals within the conformal prediction framework (Section 3.1).  Thereafter, they present Theorem 3.2, which establishes a theoretical guarantee on the average coverage of the confidence intervals, which serves as a key theoretical result of this paper.  Additionally, the authors illustrate their analysis by describing two examples of the missingness patterns (i.e., the random observation models) in Section 3.3.  Through this illustration, they explicitly evaluate the rate of the expected “weight gap” as expressed in Eq. (6), ultimately demonstrating satisfactory coverage of the constructed confidence intervals. \n\nIn Section 4, the authors report the results of their numerical simulations. These simulations serve two purposes: (1) comparing the performance of the proposed conformalized method against a model-based baseline (Sections 4.1.1 & 4.2), and (2) investigating the stability of the proposed method when the oracle knowledge of the measurement probability for each entries is not available (Section 4.1.2).\n\nSTRENGTHS:\nOne of the primary strengths of this paper lies in its adaptation of the conformal prediction framework to address the challenge of uncertainty quantification in matrix completion. Unlike previous approaches in the literature, this method does not heavily rely on specific model assumptions. This combination of conformal prediction and matrix completion is a significant and original contribution, making the paper stand out in its field.\n\nAnother notable strength is the careful organization of the paper, which effectively presents the essential components of the study to support the authors' main claims. Section 2 provides a concise overview of the problem setup and evaluation metric. The proposed method is then described in detail, accompanied by an intuitive explanation based on the concept of exchangeability in Section 3.1. The main theoretical result is presented neatly in Section 3.2, and to further enhance the understanding, concrete examples of the missingness patterns are provided in Section 3.3. The paper is further strengthened by the inclusion of numerical study results in Section 4 and a comprehensive discussion in Section 5. As a result, the paper forms a clear and self-contained report of the study, ensuring that the readers can grasp the main ideas and findings with ease.\n\nOverall, the adaptation of conformal prediction, the clear organization of the paper, and its ability to convey the study's main contributions effectively are key strengths of this research work.\n\nWEAKNESSES:\nWhile the paper exhibits notable strengths, there are a few areas that could benefit from further improvement and clarification. I have identified three main concerns that warrant attention.\n\nFirstly, the authors frequently emphasize that their proposed method is \"distribution-free\" and does not rely on “any” assumptions about the underlying matrix (e.g., in line 60). However, it is important to explicitly acknowledge that the proposed method and analysis do depend on the probabilistic random observation model, which is necessary for the conformal prediction framework. While this limitation does not appear to be critical, it would be beneficial to provide a careful clarification of the assumptions and limitations to avoid potential confusion.\n\nSecondly, although the authors highlight the limitations of existing uncertainty quantification approaches (lines 24 - 30), there is a lack of comparisons between the proposed method and these previous approaches. Consequently, it remains unclear whether and how the proposed method surpasses these existing methods. Including such comparisons would enhance the clarity and strengthen the argument for the superiority of the proposed approach. \n\nAdditionally, the issue of heavy-tailed noise is briefly mentioned by the authors (lines 32 - 33), who suggest that their method is less sensitive to the noise tail (lines 32-33, lines 257-261). However, this point is not adequately addressed beyond the comparison of the vanilla ALS method to the CMC-ALS method in Figure 1-(c), where the latter seems to be slightly better than the former, but still remains overly conservative; the difference in performance between the ALS and the CMC-ALS does not appear to be substantial (approximately 9x worse vs 6x worse than the oracle). To improve the cohesiveness of the paper, it would be beneficial to either elaborate further on this point or consider removing it if it does not significantly contribute to the main findings.\n\nIn summary, while the paper possesses strengths, it would benefit from addressing these weaknesses. Clearer clarification of assumptions and limitations, comparative analyses with existing methods, and a more comprehensive discussion of the performance in the presence of heavy-tailed noise would enhance the overall quality and cohesiveness of the paper.\n\nQUESTIONS:\n1.  I suggest the authors consider providing a clarification of the assumption on the measurement model, as mentioned in the first point of the \"Weakness\" section. This clarification would help readers better understand the specific assumptions and limitations of the proposed method.\n\n2.  It would be valuable if the authors could compare their method with the approaches mentioned in the second paragraph (lines 24-30) by conducting simulations or other appropriate means. Such comparisons would enable a clearer assessment of how the proposed method outperforms or differs from existing methods.\n\n3.  I am curious about the degree to which the proposed CMC method relies on the uncertainty estimate \\hat{s}. Many matrix estimation algorithms only provide point estimates for the entries without accompanying uncertainty estimates. In Algorithm 1, when uncertainty estimates are unavailable, the authors set \\hat{s}_{ij} = 1 by default (line 3). However, this default choice can be problematic. For instance, consider two scenarios: (1) estimating M and (2) estimating 10M. In scenario 2, the uncertainty should be ten times larger than in scenario 1, but the default choice cannot account for this. It would be beneficial if the authors could address this issue and discuss potential alternatives or adjustments to handle situations where uncertainty estimates are not provided.\n\n4.  In Section 3.3, it might be helpful for readers to have either the upper bound for the expected weight gap or the resulting lower bound for average coverage presented as explicit corollaries.\n\n5.  The caption of Figure 2 does not provide sufficient information to understand what is being compared. It would be helpful to revise the caption to clearly indicate the elements being compared or provide a brief description of the comparison being made.\n\nLIMITATIONS:\nWhile the authors outline potential future research directions, they do not explicitly address the limitations of their proposed method and approach. It would be beneficial if the authors could include a brief discussion of the technical limitations, such as the assumptions made in their approach. Additionally, providing insights into potential negative impacts when applying the method in real-world applications would further enhance the paper's practical relevance. However, it should be noted that as this paper primarily focuses on theoretical aspects, addressing these limitations in detail may not be deemed critical.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-04"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper utilizes conformal inference techniques to address uncertainty quantification in the matrix completion problem. The authors present a novel method for constructing prediction sets for the missing entries estimated by any given matrix completion algorithm, employing a simple data hold-out strategy. The underlying assumption is that the data matrix $M$ is fixed, while randomness arises from the missingness of the matrix entries.\n\nSpecifically, each index is assumed to have a distinct probability of being observed, independent of the others. The authors demonstrate that when the missingness probabilities are uniform across all entries, the data become exchangeable, allowing for the direct application of standard conformal inference methods.\n\nIn cases with heterogeneous missing probabilities, the problem becomes more intricate and aligns with the conformal inference under the covariate-shift framework introduced by Tibshirani et al. (2019). To tackle this scenario, the authors propose applying the method developed by Tibshirani et al. (2019) and employ data-driven estimates of the missingness probabilities derived from parametric models.\n\nThe validity of the proposed method is established through both theoretical arguments leveraging existing results and empirical evaluation using synthetic and real data experiments. These experiments effectively demonstrate the efficacy and practical applicability of the proposed approach.\n\nSTRENGTHS:\n1)    This paper is both interesting and original, as it contributes to connection between conformal inference and matrix completion. The authors successfully present a reasonable missingness model and introduce a principled framework to apply weighted split conformal methods.\n\n2)    The practical usefulness and potential impact of this paper are noteworthy. Until recently, uncertainty estimation in matrix completion was relatively unexplored, and prior methods were limited due to their heavy reliance on assumptions that often do not hold in practice. Therefore, the assumption-lean approach presented in this paper holds substantial promise and has the potential to significantly impact related fields.\n\n3)    The paper is well-written, and the proposed method is clearly and effectively explained.\n\nWEAKNESSES:\n1)    Technical novelty and originality of theoretical contributions. While this paper seems to rely heavily on the results of Tibshirani et al. (2019) and Barber et al. (2022), the relationship between these works could be explained more clearly. In particular, it would be beneficial to clarify which aspects of the proofs are novel and which can be considered as special instances of prior work. \n\n2)    Thoroughness of the numerical experiments. While the proposed method appears promising in theory, the numerical experiments lack convincing evidence. For example, in the four settings depicted in Figure 1, it would be desirable to observe under-coverage in $\\texttt{als}$ when the signal-to-noise ratio is low or when the incoherence condition is violated, even with an oracle rank. However, both $\\texttt{cmc}$ and $\\texttt{als}$ exhibit similar performance when the rank is correctly chosen, with the advantages of $\\texttt{cmc}$ primarily stemming from tuning the hypothesized rank. Further investigation and more comprehensive experiments would be useful to establish the illustrate of the proposed method.\n\n3)    Realism of real-data experiments. The real data application may not provide significantly more information than a synthetic experiment since the authors begin with the full matrix $M$ and manually sample missing entries using a logistic missingness model. However, it remains unclear whether this logistic model accurately represents practical scenarios. Designing experiments based on more realistic missingness patterns would enhance the informativeness of the results. Additionally, it is expected that the benchmark $\\texttt{als}$ would exhibit under-coverage in some heterogeneous settings, but the authors did not compare it to $\\texttt{als}$. By neglecting this comparison, the practical relevance of the work is not made as clear as it could be.\n\n4)    Limited empirical evaluation. The authors only consider the ``average coverage rate\" as defined in Equation (3), which provides a relatively weak coverage guarantee. It would be helpful if the numerical experiments also included appropriate conditional coverage metrics to provide a more comprehensive evaluation.\n\nQUESTIONS:\n1)    Usefulness of the one-shot shortcut. The paper introduces a one-shot weighted conformal approach to reduce computational costs. However, it is important to consider potential issues with this relaxation. Firstly, the presence of extremely small probabilities $p_{ij}$ for some test points may lead to excessively large odd ratios $h_{ij}$, resulting in overly conservative predictions that may not be useful for other test points. Secondly, it is not entirely clear why the proposed algorithm without the one-shot relaxation would be prohibitively computationally expensive. Since the weight $w_{ij}$ has a simple form, it should not significantly increase evaluation time in practice. \n\n2)    Improvement of technical notation. Some technical details require clarification. For example, in line 157 on page 5, the odd ratio is defined as $(1-p_{ij})/p_{ij}$, which poses a problem when $p_{ij}=0$ for some index $(i,j)$. Additionally, in the Supplement Material, lines 464-467, the variables $\\mathbf{Z}$ and $\\mathbf{W}$ are not explicitly defined, making it more difficult to verify the proof.\n\n3)    Missing citation. In line 233-237, the benchmark prediction sets are mentioned, which are believed to follow from the asymptotic results in Chen et al. (2019). However, the citation for this reference is missing. Including the appropriate citation will provide proper attribution and give readers an opportunity to explore the referenced work.\n\n4)    Consideration of alternative methods. There are other matrix completion algorithms available that provide uncertainty quantification, such as matrix completion with Gaussian Copula (Zhao et al, 2020). It would be beneficial for the authors to implement additional benchmarks using these alternative methods to strengthen the validity and robustness of the data experiments.\n\nLIMITATIONS:\nThe main limitations of this paper concern its technical novelty compared to prior work on conformal inference and the depth of its empirical evaluations.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-05"}

comments: SUMMARY:
This paper presents a distribution-free method for constructing prediction intervals in the matrix completion problem, where randomness only arises from the sampling of observed entries. The approach utilizes weighted conformal prediction and establishes a lower bound on the probability of each unobserved entry being included in the prediction interval. The lower bound relies on the estimation error of sampling probabilities, which can be negligible if accurately estimated.

STRENGTHS:
* Overall, the paper is clearly written.
* The utilization of weighted conformal prediction in the matrix completion problem is quite novel.
* The study shows that the resulting prediction interval performs well with well-estimated sampling probabilities.

WEAKNESSES:
* Incorrect estimation of sampling probabilities may significantly degrade the quality of the prediction interval.

QUESTIONS:
* Are there any experimental results when the sampling probabilities are completely misspecified, e.g., when the heterogeneous missingness is misspecified as uniform sampling?
* I'm curious about the impact of the estimated values of $\hat{s_{ij}}$'s. Does misspecification of these values have a substantial negative impact on the results? Are there any empirical findings regarding this?
* Is there a rationale for the estimation of theta at Line 237? If there is a reference concerning this, could you provide it?
* Since the constructed prediction interval is for M, rather than M*, wouldn't it be appropriate to employ an approach for approximately low-rank matrices in experiments? With an approximately low-rank matrix completion approach, how does the estimation of theta change?
* What does Line 277 mean?

LIMITATIONS:
Yes.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 2
DATE: 2023-07-06

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes to use conformal prediction for uncertainty quantification of matrix completion. The proposed conformalized matrix completion offers provable predictive coverage regardless of the accuracy of the low-rank model. Empirical results on simulated and real data demonstrate that cmc is robust to model misspecification while matching the performance of existing model-based methods when the model is correct.\n\nSTRENGTHS:\nStrength 1. This paper studies conformal prediction for matric completion with interesting theoretical and algorithmatical findings. \n1.1） Through conformal prediction, this paper proposes distribution-free confidence intervals for the completed entries in matrix completion. Benefiting from the use of conformal prediction, the validity is free of any assumption on the underlying matrixs and holds regardless of the choice of estimation algorithms. By proving the (weighted) exchangeabiity of unobserved and observed units when they are (non-uniformly) sampled without replacement from a finite population.  \n1.2） A provable lower bound for the coverage rate is provided when the sampling mechanism is unknown. \n1.3） A one-shot conformalized matrix completion approach is proposed for higher computational efficiency.\n\nStrength 2. Experiments on both synthetic and real data suggest the effectivenss of the proposal.\n\nWEAKNESSES:\nWeakness 1: One weakness of the proposed uncertainty quantification method for model calibration is that the probability bounds it provides may not be very precise for specific observation models and optimization algorithms. While the method is advantageous in that it is not limited to any particular model or algorithm, there is a possibility that the uncertainty estimates can be too broad for certain scenarios. This potential issue should be acknowledged and addressed in the research.\n\nWeakness 2: Another weakness is that the proposed approach does not assume any specific structure on the underlying matrix. As a result, it may not effectively utilize any existing structure in the matrix. To overcome this limitation, the proposed method can be extended to vector completion problems, which could exploit the matrix structure more efficiently. The authors should consider discussing this point and its potential implications.\n\nWeakness 3: The paper could benefit from improved clarity in its explanations. It would be more reader-friendly if the authors provided more intuitive explanations for the newly introduced quantities, such as the quantity labeled as (6) in the paper. Enhancing clarity in the presentation of the research would greatly improve its accessibility to readers.\n\nQUESTIONS:\nPlease refer to the \"Weaknesses\".\n\nLIMITATIONS:\nThe limitations have not been adequately addressed in the main text.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 2, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-07"}