SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Yu Gui Rina Foygel Barber Cong Ma
Department of Statistics, University of Chicago {yugui, rina, congm}@uchicago.edu
Matrix completion aims to estimate missing entries in a data matrix, using the assumption of a low-complexity structure (e.g., low rank) so that imputation is possible. While many effective estimation algorithms exist in the literature, uncertainty quantification for this problem has proved to be challenging, and existing methods are extremely sensitive to model misspecification. In this work, we propose a distribution-free method for predictive inference in the matrix completion problem. Our method adapts the framework of conformal prediction, which provides confidence intervals with guaranteed distribution-free validity in the setting of regression, to the problem of matrix completion. Our resulting method, conformalized matrix completion (cmc), offers provable predictive coverage regardless of the accuracy of the low-rank model. Empirical results on simulated and real data demonstrate that cmc is robust to model misspecification while matching the performance of existing model-based methods when the model is correct.
Matrix completion, the task of filling in missing entries in a large data matrix, has a wide range of applications such as gene-disease association analysis in bioinformatics (Natarajan and Dhillon, 2014), collaborative filtering in recommender systems (Rennie and Srebro, 2005), and panel data prediction and inference in econometrics (Amjad et al., 2018; Bai and Ng, 2021; Athey et al., 2021). If the underlying signal is assumed to be low-rank, a range of estimation algorithms have been proposed in the literature, including approaches based on convex relaxations of rank (Candes and Plan, 2010; Candès and Tao, 2010; Koltchinskii et al., 2011; Foygel and Srebro, 2011; Negahban and Wainwright, 2012; Yang and Ma, 2022) and nonconvex optimization over the space of low-rank matrices (Keshavan et al., 2010; Jain et al., 2013; Sun and Luo, 2016; Ma et al., 2020).
While the problem of estimation has been explored through many different approaches in the literature, the question of uncertainty quantity for matrix completion remains challenging and under-explored. Cai et al. (2016) and Carpentier et al. (2018) construct confidence intervals for the missing entries using order-wise concentration inequalities, which could be extremely loose. Chen et al. (2019) proposes a de-biased estimator for matrix completion and characterizes its asymptotic distribution, which then yields entrywise confidence intervals are then derived for the underlying incoherent matrix with i.i.d. Gaussian noise. See also Xia and Yuan (2021); Farias et al. (2022) for related approaches.
The effectiveness of the aforementioned uncertainty quantification methods rely heavily on the exact low-rank structure as well as assumptions on the tail of the noise. In practice, the low-rank assumption can only be met approximately, and heavy-tailed noise is quite common in areas such as macroeconomics and genomics (Nair et al., 2022). Without exact model specifications, the solution to the matrix completion algorithm is hard to interpret and the asymptotic distribution of the obtained estimator is no longer valid. The dependence of model-based approaches on exact model assumptions motivates us to answer the question: can we construct valid and short confidence intervals for the
missing entries that are free of both model assumptions and of constraints on which estimation algorithms we may use?
1.1 Conformal prediction for distribution-free uncertainty quantification
Our proposed method is built on the idea of conformal prediction. The conformal prediction framework, developed by Vovk et al. (2005) and Shafer and Vovk (2008), has drawn significant attention in recent years in that it enables the construction of distribution-free confidence intervals that are valid with exchangeable data from any underlying distribution and with any "black-box" algorithm. To reduce computation in full conformal prediction, variants exist based on data splitting (Vovk et al., 2005; Lei et al., 2018), leave-one-out (Barber et al., 2021), cross-validation (Vovk, 2015; Barber et al., 2021), etc.
When distribution shift is present, with covariate shift as a special case, the exchangeability is violated, and Tibshirani et al. (2019) proposes a more general procedure called weighted conformal prediction that guarantees the validity under the weighted exchangeability of data. Barber et al. (2022) further relaxes the exchangeability assumption on the data and characterizes the robustness of a generalized weighted approach via an upper bound for the coverage gap. Other related works (Lei and Candès, 2020; Candès et al., 2021; Jin et al., 2023) adapt the conformal framework to applications in causal inference, survival analysis, etc., and study the robustness of weighted conformal prediction with estimated weights.
1.2 Main contributions
In this paper, we adapt the framework of conformal prediction for the problem of matrix completion, and make the following contributions:
1. We construct distribution-free confidence intervals for the missing entries in matrix completion via conformal prediction. The validity is free of any assumption on the underlying matrix and holds regardless of the choice of estimation algorithms practitioners use. To achieve this, we prove the (weighted) exchangeability of unobserved and observed units when they are sampled (possibly nonuniformly) without replacement from a finite population.
2. When the sampling mechanism is unknown, we develop a provable lower bound for the coverage rate which degrades gracefully as the estimation error of the sampling probability increases.
3. In addition, to improve computational efficiency when faced with a large number of missing entries, we propose a one-shot conformalized matrix completion approach that only computes the weighted quantile once for all missing entries.
In this section, we formulate the matrix completion problem and contrast our distribution-free setting with the model-based settings that are more common in the existing literature. For a partially-observed  $d_{1} \times d_{2}$  matrix, the subset  $S \subseteq [d_1] \times [d_2]$  denotes the set of indices where data is observed—that is, our observations consist of  $(M_{ij})_{(i,j) \in S}$ . In much of the matrix completion literature, it is common to assume a signal-plus-noise model for the observations,
$$
M _ {i j} = M _ {i j} ^ {*} + E _ {i j},
$$
where  $\mathbf{M}^{*} = (M_{ij}^{*})_{(i,j)\in [d_1]\times [d_2]}$  is the "true" underlying signal while  $(E_{ij})_{(i,j)\in S}$  denotes the (typically zero-mean) noise. Frequently, it is assumed that  $\mathbf{M}^{*}$  is low-rank, or follows some other latent low-dimensional structure, so that recovering  $\mathbf{M}^{*}$  from the observed data is an identifiable problem. In works of this type, the goal is to construct an estimator  $\widehat{\mathbf{M}}$  that accurately recovers the underlying  $\mathbf{M}^{*}$ . In contrast, in a more model-free setting, we may no longer wish to hypothesize a signal-plus-noise model, and can instead assume that we are observing a random subset of entries of a deterministic matrix  $\mathbf{M}$ ; in this setting, estimating  $\mathbf{M}$  itself becomes the goal.
For both frameworks, many existing results focus on the problem of estimation (either of  $\mathbf{M}^*$  or of  $\mathbf{M}$ ), with results establishing bounds on estimation errors under various assumptions. For instance, see Candes and Plan (2010); Candès and Tao (2010); Negahban and Wainwright (2012); Chen et al. (2020) for results on estimating  $\mathbf{M}^*$  (under stronger conditions, e.g., a low-rank and incoherent signal,
and zero-mean sub-Gaussian noise), or Srebro and Shraibman (2005); Foygel and Srebro (2011) for results on estimating  $\mathbf{M}$  (under milder conditions); note that in the latter case, it is common to instead refer to predicting the entries of  $\mathbf{M}$ , e.g., temperature or stock market returns, since we often think of these as noisy observations of some underlying signal. On the other hand, relatively little is known about the problem of uncertainty quantification for these estimates: given some estimator  $\widehat{\mathbf{M}}$ , can we produce a confidence interval around each  $\widehat{M}_{ij}$  that is (asymptotically) guaranteed to contain the target with some minimum probability?
The results of Chen et al. (2019) address this question under strong assumptions, namely, a low-rank and incoherent signal  $\mathbf{M}^*$ , plus i.i.d. Gaussian noise  $(E_{ij})_{(i,j)\in [d_1]\times [d_2]}$ . Zhao and Udell (2020) proposes another inferential procedure under the Gaussian copula assumption for the data. In this work, we provide a complementary answer that instead addresses the model-free setting: without relying on assumptions, we aim to produce a provably valid confidence interval for the entries  $M_{ij}$  of  $\mathbf{M}$  (not of  $\mathbf{M}^*$ , because without assuming a low-rank model, there is no meaningful notion of an "underlying signal").
To do so, we will treat the matrix  $\mathbf{M}$  as deterministic, while the randomness arises purely from the random subset of entries that are observed. More specifically, we assume
$$
Z _ {i j} = \mathbb {1} \left\{(i, j) \text {i s o b s e r v e d} \right\} \sim \operatorname {B e r n} \left(p _ {i j}\right), \quad \text {i n d e p e n d e n t l y f o r a l l} (i, j) \in [ d _ {1} ] \times [ d _ {2} ], \tag {1}
$$
and let  $\mathcal{S} = \{(i,j)\in [d_1]\times [d_2]:Z_{ij} = 1\}$  be the subset of observed locations. We consider the setting where the sample probabilities  $p_{ij}$ 's are nonzero. After observing  $\mathbf{M}_{\mathcal{S}} = (M_{ij})_{(i,j)\in \mathcal{S}}$  our goal is to provide confidence intervals  $\{\widehat{C} (i,j)\}_{(i,j)\in \mathcal{S}^c}$  for all unobserved entries, with  $1 - \alpha$  coverage over the unobserved portion of the matrix—that is, we would like our confidence intervals to satisfy
$$
\mathbb {E} \left[ \operatorname {A v g C o v} (\widehat {C}; \mathbf {M}, \mathcal {S}) \right] \geq 1 - \alpha , \tag {2}
$$
$$
\operatorname {A v g C o v} (\widehat {C}; \mathbf {M}, \mathcal {S}) = \frac {1}{| \mathcal {S} ^ {c} |} \sum_ {(i, j) \in \mathcal {S} ^ {c}} \mathbb {1} \left\{M _ {i j} \in \widehat {C} (i, j) \right\} \tag {3}
$$
measures the "average coverage rate" of the confidence intervals. The goal of this work is to provide an algorithm for constructing confidence intervals  $\{\widehat{C}(i,j)\}_{(i,j) \in \mathcal{S}^c}$  based on the observed entries  $\mathbf{M}_S$ , satisfying (2) with no assumptions on  $\mathbf{M}$ . In particular, given the strong success of various matrix completion algorithms for the problem of estimation, we would like to be able to provide uncertainty quantification around any base estimator—i.e., given any algorithm for producing an estimate  $\widehat{\mathbf{M}}$ , our method constructs confidence intervals  $\{\widehat{C}(i,j)\}_{(i,j) \in \mathcal{S}^c}$  around this initial estimate.
Notation. For a matrix  $\mathbf{A} = (A_{ij})\in \mathbb{R}^{m_1\times m_2}$ , we define  $\| \mathbf{A}\|_{\infty}\coloneqq \max_{(i,j)\in [m_1]\times [m_2]}|A_{ij}|$ , and  $\| \mathbf{A}\| _1\coloneqq \sum_{(i,j)\in [m_1]\times [m_2]}|A_{ij}|$ . We use  $\mathbf{A}_{i,\cdot}$  and  $\mathbf{A}_{\cdot ,j}$  to refer to the  $i$ th row and  $j$ th column of  $\mathbf{A}$ , respectively. For any  $t\in \mathbb{R}$ ,  $\delta_t$  denotes the distribution given by a point mass at  $t$ .
3 Conformalized matrix completion
We next present our method, conformalized matrix completion (cmc). Our procedure adapts the split conformal prediction method (Vovk et al., 2005) to the problem at hand. As is standard in the conformal prediction framework, the goal of cmc is to provide uncertainty quantification (i.e., confidence intervals) around the output of any existing estimation procedure that the analyst chooses to use—this is a core strength of the conformal methodology, as it allows the analyst to use state-of-the-art estimation procedures without compromising on the validity of inference. At a high level, the cmc procedure will output a confidence interval of the form
$$
\widehat {C} (i, j) = \widehat {M} _ {i j} \pm \widehat {q} \cdot \widehat {s} _ {i j}
$$
for the matrix value  $M_{ij}$  at each unobserved entry  $(i,j)$ . Here, after splitting the observed entries into a training set and a calibration set, the training set is used to produce  $\widehat{M}_{ij}$  (a point estimate for the
To handle the degenerate case where  $S = [d_1] \times [d_2]$  (i.e., we happen to have observed the entire matrix) and thus  $S^c = \emptyset$ , we simply define  $\operatorname{AvgCov}(\widehat{C}; \mathbf{M}, S) \equiv 1$ .
target value  $M_{ij}$ ) and  $\widehat{s}_{ij}$  (a scaling parameter that estimates our uncertainty for this point estimate); then, the calibration set is used to tune the scalar  $\widehat{q}$ , to adjust the width of these confidence intervals and ensure coverage at the desired level  $1 - \alpha$ .<sup>2</sup>
Algorithm 1 Conformalized matrix completion (cmc)
1: Input: target coverage level  $1 - \alpha$ ; data splitting proportion  $q \in (0,1)$ ; observed entries  $\mathbf{M}_{\mathcal{S}}$ .
2: Split the data: draw  $W_{ij} \stackrel{\mathrm{i.i.d.}}{\sim} \operatorname{Bern}(q)$ , and define training and calibration sets
$$
\mathcal {S} _ {\mathrm {t r}} = \{(i, j) \in \mathcal {S}: W _ {i j} = 1 \}, \quad \text {a n d} \quad \mathcal {S} _ {\mathrm {c a l}} = \{(i, j) \in \mathcal {S}: W _ {i j} = 0 \}.
$$
3: Using the training data  $\mathbf{M}_{S_{\mathrm{tr}}}$  indexed by  $S_{\mathrm{tr}} \subseteq [d_1] \times [d_2]$ , compute:
- An initial estimate  $\widehat{\mathbf{M}}$  using any matrix completion algorithm (with  $\widehat{M}_{ij}$  estimating the target  $M_{ij}$ );
- Optionally, a local uncertainty estimate  $\widehat{\mathbf{s}}$  (with  $\widehat{s}_{ij}$  estimating our relative uncertainty in the estimate  $\widehat{M}_{ij}$ ), or otherwise set  $\widehat{s}_{ij} \equiv 1$ ;
- An estimate  $\widehat{\mathbf{P}}$  of the observation probabilities (with  $\widehat{p}_{ij}$  estimating  $p_{ij}$ , the probability of entry  $(i,j)$  being observed).
4: Compute normalized residuals on the calibration set,  $R_{ij} = \frac{|M_{ij} - \widehat{M}_{ij}|}{\widehat{s}_{ij}}$ ,  $(i,j) \in S_{\mathrm{cal}}$ .
5: Compute estimated odds ratios for the calibration set and test set,  $\widehat{h}_{ij} = \frac{1 - \widehat{p}_{ij}}{\widehat{p}_{ij}}$ ,  $(i,j) \in S_{\mathrm{cal}} \cup S^c$ , and then compute weights for the calibration set and test point,
$$
\widehat {w} _ {i j} = \frac {\widehat {h} _ {i j}}{\sum_ {\left(i ^ {\prime} , j ^ {\prime}\right) \in \mathcal {S} _ {\mathrm {c a l}}} \widehat {h} _ {i ^ {\prime} j ^ {\prime}} + \max  _ {\left(i ^ {\prime} , j ^ {\prime}\right) \in \mathcal {S} ^ {c}} \widehat {h} _ {i ^ {\prime} j ^ {\prime}}}, (i, j) \in \mathcal {S} _ {\mathrm {c a l}}, \widehat {w} _ {\text {t e s t}} = \frac {\max  _ {(i , j) \in \mathcal {S} ^ {c}} \widehat {h} _ {i j}}{\sum_ {\left(i ^ {\prime} , j ^ {\prime}\right) \in \mathcal {S} _ {\mathrm {c a l}}} \widehat {h} _ {i ^ {\prime} j ^ {\prime}} + \max  _ {\left(i ^ {\prime} , j ^ {\prime}\right) \in \mathcal {S} ^ {c}} \widehat {h _ {i ^ {\prime} j ^ {\prime}}}}. \tag {4}
$$
6: Compute threshold  $\widehat{q} = \mathrm{Quantile}_{1 - \alpha}\left(\sum_{(i,j)\in S_{\mathrm{cal}}}\widehat{w}_{ij}\cdot \delta_{R_{ij}} + \widehat{w}_{\mathrm{test}}\cdot \delta_{+\infty}\right)$
7: Output: confidence intervals  $\widehat{C}(i,j) = \widehat{M}_{ij} \pm \widehat{q} \cdot \widehat{s}_{ij}$  for each unobserved entry  $(i,j) \in S^c$ .
3.1 Exchangeability and weighted exchangeability
We split the set of observed indices  $S$  into a training and a calibration set  $S = S_{\mathrm{tr}} \cup S_{\mathrm{cal}}$  as is shown in Algorithm 1. We should notice that the sampling without replacement may introduce implicit dependence as well as a distribution shift from the i.i.d. sampling. Thus the two sets are dependent, which is different from the split conformal methods in regression problems. Before we present the coverage guarantees for our method, we first examine the role of (weighted) exchangeability in this method, to build intuition for how the method is constructed.
3.1.1 Intuition: the uniform sampling case
First, for intuition, consider the simple case where the entries are sampled with constant probability,  $p_{ij} \equiv p$ . If this is the case, then the set of calibration samples and the test point are exchangeable—that is, for  $(i_*, j_*)$  denoting the test point that is drawn uniformly from  $S^c$ , if we are told that the combined set  $S_{\mathrm{cal}} \cup \{(i_*, j_*)\}$  is equal to  $\{(i_1, j_1), \ldots, (i_{n_{\mathrm{cal}} + 1}, j_{n_{\mathrm{cal}} + 1})\}$  in no particular order (where  $n_{\mathrm{cal}} = |\mathcal{S}_{\mathrm{cal}}|$ ), then the test point location  $(i_*, j_*)$  is equally likely to be at any one of these  $n_{\mathrm{cal}} + 1$  locations. Consequently, by exchangeability, the calibration residuals  $\{R_{ij} : (i, j) \in S_{\mathrm{cal}}\}$ , defined in Algorithm 1 above, are exchangeable with the test residual  $R_{i_j * j_*} = |M_{i_j * j_*} - \widehat{M}_{i_j * j_*}| / \widehat{s}_{i_j * j_*}$ ; as a result, we can construct our confidence interval  $\widehat{C}(i_*, j_*)$  with  $\widehat{q}$  determined by the  $(1 - \alpha)$ -quantile of the calibration residuals  $\{R_{ij} : (i, j) \in S_{\mathrm{cal}}\}$ .
2More generally, cmc can be implemented with any choice of a nonconformity score that determines the shape of the resulting confidence intervals—and indeed, this generalization allows for categorical rather than real-valued matrix entries  $M_{ij}$ . We will address this extension in the Supplementary Material, as well as a version of cmc based on full conformal rather than split conformal, in order to avoid data splitting.
Indeed, this is exactly what cmc does in this case: if we are aware that sampling is uniform, it would naturally follow that we estimate  $\widehat{p}_{ij} \equiv \widehat{p}_0$  by some (potentially data-dependent) scalar  $\widehat{p}_0$ ; consequently, all the weights are given by  $\widehat{w}_{ij} \equiv \frac{1}{n_{\mathrm{cal}} + 1}$  and  $\widehat{w}_{\mathrm{test}} = \frac{1}{n_{\mathrm{cal}} + 1}$ , and thus  $\widehat{q}$  is the (unweighted)  $(1 - \alpha)$ -quantile of the calibration residuals (with a small correction term, i.e., the term  $+\widehat{w}_{\mathrm{test}} \cdot \delta_{+\infty}$  appearing in the definition of  $\widehat{q}$ , to ensure the correct probability of coverage at finite sample sizes).
3.1.2 Weighted exchangeability for the matrix completion problem
Next, we move to the general case, where now the sampling may be highly nonuniform. First suppose the sampling probabilities  $p_{ij}$  are known, and suppose again that we are told that the combined set  $S_{\mathrm{cal}} \cup \{(i_{*}, j_{*})\}$  is equal to  $\{(i_1, j_1), \ldots, (i_{n_{\mathrm{cal}} + 1}, j_{n_{\mathrm{cal}} + 1})\}$  in no particular order. In this case, the test point  $(i_{*}, j_{*})$  is not equally likely to be at any one of these  $n_{\mathrm{cal}} + 1$  locations. Instead, we have the following result:
Lemma 3.1. Following the notation defined above, if  $(i_{*},j_{*})\mid S\sim \mathrm{Unif}(S^{c})$  , it holds that
$$
\mathbb {P} \left\{\left(i _ {*}, j _ {*}\right) = \left(i _ {k}, j _ {k}\right) \mid S _ {\mathrm {c a l}} \cup \left\{\left(i _ {*}, j _ {*}\right) \right\} = \left\{\left(i _ {1}, j _ {1}\right), \dots , \left(i _ {n _ {\mathrm {c a l}} + 1}, j _ {n _ {\mathrm {c a l}} + 1}\right) \right\}, S _ {\mathrm {t r}} \right\} = w _ {i _ {k} j _ {k}},
$$
where we define the weights  $w_{i_kj_k} = \frac{h_{i_kj_k}}{\sum_{k' = 1}^{n_{\mathrm{cal}} + 1}h_{i_{k'}j_{k'}}}$  for odds ratios given by  $h_{ij} = \frac{1 - p_{ij}}{p_{ij}}$ .
Consequently, while the test point's residual  $R_{i_*j_*} = |M_{i_*j_*} - \widehat{M}_{i_*j_*}| / \widehat{s}_{i_*j_*}$  is not exchangeable with the calibration set residuals  $\{R_{ij}:(i,j)\in S_{\mathrm{cal}}\}$ , the framework of weighted exchangeability (Tibshirani et al., 2019) tells us that we can view  $R_{i_*j_*}$  as a draw from the weighted distribution that places weight  $w_{i_kj_k}$  on each residual  $R_{i_kj_k}$  and in particular,
$$
\mathbb {P} \left\{M _ {i _ {*} j _ {*}} \in \widehat {M} _ {i _ {*} j _ {*}} \pm q _ {i _ {*}, j _ {*}} ^ {*} \cdot \widehat {s} _ {i _ {*} j _ {*}} \right\} = \mathbb {P} \left\{R _ {i _ {*} j _ {*}} \leq q _ {i _ {*}, j _ {*}} ^ {*} \right\} \geq 1 - \alpha ,
$$
where  $q_{i_{*},j_{*}}^{*} = \mathrm{Quantile}_{1 - \alpha}\big(\sum_{k = 1}^{n_{\mathrm{cal}} + 1}w_{i_kj_k}\cdot \delta_{R_{i_kj_k}}\big)$ . Noting that this threshold  $q_{i_{*},j_{*}}^{*}$  may depend on the test location  $(i_{*},j_{*})$ , to accelerate the computation of prediction sets for all missing entries, we instead propose a one-shot weighted conformal approach by upper bounding the threshold uniformly over all test points:  $q_{i_{*},j_{*}}^{*}\leq q^{*}\coloneqq \mathrm{Quantile}_{1 - \alpha}\Big(\sum_{(i,j)\in \mathcal{S}_{\mathrm{cal}}}w_{ij}^{*}\cdot \delta_{R_{ij}} + w_{\mathrm{test}}^{*}\cdot \delta_{+\infty}\Big)$  for  $(i,j)\in S_{\mathrm{cal}}$ ,
$$
w _ {i j} ^ {*} = \frac {h _ {i j}}{\sum_ {\left(i ^ {\prime} , j ^ {\prime}\right) \in \mathcal {S} _ {\mathrm {c a l}}} h _ {i ^ {\prime} j ^ {\prime}} + \max  _ {\left(i ^ {\prime} , j ^ {\prime}\right) \in \mathcal {S} ^ {c}} h _ {i ^ {\prime} j ^ {\prime}}}, w _ {\text {t e s t}} ^ {*} = \frac {\underset {(i , j) \in \mathcal {S} ^ {c}} {\max } h _ {i j}}{\sum_ {\left(i ^ {\prime} , j ^ {\prime}\right) \in \mathcal {S} _ {\mathrm {c a l}}} h _ {i ^ {\prime} j ^ {\prime}} + \underset {\left(i ^ {\prime} , j ^ {\prime}\right) \in \mathcal {S} ^ {c}} {\max } h _ {i ^ {\prime} j ^ {\prime}}}. \tag {5}
$$
Now the threshold  $q^{*}$  no longer depends on the location  $(i_{*}, j_{*})$  of the test point<sup>3</sup>. If the true probabilities  $p_{ij}$  were known, then, we could define "oracle" conformal confidence intervals
$$
\widehat {C} ^ {*} (i, j) = \widehat {M} _ {i j} \pm q ^ {*} \cdot \widehat {s} _ {i j}, \quad (i, j) \in \mathcal {S} ^ {c}.
$$
As we will see in the proofs below, weighted exchangeability ensures that, if we do have oracle knowledge of the  $p_{ij}$ 's, then these confidence intervals satisfy the goal (2) of  $1 - \alpha$  average coverage. Since the  $p_{ij}$ 's are not known in general, our algorithm simply replaces the oracle weights  $\boldsymbol{w}^*$  in (5) with their estimates  $\widehat{\boldsymbol{w}}$  defined in (4), using  $\widehat{p}_{ij}$  as a plug-in estimate of the unknown  $p_{ij}$ .
3.2 Theoretical guarantee
In the setting where the  $p_{ij}$ 's are not known but instead are estimated, our cmc procedure simply repeats the procedure described above but with weights  $\widehat{w}$  calculated using  $\widehat{p}_{ij}$ 's in place of  $p_{ij}$ 's. Our theoretical results therefore need to account for the errors in our estimates  $\widehat{p}_{ij}$ , to quantify the coverage gap of conformal prediction with the presence of these estimation errors. To do this, we define an estimation gap term
$$
\Delta = \frac {1}{2} \sum_ {(i, j) \in \mathcal {S} _ {\mathrm {c a l}} \cup \{(i _ {*}, j _ {*}) \}} \left| \frac {\widehat {h} _ {i j}}{\sum_ {(i ^ {\prime} , j ^ {\prime}) \in \mathcal {S} _ {\mathrm {c a l}} \cup \{(i _ {*} , j _ {*}) \}} \widehat {h} _ {i ^ {\prime} j ^ {\prime}}} - \frac {h _ {i j}}{\sum_ {(i ^ {\prime} , j ^ {\prime}) \in \mathcal {S} _ {\mathrm {c a l}} \cup \{(i _ {*} , j _ {*}) \}} h _ {i ^ {\prime} j ^ {\prime}}} \right|, \tag {6}
$$
3 Comparison between the one-shot weighted approach and the exact weighted approach (Algorithm 3) is conducted in Section B.1