SlowGuess/NeurIPS_2023_OCR huggingface dataset:
The estimation error in  $\widehat{p}_{ij}$  can be visualized from the following heatmaps comparing  $\mathbf{P}$  and  $\widehat{\mathbf{P}}$ . Here the entries are sorted by the order of  $p_{ij}$  for each row and each column.
(a) Setting 1: Large sample size + Gaussian noise
(b) Setting 2: Small sample size+Gaussian noise
(c) Setting 3: Large sample size + heavy-tailed noise
(d) Setting 4: Violation of incoherence
Figure 6: Comparison between conformalized and model-based matrix completion approaches.
(b)  $k^{*} = 1\colon d_{1} = d_{2} = 2000.$
(a)  $k^{*} = 1\colon d_{1} = d_{2} = 500$
(c)  $k^{*} = 5\colon d_{1} = d_{2} = 500$
Figure 7: Heatmaps for  $\mathbf{P}$  and  $\widehat{\mathbf{P}}$ .
(d)  $k^{*} = 5$ :  $d_{1} = d_{2} = 2000$
B.5 Additional results for heterogeneous missingness
In Figure 8, we present the results for synthetic simulation with the convex relaxation cvx as the base algorithm, where we denote cmc-cvx and cmc\*-cvx as the conformalized matrix completion method with estimated weights and true weights, respectively. Three settings with heterogeneous missingness are the same as Figure 3.
B.6 Misspecified sampling probability
We consider the following four settings:
(a) The underlying missingness follows the rank-one model where  $p_{ij} = a_i b_j$ , both  $a_i$  and  $b_j$  are generated i.i.d. from Unif(0.2, 1). The noise is adversarial. But we estimate  $p_{ij}$  via the one-bit matrix completion based on the logistic model (working model with hypothesized rank  $k = 5$ ).
(b) The underlying missingness follows the logistic model as in Example 3.3.2 with  $k^* = 5$  and we adopt the adversarial noise. But  $p_{ij}$  is estimated under the assumption of uniform sampling (working model), i.e.  $\widehat{p}_{ij} = \widehat{p}$ .
(c) Adversarial heterogeneous noise:  $k^{*} = 1$ .
(e) Random heterogeneous noise:  $k^{*} = 1$ .
Figure 8: Comparison under heterogenous missingness.
(a) Estimating rank-1 model with logistic model (adversarial noise).
(c) Estimating rank-1 model with logistic model (random noise).
(b) Estimating logistic model with uniform sampling (adversarial noise).
(d) Estimating logistic model with uniform sampling (random noise).
(c) Same as (a) except that we use the random heterogeneous noise.
(d) Same as (b) except that we use the random heterogeneous noise.
From the results, we can see that when the noise is generated following the random heterogeneous model, where the values of entries are independent of the sampling probabilities, the misspecification of the sampling model only slightly affects the coverage. Moreover, when the noise is generated in the adversarial way, where the values of entries depend on  $p_{ij}$ 's, we can see that the coverage with a misspecified sampling model is lower than the target level, but is above 0.85 in practice, which depends on the divergence between the true and the working sampling models.
B.7 Additional results for the sales dataset
Denote  $\mathbf{M}$  the underlying matrix in the sales dataset. In Figure 10a, we plot singular values of  $\mathbf{M}$  and top-5 singular values contain a large proportion of the information. In Figure 10b, we plot the histogram of entries  $M_{ij}$ 's of the underlying matrix, and the sales dataset has the range from 0 to over 20 thousand with a heavy tail.
(a) Singular values of the underlying matrix  $\mathbf{M}$ .
(b) Histogram of entries in  $\mathbf{M}$ .
Figure 10: Descriptive plots for the underlying matrix  $\mathbf{M}$ .
B.7.1 Comparison under misspecified missingness
We consider the sales dataset where the missingness occurs in the way that: for weekdays, each entry is observed with  $p = 0.8$ . On weekends, as the stores are likely to be operated by less experienced interns or are less frequent to report the sales data, each entry is observed with a lower probability, e.g.  $0.8/3$ . Moreover, as there could be a subgroup of stores that are less frequent in sales reporting, 200 stores are randomly sampled, for which  $p = 0.8/3$ . We use the logistic model with  $k = 5$  to estimate  $p_{ij}$ . Comparison between a1s and cmc-als is shown in Figure 11.
B.7.2 Results with convex optimization
In Figure 12, cmc-cvx has nearly exact coverage at  $1 - \alpha$ , but cvx tends to have higher coverage than the target level. Besides, the convex approach has much larger intervals when  $r$  is large, which can be caused by the overfitting of the observed entries. As conformalized approach leaves out a proportion of observed entries as the training set, intervals produced by cmc-cvx are less accurate than cvx due to the poorly behaved base algorithm.
Figure 11: Sales dataset with heterogeneous and misspecified missingness.
(a) Homogeneous missingness.
Figure 12: Comparison between conformalized and model-based matrix completion with sales dataset.
(b) Heterogeneous missingness with  $k^{*} = 1$
C Additional details of algorithms and extensions
C.1 Extension to likelihood-based scores and categorical matrices
We will show in this section that cmc can also be applied to categorical matrix completion (Cao and Xie, 2015) or a more general setting in (19), the validity of which is also guaranteed by the presented theorem.
Setup To formulate the problem, consider an underlying parameter matrix  $\mathbf{M}^{*}\in [d_{1}]\times [d_{2}]$  and the observations  $\{M_{ij}:i\in [d_1],j\in [d_2]\}$  are drawn from the distribution
$$
M _ {i j} \mid M _ {i j} ^ {*} \sim \mathcal {P} _ {M _ {i j} ^ {*}}, \tag {19}
$$
where  $\{\mathcal{P}_{\theta}\}_{\theta \in \Theta}$  can be a family of parametric distributions with probability density  $p_{\theta}$ . The categorical matrix completion is a specific example where the support of  $M_{ij}$  is finite or countable. For example, a Poisson matrix is generated by  $M_{ij} \sim \mathrm{Pois}(M_{ij}^{*})$ , where  $M_{ij}^{*}$  is the Poisson mean. Similar to the previous setup, we treat  $\mathbf{M}$  as deterministic, and entries in the subset  $\mathcal{S} \subseteq [d_1] \times [d_2]$  are available. Here  $\mathcal{S}$  is sampled in the same manner as before with the matrix  $\mathbf{P} \in [d_1] \times [d_2]$ .
Split conformal approach Consider the split approach with the partition  $S = S_{\mathrm{tr}} \cup S_{\mathrm{cal}}$  as in Algorithm 1. With the training set  $\mathbf{M}_{\mathrm{tr}}$ , we obtain an estimated likelihood function  $\widehat{\pi}(m; i, j)$  such that
$$
\widehat {\pi} (m; i, j) = \widehat {p} _ {M _ {i j} ^ {*}} (m),
$$
which is an estimate for the true likelihood of  $M_{ij}$  at  $m$ . The estimation can be feasible given certain low-complexity structures. For example, if a hypothesized distribution family  $\{\mathcal{Q}_{\theta}\}_{\theta \in \Theta}$  with probability density  $q_{\theta}$  is given and the underlying mean matrix  $\mathbf{M}^*$  is assumed to be low-rank. Then,  $\mathbf{M}$  can be viewed as a perturbation of  $\mathbf{M}^*$  and we can estimate  $\mathbf{M}^*$  via matrix completion algorithms with entries in  $\mathbf{M}_{\mathrm{tr}}$ . Denote  $\widehat{\mathbf{M}}$  as the estimate for  $\mathbf{M}^*$ , then we have the estimated likelihood
$$
\widehat {\pi} (m; i, j) = q _ {\widehat {M} _ {i j}} (m).
$$
The odds ratios are also estimated from the training set, i.e.  $\widehat{h}_{ij}$ , from which we compute the weights
$$
\widehat {w} _ {i j} = \frac {\widehat {h} _ {i j}}{\sum_ {(i ^ {\prime} , j ^ {\prime}) \in \mathcal {S} _ {\mathrm {c a l}}} \widehat {h} _ {i ^ {\prime} j ^ {\prime}} + \max  _ {(i ^ {\prime} , j ^ {\prime}) \in \mathcal {S} ^ {c}} \widehat {h} _ {i ^ {\prime} j ^ {\prime}}}, (i, j) \in \mathcal {S} _ {\mathrm {c a l}}, \widehat {w} _ {\text {t e s t}} = \frac {\widehat {h} _ {i * j *}}{\sum_ {(i ^ {\prime} , j ^ {\prime}) \in \mathcal {S} _ {\mathrm {c a l}}} \widehat {h} _ {i ^ {\prime} j ^ {\prime}} + \max  _ {(i ^ {\prime} , j ^ {\prime}) \in \mathcal {S} ^ {c}} \widehat {h} _ {i ^ {\prime} j ^ {\prime}}}.
$$
For each  $(i,j)\in S_{\mathrm{cal}}$  , calculate the likelihood-based nonconformity score
$$
R _ {i j} = - \widehat {\pi} (M _ {i j}; i, j).
$$
Then, for any test point  $(i_{*},j_{*})\in S^{c}$ , we can construct the confidence interval
$$
\widehat {C} \left(i _ {*}, j _ {*}\right) = \left\{m \in [ K ]: \widehat {\pi} \left(m; i _ {*}, j _ {*}\right) \leq \widehat {q} \right\},
$$
where  $\widehat{q}$  is the weighted quantile
$$
\widehat {q} = \operatorname {Q u a n t i l e} _ {1 - \alpha} \left(\sum_ {(i, j) \in \mathcal {S} _ {\mathrm {c a l}}} \widehat {w} _ {i j} \cdot \delta_ {R _ {i j}} + \widehat {w} _ {\text {t e s t}} \cdot \delta_ {+ \infty}\right).
$$
More examples of conformal methods for classification are shown in Romano et al. (2020), Angelopoulos et al. (2020), etc.
C.2 Full conformalized matrix completion
In Algorithm 2, the procedure of the full conformalized matrix completion (full-cmc) is presented. This full conformal version of cmc offers the same coverage guarantee as given in Theorem 3.2 for the split version of cmc (except with the entire observed set  $S$  in place of  $S_{\mathrm{cal}}$ , when defining  $\Delta (\widehat{\pmb{w}})$ ; the formal proof of this bound for full conformal is very similar to the proof of Theorem 3.2, using an analogous weighted exchangeability argument as in Lemma 3.1, and so we omit it here.
To define this algorithm, we need some notation: given the observed data  $\mathbf{M}_S$ , plus a test point location  $(i_{*},j_{*})$  and a hypothesized value  $m$  for the test point  $M_{i_{*}j_{*}}$ , define a matrix  $\mathbf{M}^{(m)}$  with entries
$$
M _ {i j} ^ {(m)} = \left\{ \begin{array}{l l} M _ {i j}, & (i, j) \in \mathcal {S}, \\ m, & (i, j) = \left(i _ {*}, j _ {*}\right), \\ \varnothing , & \text {o t h e r w i s e}, \end{array} \right. \tag {20}
$$
where, abusing notation,  $M_{ij} = \emptyset$  denotes that no information is observed in this entry.
Algorithm 2 full-cmc: full conformalized matrix completion
1: Input: target level  $1 - \alpha$ ; partially observed matrix  $\mathbf{M}_{\mathcal{S}}$ .
2: Using the training data  $\mathbf{M}_{\mathcal{S}}$ , compute an estimate  $\widehat{\mathbf{P}}$  of the observation probabilities (with  $\widehat{p}_{ij}$  estimating  $p_{ij}$ , the probability of entry  $(i,j)$  being observed).
3: for  $(i_{*},j_{*})$  in  $S^c$  do
4: for  $m\in \mathcal{M}$  do
5: Augment  $\mathbf{M}_{\mathcal{S}}$  with one additional hypothesized entry,  $\{M_{i_{*},j_{*}} = m\}$ , to obtain  $\mathbf{M}^{(m)}$  defined as in (20).
6: Using the imputed matrix  $\mathbf{M}_S^{(m)}$ , compute:
- An initial estimate  $\widehat{\mathbf{M}}^{(m)}$  using any matrix completion algorithm (with  $\widehat{M}_{ij}^{(m)}$  estimating the target  $M_{ij}$ );
- Optionally, a local uncertainty estimate  $\widehat{\mathbf{s}}^{(m)}$  (with  $\widehat{s}_{ij}^{(m)}$ ) estimating our relative uncertainty in the estimate  $\widehat{M}_{ij}^{(m)}$ ), or otherwise set  $\widehat{s}_{ij}^{(m)} \equiv 1$ ;
- An estimate  $\widehat{\mathbf{P}}$  of the observation probabilities (with  $\widehat{p}_{ij}$  estimating  $p_{ij}$ , the probability of entry  $(i,j)$  being observed).
7: Compute normalized residuals for  $(i,j)\in S\cup \{(i_{*},j_{*})\}$
$$
R _ {i j} ^ {(m)} = \frac {| M _ {i j} - \widehat {M} _ {i j} ^ {(m)} |}{\widehat {s} _ {i j} ^ {(m)}}.
$$
8: Compute weights
$$
\widehat {w} _ {i j} = \frac {\widehat {h} _ {i j}}{\sum_ {(i ^ {\prime} , j ^ {\prime}) \in \mathcal {S} \cup \{(i _ {*} , j _ {*}) \}} \widehat {h} _ {i ^ {\prime} j ^ {\prime}}}, \qquad \widehat {h} _ {i j} = \frac {1 - \widehat {p} _ {i j}}{\widehat {p} _ {i j}}, \qquad (i, j) \in \mathcal {S} \cup \{(i _ {*}, j _ {*}) \}.
$$
9: Compute the weighted quantile
$$
\widehat {q} ^ {(m)} \left(i _ {*}, j _ {*}\right) = \operatorname {Q u a n t i l e} _ {1 - \alpha} \left(\sum_ {(i, j) \in \mathcal {S}} \widehat {w} _ {i j} \delta_ {R _ {i j} ^ {(m)}} + \widehat {w} _ {i * j _ {*}} \delta_ {+ \infty}\right) \tag {21}
$$
10: end for
11: end for
12: Output:  $\left\{\widehat{C}\left(i_{*},j_{*}\right) = \left\{m\in \mathcal{M}:R_{i_{*}j_{*}}^{(m)}\leq \widehat{q}^{(m)}(i_{*},j_{*})\right\} : (i_{*},j_{*})\in \mathcal{S}^{c}\right\}$
We note that, when  $\mathcal{M} = \mathbb{R}$  (or an infinite subset of  $\mathbb{R}$ ), the computation of the prediction set is impossible in most of the cases. In that case, our algorithm can be modified via a trimmed or discretized approximation; these extensions are presented for the regression setting in the work of
Chen et al. (2016, 2018), and can be extended to the matrix completion setting in a straightforward way.
C.3 Exact split conformalized matrix completion
In Algorithm 3, we present the exact split approach, which is less conservative than our one-shot approach given in Algorithm 1, but may be less computationally efficient. In this version of the algorithm, the quantile  $\widehat{q} = \widehat{q}(i_{*}, j_{*})$  needs to be computed for each missing entry since the weight vector  $\widehat{\boldsymbol{w}}$  depends on the value of  $\widehat{p}_{i_{*},j_{*}}$ .
Algorithm 3 split-cmc: split conformalized matrix completion
1: Input: target coverage level  $1 - \alpha$ ; data splitting proportion  $q \in (0,1)$ ; observed entries  $\mathbf{M}_S$ .
2: Split the data: draw  $W_{ij} \stackrel{\mathrm{i.i.d.}}{\sim} \operatorname{Bern}(q)$ , and define training and calibration sets,
$$
\mathcal {S} _ {\mathrm {t r}} = \{(i, j) \in \mathcal {S}: W _ {i j} = 1 \}, \quad \mathcal {S} _ {\mathrm {c a l}} = \{(i, j) \in \mathcal {S}: W _ {i j} = 0 \}.
$$
4: Compute normalized residuals on the calibration set,
$$
R _ {i j} = \frac {\left| M _ {i j} - \widehat {M _ {i j}} \right|}{\widehat {s} _ {i j}}, (i, j) \in \mathcal {S} _ {\mathrm {c a l}}.
$$
5: Compute estimated odds ratios for the calibration set and test set,
$$
\widehat {h} _ {i j} = \frac {\widehat {p} _ {i j}}{1 - \widehat {p} _ {i j}}, (i, j) \in \mathcal {S} _ {\mathrm {c a l}} \cup \mathcal {S} ^ {c},
$$
6: for  $(i_{*},j_{*})\in S^{c}$  do
7: Compute weights for the calibration set and test point,
$$
\widehat {w} _ {i j} = \frac {\widehat {h} _ {i j}}{\sum_ {(i ^ {\prime} , j ^ {\prime}) \in \mathcal {S} _ {\mathrm {c a l}}} \widehat {h} _ {i ^ {\prime} j ^ {\prime}} + \widehat {h} _ {i _ {*} j _ {*}}}, (i, j) \in \mathcal {S} _ {\mathrm {c a l}}, \widehat {w} _ {\mathrm {t e s t}} = \frac {\widehat {h} _ {i _ {*} j _ {*}}}{\sum_ {(i ^ {\prime} , j ^ {\prime}) \in \mathcal {S} _ {\mathrm {c a l}}} \widehat {h} _ {i ^ {\prime} j ^ {\prime}} + \widehat {h} _ {i _ {*} j _ {*}}}.
$$
8: Compute threshold
$$
\widehat {q} \left(i _ {*}, j _ {*}\right) = \operatorname {Q u a n t i l e} _ {1 - \alpha} \left(\sum_ {(i, j) \in S _ {\mathrm {c a l}}} \widehat {w} _ {i j} \cdot \delta_ {R _ {i j}} + \widehat {w} _ {\text {t e s t}} \cdot \delta_ {+ \infty}\right),
$$
where  $\delta_t$  denotes the point mass at  $t$ .
10: Output: confidence intervals
$$
\widehat {C} \left(i _ {*}, j _ {*}\right) = \widehat {M} _ {i _ {*} j _ {*}} \pm \widehat {q} \left(i _ {*}, j _ {*}\right) \cdot \widehat {s} _ {i _ {*} j _ {*}}
$$
for each unobserved entry  $(i_{*},j_{*})\in S^{c}$