SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Adam Tauman Kalai *
Varun Kanade *
Clara Mohri *
The matrix completion problem aims to reconstruct a low-rank matrix based on a revealed set of possibly noisy entries. Prior works consider completing the entire matrix with generalization error guarantees. However, the completion accuracy can be drastically different over different entries. This work establishes a new framework of partial matrix completion, where the goal is to identify a large subset of the entries that can be completed with high confidence. We propose an efficient algorithm with the following provable guarantees. Given access to samples from an unknown and arbitrary distribution, it guarantees: (a) high accuracy over completed entries, and (b) high coverage of the underlying distribution. We also consider an online learning variant of this problem, where we propose a low-regret algorithm based on iterative gradient updates. Preliminary empirical evaluations are included.
In the classical matrix completion problem, a subset of entries of a matrix are revealed and the goal is to reconstruct the full matrix. This is in general impossible, but if the matrix is assumed to be low rank and the distribution over revealed entries is uniformly random, then it can be shown that reconstruction is possible. A common application of matrix completion is in recommendation systems. For example, the rows of the matrix can correspond to users, the columns to movies, and an entry of the matrix is the preference score for the user over the corresponding movie. The completed entries can then be used to predict user preferences over unseen movies. The low rank assumption in this case is justified if the preference of users over movies is mostly determined by a small number of latent factors such as the genre, director, artistic style, and so forth.
However, in many cases, it is both infeasible and unnecessary to complete the entire matrix. First consider the trivial example of movie recommendations with users and movies coming from two countries  $A$  and  $B$ , where each user rates random movies only from their country. Without any cross ratings of users from country  $A$  on movies from country  $B$  or vice versa, it is impossible to accurately complete these entries. A solution here is straightforward: partition the users and movies into their respective groups and then complete only the part of the matrix corresponding to user ratings of movies from their own country.
In reality, many users have categories of movies with few or no ratings based on genre, language, or time period. For such users, it is difficult to accurately complete unrated movie categories which they do not like to watch from those which they have not even been exposed to. Thus it may be preferable to abstain from making rating predictions for such users on these unpredictable categories. This is
*Authors ordered alphabetically
further complicated by the fact that such categories are not crisply defined and relevant categories may vary across user demographics like country and age.
Figure 1: A simple example where some users rate only comedies and some rate only horror movies. However, these two groups are unknown and the rows and columns are permuted according to  $\pi = \pi_{C} \circ \pi_{R}$ . The partial completions made by our algorithm, shown on simulated data, correspond to completing the ratings for the half of the matrix as one would like. Note that our algorithms can handle arbitrary revealed subsets, not only stochastic block models. The experimental setup and more preliminary experiments are described in Section 6.
This motivates a more general question: can we identify a subset of the matrix that can be completed by the existing observations with high confidence? We call this problem partial matrix completion. Let  $M^{\star}$  be a target matrix. A non-convex formulation would be to find a matrix  $\hat{M}$  and a set of entries  $C \subseteq [m] \times [n]$  which maximizes:
$$
\max  _ {C \subseteq [ m ] \times [ n ]} | C | \text {s . t .} \frac {1}{| C |} \sum_ {(i, j) \in C} \left(M _ {i j} ^ {\star} - \hat {M} _ {i j}\right) ^ {2} \leq \varepsilon . \tag {1}
$$
This is non-convex because the set  $C$  is discrete and also because the choice of  $\hat{M}$  interacts with  $C$ . Fortunately, there are already several matrix completion algorithms which output  $\hat{M}$  that guarantee low "generalization error" with respect to future examples from the same distribution of entries as those from the training distribution, though they are not accurate over the entire matrix. We will fix any one of these completions  $\hat{M}$ .
We thus relax the goal of partial matrix completion to identify a matrix  $C$  which gives a confidence score for each entry between 0 and 1. A value of 0 indicates that this entry should not be completed, whereas 1 indicates we have absolute confidence. We call this matrix the confidence matrix. Ideally, we would like that the confidence matrix to have a value of 1 on all entries that are well supported by the observation distribution.
We formalize this problem through the following definitions. The precise definitions of some of the terms will be introduced in subsequent sections. Let  $M^{\star}$  be a target matrix and  $\hat{M}$  be a completion given by a matrix completion algorithm that guarantees low generalization error over future examples from the same entry distribution (formal definition given in Def.8), and let  $\varepsilon > 0$ . The coverage of  $C$  is defined to be  $\| C\|_1 \stackrel{\mathrm{def}}{=} \sum_{ij} |C_{ij}|$ .
Definition 1 (Partial matrix completion problem). Let  $M^{\star}$  be a target matrix and  $\hat{M}$  be a full completion (Def.8), and  $\varepsilon >0$  be an error tolerance. Find a confidence matrix  $C\in [0,1]^{m\times n}$  with maximal coverage, i.e.
$$
\operatorname *{arg  max}_{C\in [0,1]^{m\times n}}\| C\|_{1}\text{s.t.}\frac{1}{\|C\|_{1}}\sum_{i\in [m],j\in [n]}C_{ij}(M_{ij}^{\star} - \hat{M}_{ij})^{2}\leq \varepsilon .
$$
The matrix  $C$  that solves the above optimization problem has the following property:  $C_{ij}$  indicates whether the  $(i,j)$ -th entry shall be completed, and a fractional value can be interpreted as either a fractional completion or a probability of completion.  $C$  provides the largest coverage among all that satisfy the property that the average error is at most  $\varepsilon$  over the completed entries.
Challenges of partial matrix completion. The above formulation is intuitive and convex, but notice that we do not know  $M^{\star}$ , and thus cannot apply convex optimization directly.
Key challenges are illustrated by two intuitive, but naïve approaches. First, consider completing just those matrix entries  $(i,j)$  where all rank-  $k$  completions have nearly the same value  $\hat{M}_{ij}$ . Even in a simple case where all revealed entries are 1, any single entry could be  $\pm 1$  for some rank 2 completion of the matrix. Based on this, one often will not be able to complete any entries whatsoever.
Second, consider the generalization of our previous example where the sampling distribution is uniform over an arbitrary subset  $U$  of  $m \times n$  possible indices. For simplicity, think of  $|U| = mn / 2$  as covering half the matrix. It follows from previous generalization bounds on matrix completion that one could accurately complete all entries in  $U$ . However, in general  $U$  is unknown and arbitrarily complex. The second naive approach would be to try to learn  $U$ , but this requires  $\approx |U| = mn / 2 \gg m + n$  observations.
We present an inefficient algorithm and its efficient relaxation to solve the partial matrix completion problem with provable guarantees over arbitrary sampling distributions. In particular, if  $\mu$  is uniform over some arbitrary subset  $U \subseteq [m] \times [n]$  of a rank- $k$  matrix  $M^{\star} \in [-1, 1]^{m \times n}$ , the theoretical results obtained in this work have the following implications. The first, is an inefficient algorithm that solves the partial matrix completion problem for uniform distribution:
Corollary 2 (Inefficient algorithm, uniform sampling distribution). For any  $\varepsilon, \delta > 0$ , with probability  $\geq 1 - \delta$  over  $N \geq c_0 \cdot \frac{k(m + n) + \log(1 / \delta)}{\varepsilon^2}$  revealed entries from  $M^{\star}$  drawn from  $\mu$ , for some constant  $c_0$ , Algorithm  $\boxed{1}(\text{Sec. 3})$  outputs  $\hat{M}$  and  $C \in [0,1]^{m \times n}$ , such that
(1):  $\|C\|_1 \geq |U|$ ,
(2):  $\frac{1}{\|C\|_1}\sum_{i\in [m],j\in [n]}C_{ij}(M_{ij}^\star -\hat{M}_{ij})^2\leq \varepsilon .$
Next, we describe the guarantees of an efficient algorithm. It has a worse sample complexity, up to polynomial factors in  $k, \varepsilon$ . The advantage is that it runs in polynomial time. Otherwise, its guarantees are similar to that of the first algorithm.
Corollary 3 (Efficient algorithm, uniform sampling distribution). For any  $\varepsilon, \delta > 0$ , with probability  $\geq 1 - \delta$  over  $N \geq c_0 \cdot \frac{k^2}{\varepsilon^4} \cdot (k(m + n) + \log(1 / \delta))$  revealed entries from  $M^\star$  drawn from  $\mu$ , for some constant  $c_0$ , Algorithm [2] (Sec. 4) outputs  $\hat{M}$  and  $C \in [0,1]^{n \times m}$ , such that
(1):  $\| C \|_1 \geq |U|$ ,
(2):  $\frac{1}{\|C\|_1}\sum_{i\in [m],j\in [n]}C_{ij}(M_{ij}^\star -\hat{M}_{ij})^2\leq \varepsilon .$
Corollary 2 is a simplification of Theorem 10 and Corollary 3 is a simplification of Theorem 11
The results can be extended to the setting where the observed entries are noisy with zero-mean bounded noise. A remarkable feature of our algorithms is that once the full completion  $\hat{M}$  is obtained using existing procedures, they only rely on the locations of observed entries and not the values. Thus, the decisions regarding which entries to complete, i.e., to add to  $C$  is completely agnostic to the actual values of the revealed entries. Our framework is able to handle arbitrary sampling distributions, which captures scenarios such as overlapping groups or idiosyncratic rating habits.
The online setting. We formulate the partial matrix completion problem in the online learning setting and propose an iterative gradient based online algorithm with provable guarantees. Corollary 4 is a special case of Theorem 22. A simple simulation of the online algorithm in this special case of uniform sampling over a subset of entries is demonstrated in Fig.1
Corollary 4 (Online algorithm, uniform sampling distribution). Suppose  $M^{\star} \in [-1,1]^{n \times n}$  is a bounded matrix with max-norm (Eq. [2]) bounded by  $K$ . The sampling distribution  $\mu$  is uniform over a fraction  $0 < c \leq 1$  of the  $n^2$  entries. For any  $\delta > 0$ , after  $T = \tilde{O}(\delta^{-2}K^2n)$  iterations, the
confidence matrix  $C \in [0,1]^{m \times n}$  output by the online algorithm ODD satisfies with probability at least  $1 - c_{1} \exp(-c_{2} \delta^{2} T)$  for some universal constants  $c_{1}, c_{2} > 0$ ,
(1):  $\|C\|_1 \geq (c - \delta^{1/6})n^2$ ,
(2):  $\frac{1}{\|C\|_1}\sup_{M\in \mathcal{V}}\sum_{i\in [m],j\in [n]}C_{ij}(M_{ij} - M_{ij}^\star)^2\leq \frac{\delta^{1 / 6}}{c - \delta^{1 / 6}},$
where  $\mathcal{V}$  is the set of all matrices  $M\in [-1,1]^{n\times n}$  with max-norm bounded by  $K$  satisfying that  $\mathbb{E}_{i,j\sim \mu}[(M_{ij} - M_{ij}^{\star})^2 ]\leq \delta$
Matrix completion and recommendation systems. The common approach to collaborative filtering is that of matrix completion with trace norm minimization [Srebro 2004]. It was shown that the trace norm regularization requires  $\Omega((n + m)^{3/2})$  observations to complete the matrix under arbitrary distributions, even when the rank of the underlying matrix is constant, and this suffices even in the more challenging agnostic online learning setting [Hazan et al., 2012a; Shamir and Shalev-Shwartz, 2014; Srebro and Shraibman, 2005] study matrix completion with the max-norm, which behaves better under arbitrary distributions, namely for low-rank matrices,  $\tilde{O}(n + m)$  observations suffice.
Matrix completion and incoherence assumptions. A line of works in the matrix completion literature considered the goal of finding a completion of the ground truth matrix with low Frobenius norm error guarantee [Candès and Recht 2009, Candès and Tao 2010, Keshavan et al. 2010, Gross 2011, Recht 2011, Negahban and Wainwright 2012, Jain et al. 2013, Chen et al. 2020, Abbe et al. 2020]. Such guarantee is strong but usually requires two restrictive assumptions on sampling distribution of observations and the ground truth matrix structure: (1) the sampling of observations is uniform across all entries, and (2) the ground truth matrix  $M^{\star}$  satisfies some incoherence conditions. The incoherence condition is an assumption imposed on the singular vectors of  $M^{\star} = U\Sigma V^{\top}$ , which ensures the vectors  $u_{i}, v_{j}$ 's to be sufficiently spread out on the unit sphere. The main reason the incoherence condition is necessary in establishing meaningful guarantees in low-rank matrix completion is that without such assumptions, there might exist multiple low-rank matrices that could explain the observed entries equally well but differ substantially on unobserved entries. When the incoherence condition is satisfied, it implies that the observed entries capture enough information about the low-rank structure, making it possible to recover the original matrix with sufficiently well. However, uniform sampling and incoherence assumptions do not hold in many realistic scenarios.
Subsequently, to circumvent these restrictive assumptions, another line of work evaluates the generalization error as an alternative metric for completion performance [Srebro et al., 2004, Shamir and Shalev-Shwartz, 2014]. To formalize, consider the task of completing  $M^{\star}$  with an arbitrary observation sampling distribution  $\mu$  over  $[m] \times [n]$ . This can be conceptualized as predicting a hypothesis mapping from the domain  $[m] \times [n]$  to the codomain  $[-1, 1]$ . The goal is to characterize the guarantees on the expected prediction error, quantified for a given completion  $\hat{M}$  as  $\mathbb{E}_{(i,j)} \sim \mu[(\hat{M}_{ij} - M_{ij}^{\star})^2]$ . When  $\mu$  is the uniform distribution over  $[m] \times [n]$ , the generalization error bound translates to a guarantee over the average error across all entries. However, if  $\mu$  is arbitrary (e.g. supported over a fraction of the entries), then no guarantee could be established for entries that lie in the complement of the support of  $\mu$ . This prohibits the use of  $\hat{M}$  in settings where abstention is imperative.
Our work takes an alternative approach to this fundamental limitation in matrix completion. The focus of our work is to identify entries where we can predict with high confidence to guarantee a low completion error weighted by evaluated confidence. We provide formulation of the problem, its convex relaxation, and an efficient gradient-based online algorithm due to a formulation of the problem as an online two-player game.
Randomized rounding and semi-definite relaxations. For the efficient algorithm, the main analysis technique we deploy is based on randomized rounding solutions of semi-definite programming, due to the seminal work of Goemans and Williamson [1995]. These were originally developed in the context of approximation algorithms for MAX-CUT and other combinatorial problems. Here, we use this analysis technique for analyzing inner products of high dimensional vectors, but in a different context and to argue about a convex relaxation of a continuous optimization problem.
Abstention in classification and regression. Abstaining from prediction has a long history in other learning problems such as binary classification [Chow 1957]. There are several notable similarities. First, many of the algorithms work as ours does, by first fitting a complete classifier and then deciding afterwards where to abstain [e.g., Hellman 1970 Goldwasser et al. 2020]. Second, numerous abstention models have been considered, including a fixed cost for abstaining [e.g., Chow, 1957, Kalai and Kanade 2021], as well as fixing the error rate and maximizing the coverage [e.g., Geifman and El-Yaniv 2017], as in our work, and fixing the coverage rate while minimizing the error [e.g., Geifman and El-Yaniv 2019]. Third, there are algorithmic similarities, in particular the observation that worst-case performance guarantees do not depend on knowing the underlying distribution [Kalai and Kanade 2021]. Online models of abstention have also been considered [e.g., Li et al. 2011]. For detailed survey on selective classification see [El-Yaniv and Wiener 2010].
2 Problem Setup and Preliminaries
For a natural number  $n$ , let  $[n] = \{1, 2, \ldots, n\}$ . Consider a fixed set of indices  $\mathcal{X} = [m] \times [n]$  throughout. Thus, in this paper  $x \in \mathcal{X}$  is  $x = (i, j)$  where  $i \in [m], j \in [n]$ . The set of  $m \times n$  real-valued matrices is thus written as  $\mathbb{R}^{\mathcal{X}}$ . We will thus view a matrix  $M$  over index set  $\mathcal{X}$  as a function from  $\mathcal{X} \rightarrow \mathbb{R}$ , and for  $x \in \mathcal{X}$ , by slight abuse of notation denote by both  $M_x$  and  $M(x)$  the entry of  $M$  at index  $x$ .
We focus on the squared loss in this paper, though it may be interesting to consider other loss functions in future work. Suppose  $\nu$  is a distribution over  $\mathcal{X}$ , for matrices,  $M, M'$ , we denote by  $\ell(\nu, M, M') \stackrel{\mathrm{def}}{=} \mathbb{E}_{x \sim \nu}[(M(x) - M'(x))^2]$ ; we will also use the shorthand  $\|M - M'\|_{\nu}^2 \stackrel{\mathrm{def}}{=} \ell(\nu, M, M')$ .
When  $T \in \mathcal{X}^N$  is a sequence of elements of  $\mathcal{X}$  of length  $N$ , we denote by  $\ell(T, M, M') \stackrel{\mathrm{def}}{=} \frac{1}{N} \sum_{x \in T} (M(x) - M'(x))^2$  and the shorthand  $\| M - M' \|_T^2 \stackrel{\mathrm{def}}{=} \ell(T, M, M')$ .
2.1 Matrix Classes, Version Space and Generalization
Matrix Norms. Given a matrix  $M$ , we denote by  $\| M \|_{2,\infty}$  the maximum row norm of  $M$ . We denote by  $\| M \|_{\max}$  the max-norm of  $M$  defined as,
$$
\| M \| _ {\max } \stackrel {\text {d e f}} {=} \min  _ {U V ^ {\top} = M} \| U \| _ {2, \infty} \cdot \| V \| _ {2, \infty}. \tag {2}
$$
The trace norm of a matrix  $M$ , denoted by  $\| M \|_{\mathrm{tr}}$ , is the sum of its singular values. The Frobenius norm of a matrix  $M$ , denoted by  $\| M \|_{\mathrm{fr}}$ , is the square root of the sum of its squared entries.
We will restrict attention to matrices with entries in  $[-1, 1]$ . The following three classes of matrices with restrictions on respectively the rank, max-norm and trace-norm are of interest in this work. Formally, let  $\tilde{\mathcal{K}} = \{M \in [-1, 1]^{\mathcal{X}}\}$ , and define,
$$
\mathcal {M} _ {k} ^ {\mathrm {r k}} \stackrel {{\mathrm {d e f}}} {{=}} \{\operatorname {r a n k} (M) \leq k \} \cap \tilde {\mathcal {K}}, \quad \mathcal {M} _ {K} ^ {\max } \stackrel {{\mathrm {d e f}}} {{=}} \{\| M \| _ {\max } \leq K \} \cap \tilde {\mathcal {K}}, \quad \mathcal {M} _ {\tau} ^ {\mathrm {t r}} \stackrel {{\mathrm {d e f}}} {{=}} \{\| M \| _ {\mathrm {t r}} \leq \tau \} \cap \tilde {\mathcal {K}}.
$$
The standard assumption in matrix completion is that the target matrix is low-rank; however, most completion algorithms exploit convex optimization methods and optimize over max-norm or trace-norm bounded matrices. It is known that  $\| M\|_{\max}\leq \sqrt{\operatorname{rank}(M)}$  if  $M\in [-1,1]^{\mathcal{X}}$  [cf. Linial et al. 2007 Lemma 4.2] and it is easy to see that  $\| M\|_{\mathrm{tr}}\leq \mathrm{rank}(M)\sqrt{mn}$  for  $M\in [-1,1]^{\mathcal{X}}$ . Thus,  $\mathcal{M}_k^{\mathrm{rk}}\subseteq \mathcal{M}_{\sqrt{k}}^{\mathrm{max}}$  and  $\mathcal{M}_k^{\mathrm{rk}}\subseteq \mathcal{M}_{k\sqrt{mn}}^{\mathrm{tr}}$ . Hence, we will work with the latter two classes which satisfy the following lemma which is proved in Appendix A
Lemma 5. The classes,  $\mathcal{M}_K^{\mathrm{max}}$  and  $\mathcal{M}_{\tau}^{\mathrm{tr}}$ , are closed under negations and are convex.
Version Space. We define the notion of version spaces that are used in our key results. For a sequence  $T \in \mathcal{X}^N$ , for any class of matrices  $\mathcal{M}$ , a matrix  $M \in \mathcal{M}$  and  $\beta > 0$ , we define the version space around  $M$  of radius  $\beta$  based on  $T$  w.r.t.  $\mathcal{M}$  as
$$
\mathcal {V} (M, \beta , T; \mathcal {M}) = \{M ^ {\prime} \in \mathcal {M} \mid \ell (T, M, M ^ {\prime}) \leq \beta \}. \tag {3}
$$
Intuitively, version space is the set of matrices in a particular matrix class that are "close" to a given matrix with respect to  $T$ .
Generalization Bounds. For a general class of matrices  $\mathcal{M}$ , we define a notion of sample complexity that will guarantee the proximity of empirical and population measures of interest for all matrices in the class; we denote this notion by  $\mathfrak{sc}(\varepsilon, \delta, \mathcal{M})$ .
Definition 6 (Sample Complexity). For  $\varepsilon, \delta > 0$  and a class of matrices  $\mathcal{M}$ , denote by  $\operatorname{sc}(\varepsilon, \delta, \mathcal{M})$ , the sample complexity, to be the smallest natural number  $N_0$ , such that for any distribution  $\mu$  over  $\mathcal{X}$  for  $S \sim \mu^{N_0}$ , and for any fixed  $\hat{M} \in \mathcal{M}$  (possibly depending on  $S$ ), with probability at least  $1 - \delta$ ,
$$
\sup  _ {M \in \mathcal {M}} \left| \ell (S, M, \hat {M}) - \ell (\mu , M, \hat {M}) \right| \leq \varepsilon .
$$
If no such  $N_0$  exists,  $\mathfrak{sc}(\varepsilon, \delta, \mathcal{F}) \stackrel{\mathrm{def}}{=} \infty$ .
Bounds on the sample complexity for matrix completion can be derived in terms of rank, max-norm and trace-norm using standard results in the literature, and the fact that the squared loss is 2-Lipschitz and bounded by 4 when both its arguments take values in  $[-1,1]$ . In the proposition below, the max-norm result follows from Theorem 5 in [Srebro and Shraibman 2005] and the trace-norm result follows from Theorem 4 in [Shamir and Shalev-Shwartz 2014].
Proposition 7. For the classes,  $\mathcal{M}_K^{\mathrm{max}}$ ,  $\mathcal{M}_{\tau}^{\mathrm{tr}}$ , the following hold,
1.  $\mathfrak{sc}(\varepsilon, \delta, \mathcal{M}_K^{\max}) = O\left(\frac{1}{\varepsilon^2}\left(K^2(m + n) + \log \frac{1}{\delta}\right)\right)$ .
2.  $\mathfrak{sc}(\varepsilon, \delta, \mathcal{M}_{\tau}^{\mathrm{tr}}) = O\left(\frac{1}{\varepsilon^{2}}\left(\tau \sqrt{m + n} + \log \frac{1}{\delta}\right)\right)$ .
It is worth comparing the two bounds in Proposition 7 above. Consider the matrix  $M^{\star}$  consisting of all 1's - from a matrix completion point of view this is particularly easy as every "user" likes every "movie". Note however that  $\| M^{\star}\|_{\max} = 1$  and  $\| M^{\star}\|_{\mathrm{tr}} = \sqrt{mn}$ . For this example, ignoring the dependence on  $\varepsilon, \delta$ , the sample complexity bound obtained using the trace-norm result would be  $O((m + n)\sqrt{m + n})$ , while that using max-norm would be  $O(m + n)$ . In general, it is always the case that  $\| M\|_{\mathrm{tr}} / \sqrt{mn} \leq \| M\|_{\max}$ , so it may seem that the bounds in terms of trace-norm are weaker. However, there are matrices for which this gap can be large and the sample complexity bound in terms of trace-norm is shown to be tight in general (see [Srebro and Shraibman, 2005] [Shamir and Shalev-Shwartz, 2014] for further details).
2.2 Full Completion Problem (with Noise)