SlowGuess/NeurIPS_2023_OCR huggingface dataset:
We now define the matrix completion problem with zero-mean noise, in fact a matrix estimation problem in the noisy case. Let  $\mathcal{D}$  be a distribution supported on  $\mathcal{X} \times [-1, 1]$ ; the results in the paper can all be easily extended when  $\mathcal{D}$  is supported on  $\mathcal{X} \times [-B, B]$ , increasing squared error by a  $B^2$  factor. Let  $\mu$  be the marginal distribution of  $\mathcal{D}$  over  $\mathcal{X}$ . Let  $S_{XY} = \langle (x_t, y_t) \rangle_{t=1}^N$  be an iid sample drawn from  $\mathcal{D}^N$ . Note here that  $x_t = (i_t, j_t)$  denotes the index of the matrix and  $y_t$  the observed value. We let  $S = (x_1, \ldots, x_N)$  denote the sequence of  $x_t$ 's from  $S_{XY}$  (with repetitions allowed); note that  $S$  is distributed as  $\mu^N$ . Let  $M^\star \in [-1, 1]^{\mathcal{X}}$  be the matrix where  $M_{ij}^\star = \mathbb{E}_{\mathcal{D}}[Y|X = (i,j)]$ . We say that  $\hat{M}$  is an  $\varepsilon$ -accurate completion of  $M^\star$ , if  $\ell(\mu, M^\star, \hat{M}) \leq \varepsilon$ .
Definition 8 (Full Completion Algorithm). We say that  $\mathrm{FullComp}(S_{XY},\varepsilon ,\delta ,\mathcal{M})$  is a full completion algorithm with sample complexity  $\mathfrak{sc}_{\mathsf{FC}}(\varepsilon ,\delta ,\mathcal{M})$  for  $\mathcal{M}$ , if provided  $S_{XY}\sim \mathcal{D}^N$  for some  $\mathcal{D}$  over  $\mathcal{X}\times [-1,1]$  with  $M^{\star}\in \mathcal{M}$ ,  $N\geq \mathsf{sc}_{\mathsf{FC}}(\varepsilon ,\delta ,\mathcal{M})$ ,  $\mathrm{FullComp}(S_{XY},\varepsilon ,\delta ,\mathcal{M})$  outputs  $\hat{M}\in \mathcal{M}$  that with probability at least  $1 - \delta$  satisfies,  $\ell (\mu ,M^{\star},\hat{M})\leq \varepsilon$
The following result follows from [Srebro and Shraibman, 2005, Shamir and Shalev-Shwartz, 2014] (see also Prop.7).
Proposition 9. There exists polynomial time (in  $mn / \varepsilon$ ) full completion algorithms for  $\mathcal{M}_K^{\max}$  and  $\mathcal{M}_{\tau}^{\mathrm{tr}}$  (e.g. ERM methods) with the following sample complexity bounds:
1.  $\mathfrak{sc}_{\mathsf{FC}}(\varepsilon, \delta, \mathcal{M}_K^{\max}) = O\left(\frac{1}{\varepsilon^2}\left(K^2(m + n) + \log \frac{1}{\delta}\right)\right)$ .
2.  $\mathfrak{sc}_{\mathcal{M}_{\tau}^{\mathrm{tr}}}\big(\varepsilon ,\delta ,\mathcal{M}_{\tau}^{\mathrm{tr}}\big) = O\left(\frac{1}{\varepsilon^{2}}\left(\tau \sqrt{m + n} +\log \frac{1}{\delta}\right)\right).$
2.3 The Partial Matrix Completion Problem
Recall the definition of the (fractional) partial matrix completion in Eq.1 and Def. In this work, we will assume that a full completion matrix  $\hat{M}$  is already obtained. The focus of the present work
is finding a completion matrix  $C \in \{0,1\}^{\mathcal{X}}$  (or  $C \in [0,1]^{\mathcal{X}}$  for fractional coverage) with large coverage. The coverage in both cases are defined as:  $|C| \stackrel{\text{def}}{=} \sum_{x \in \mathcal{X}} C_x$ , and a low loss, defined as:
$$
\ell (C, M ^ {\star}, \hat {M}) \stackrel {{\mathrm {d e f}}} {{=}} \frac {1}{| C |} \sum_ {x \in \mathcal {X}} C _ {x} \left(M _ {x} ^ {\star} - \hat {M} _ {x}\right) ^ {2}.
$$
Note that  $C \in [0,1]^{\mathcal{X}}$  can be viewed as a "fractional" set and we are overloading the notation  $\ell$  to allow that. Such a fractional coverage can be randomly rounded to a set whose size is within 1 of  $|C|$ . In an ideal world, we would have at most  $\varepsilon$  loss measured over the cells we complete as defined above, and large or even full coverage  $|C| = mn$ .
Note that although the requirement in the optimization problems in Eq. (1) and Def. (1) is to guarantee  $\ell(C, M^{\star}, M) \leq \varepsilon$ , as we don't know  $M^{\star}$ , the only guarantee we have by using a full completion algorithm (and generalization bounds) is that (with high probability)  $M^{\star}$  is in some version space, say  $\mathcal{V}$ , centered at  $\hat{M}$ . So we actually show a stronger guarantee that,  $\sup_{M \in \mathcal{V}} \ell(C, M, \hat{M}) \leq \varepsilon$ .
A second observation is that if  $M$  is a version space around  $\hat{M}$ , then by convexity of the matrix classes, the matrix  $(M - \hat{M}) / 2$ , must be in some version space, say  $\mathcal{V}_0$ , centered around the zero matrix,  $\mathbf{0}_{\mathcal{X}}$ . Thus, we will actually find a  $C$  that guarantees,  $\ell(C, M, \mathbf{0}_{\mathcal{X}}) \leq \varepsilon$  for every  $M \in \mathcal{V}_0$ . This means that for maximizing coverage our algorithm has the remarkable property we only need to know the locations of the revealed entries as indicated by  $S$ .
Section3 presents a computationally inefficient but statistically superior algorithm in terms of sample complexity for the coverage problem and its consequences for partial matrix completion. Section4 presents a computationally efficient algorithm at the cost of a slightly worse sample complexity when using the class  $\mathcal{M}_K^{\mathrm{max}}$ .
3 An Inefficient Algorithm
The main novelty in the partial matrix completion problem is that of finding an optimal coverage, defined as the matrix  $C$  in the formulation in Def. In this section we give an (inefficient) algorithm for finding the optimal coverage, and the generalization error bounds for Partial Matrix Completion that arise from it. In the next section we give an efficient approximation algorithm for doing the same, with slightly worse sample complexity bounds and more complex analysis. This differs substantially from prior work on abstention in classification and regression [Kalai and Kanade 2021] where the optimal solution can be found in polynomial time.
Let  $C \in [0,1]^{\mathcal{X}}$  be a target confidence matrix. For such a  $C$ , we will denote by  $\nu_{C}$  the probability distribution where  $\nu_{C}(x) = C_{x} / \|C\|_{1}$  for  $x \in \mathcal{X}$ . Let  $S \sim \mu^{N}$  be a sample obtained from the target distribution  $\mu$ . We output a  $C$  which is an optimal solution to the following optimization problem, inspired by a similar problem studied by Kalai and Kanade [2021] for classification with abstention.
Parameters and inputs:  $\gamma, \beta, S, \mathcal{M}$
maximize  $\| C\| _1$
subject to  $C\in [0,1]^{\mathcal{X}}$  , and  $\forall M\in \mathcal{V}(\mathbf{0}_{\mathcal{X}},\beta ,S;\mathcal{M}),\ell (\nu_{C},M,\mathbf{0}_{\mathcal{X}})\leq \gamma$
1: Inputs:  $S_{XY} = \langle (x_t,y_t)\rangle_{t = 1}^N\sim \mathcal{D}^N$ $\varepsilon ,\delta ,\mathcal{M}$  FullComp (cf. Def. 8).
2: Obtain  $\tilde{M} \in \mathcal{M}$  using FullComp $(S_{XY}, \varepsilon/4, \delta/3, \mathcal{M})$ .
3: Obtain  $C$  using MP4 with  $\gamma \stackrel{\mathrm{def}}{=} \varepsilon / 4, \beta \stackrel{\mathrm{def}}{=} \varepsilon / 8, S = (x_1, x_2, \ldots, x_N)$  from  $S_{XY}$ .
4: return  $(M, C)$ .
MP4 defines a family of optimization problems; we will typically use  $\mathcal{M}$  to be  $\mathcal{M}_K^{\mathrm{max}}$  or  $\mathcal{M}_{\tau}^{\mathrm{tr}}$ . The above optimization problem is in fact a linear program. The only unknowns are  $C_x$  for  $x \in \mathcal{X}$ . However, the set of constraints is infinite and it is unclear how a separation oracle for the constraint set
may be designed. Such an oracle could be designed by solving the following optimization problem:
$$
\max  _ {M \in \mathcal {V} (\mathbf {0} _ {\mathcal {X}}, \beta , S; \mathcal {M})} \ell \left(\nu_ {C}, M, \mathbf {0} _ {\mathcal {X}}\right). \tag {5}
$$
This optimization problem requires maximizing a quadratic function, and we prove it to be NP-hard with the additional PSD and symmetric constraints in Appendix A.2
The following result subsumes Corollary 2 We defer the proof of this theorem to Appendix A as the proof is similar to that of our efficient algorithm in the next section.
Theorem 10. Let  $\mathcal{M}$  be either  $\mathcal{M}_K^{\mathrm{max}}$  or  $\mathcal{M}_r^{\mathrm{tr}}$ . Let  $\mathcal{D}$  be distribution over  $\mathcal{X} \times [-1,1]$ ,  $\mu$  the marginal of  $\mathcal{D}$  over  $\mathcal{X}$ ,  $\mu_{\mathrm{max}} = \max_{i,j} \mathbb{P}_{\mu}((i,j)$  is sampled). Suppose that  $M^{\star}$ , defined as  $M_{ij}^{\star} = \mathbb{E}_{(X,Y) \sim \mathcal{D}}[Y|X = (i,j)]$  satisfies that  $M^{\star} \in \mathcal{M}$ . Furthermore, suppose that  $S_{XY} \sim \mathcal{D}^{N}$  and that FullComp is a full completion algorithm as in Defn. [8] Then, provided  $N \geq \max\{\mathsf{sc}_{\mathrm{FC}}(\varepsilon/4, \delta/3, \mathcal{M}), \mathsf{sc}(\varepsilon/8, \delta/3, \mathcal{M})\}$ , for  $(\hat{M}, C)$  output by Alg. [1] it holds that:
1.  $\| C \|_1 \geq 1 / \mu_{\max}$ ,
2.  $\frac{1}{\|C\|_1}\sum_{x\in \mathcal{X}}C_x(\hat{M}_x - M_x^\star)^2\leq \varepsilon .$
4 An Efficient Algorithm
In this section, we show how the result from Proposition 14 can be achieved using an efficient algorithm at a modest cost in terms of sample complexity when using the matrix class  $\mathcal{M}_K^{\mathrm{max}}$ . In particular, we consider the optimization problem defined in MP6 where the constraint  $\| M\|_{\nu_C}^2\leq \gamma$  for all  $M\in \mathcal{V}(\mathbf{0}_{\mathcal{X}},\beta ,S;\mathcal{M}_K^{\mathrm{max}})$  from MP4 is replaced by  $\mathbb{E}_{x\sim \nu_C}[M_x]\leq \gamma$ . The efficiency comes from the fact that we can now implement a separation oracle for the constraint set by solving, for a given  $C$ , the problem,
$$
\max_{M\in \mathcal{V}(\mathbf{0}_{\mathcal{X}},\beta ,S,\mathcal{M}_{K}^{\max})}\mathbb{E}_{x\sim \nu_{C}}[M_{x}].
$$
Mathematical program 6 is a convex optimization problem since the constraint set is convex and the objective function is linear. However, it is not immediately clear how this relaxed optimization problem relates to the original, which is the main technical contribution of this section.
Parameters and inputs:  $\gamma, \beta, S, \mathcal{M}_K^{\mathrm{max}}$
maximize  $\| C\| _1$  (6)
subject to  $C\in [0,1]^{\mathcal{X}}$  , and  $\forall M\in \mathcal{V}(\mathbf{0}_{\mathcal{X}},\beta ,S;\mathcal{M}_K^{\max})$ $\mathbb{E}_{x\sim \nu_C}[M_x]\leq \gamma$
Algorithm 2 (Efficient) Offline Algorithm for Partial Matrix Completion
1: Inputs:  $S_{XY} = \langle (x_t,y_t)\rangle_{t=1}^N \sim \mathcal{D}^N$ , FullComp (cf. Def. 8),  $\varepsilon, \delta$ .
2: Obtain  $\hat{M} \in \mathcal{M}_K^{\max}$  using FullComp $(S_{XY}, \varepsilon^2/(4\pi^2 K^2), \delta/3, \mathcal{M}_K^{\max})$ .
3: Obtain  $C$  using MP $\underline{6}$  with  $\gamma \stackrel{\mathrm{def}}{=} \varepsilon/(2\pi K)$ ,  $\beta \stackrel{\mathrm{def}}{=} \varepsilon^2/(8\pi^2 K^2)$ ,  $S = (x_1, x_2, \ldots, x_N)$  from  $S_{XY}$ .
4: return  $(M, C)$ .
The following result subsumes Corollary 3
Theorem 11. Let  $\mathcal{D}$  be a distribution over  $\mathcal{X} \times [-1,1]$ ,  $\mu$  the marginal of  $\mathcal{D}$  over  $\mathcal{X}$ ,  $\mu_{\max} = \max_{i,j} \mathbb{P}_{\mu}((i,j)$  is sampled). Suppose that  $M^{\star}$ , defined as  $M_{ij}^{\star} = \mathbb{E}_{(X,Y) \sim \mathcal{D}}[Y|X = (i,j)]$  satisfies that  $M^{\star} \in \mathcal{M}_K^{\max}$ . Furthermore, suppose that  $S_{XY} \sim \mathcal{D}^N$  and that FullComp is a full completion algorithm as in Defn. Then, provided  $N \geq \max\{\mathsf{sc}_{\mathrm{FC}}(\varepsilon^2/(4\pi^2 K^2), \delta/3, \mathcal{M}_K^{\max}), \mathsf{sc}(\varepsilon^2/(8\pi^2 K^2), \delta/3, \mathcal{M}_K^{\max})\}$ , for  $(\hat{M}, C)$  output by Alg. it holds that:
1.  $\| C\| _1\geq 1 / \mu_{\mathrm{max}}$
2.  $\frac{1}{\|C\|_1}\sum_{x\in \mathcal{X}}C_x(\hat{M}_x - M_x^\star)^2\leq \varepsilon .$
5 Online Setting
The previous section introduces the offline setting for the partial matrix completion problem. In Appendix B we describe the online version of the problem, which is motivated by two important considerations. First, in many applications, the observation pattern is more general than a fixed distribution. It can be a changing distribution or be comprised of adversarial observations. Second, our online algorithm incrementally updates the solution via iterative gradient methods, which is more efficient than the offline methods. For space considerations, details are deferred to Appendix B which contains the setting, definitions, algorithm specification, and main results, and Appendix C which details the proofs. In particular, the online algorithm, called Online Dual Descent (ODD), is described in Algorithm 3 and its regret guarantee in Theorem 18. The online regret guarantee implies the statistical learning guarantees of the previous sections when the support size is a constant fraction of the full matrix, and this implication is spelled out precisely in Corollary 22
6 Experiments and Implementation
The MovieLens dataset ([Harper and Konstan 2015]) consists of, among other data, a set of users with their rankings of a set of movies. It is a common benchmark for matrix completion because some of the rankings are missing, and one can make predictions on the rankings of user preferences. We used the dataset differently, aiming to test our online algorithm on generating a satisfying confidence matrix. The experimental procedure is outlined as follows: we used training data from 250 users and their ratings on 250 movies, giving us a total of 5189 completed samples from the incomplete matrix of size  $250 \times 250$ . We ran our algorithm, ODD (Algorithm 3), to get a confidence matrix  $C$ . In parallel, we used another standard matrix completion tool, fancyimpute (Rubinstein and Feldman 2016), to fill in the missing entries of the matrix and obtain a completion  $M_f$ . After  $C$  and  $\hat{M}_f$  are obtained, we reveal the true ratings at the missing entries using the validation set and computed the mean squared error of the predicted rating and true rating at each entry. The following plots show the distribution of the mean squared error with respect to the confidence score  $C$  assign at the particular entry.
Figure 2: (a) ODD heuristic on  $250 \times 250$  user-movie rating data from the MovieLens Dataset. Squared error in entry-wise prediction and entry-wise confidence level estimated by the ODD heuristic. (b) The average squared error over various confidence levels suggests that a higher confidence level is correlated with a lower squared error.
In this work we define the setting of partial matrix completion, with our high-level contributions outlined as the following:
A new framework. We propose a new framework called partial matrix completion. In this problem, one is given possibly noisy observations from a low-rank matrix, and the goal is to identify entries that can be predicted with high confidence. The partial matrix completion problem answers two key questions:
- Based on possibly noisy samples from an arbitrary unknown sampling distribution  $\mu$ , how many entries can be completed with  $\leq \varepsilon$  error?
- How can we efficiently identify the entries that can be accurately completed?
When the underlying matrix has low rank  $k$ , we show that it is possible to complete a large fraction of the matrix using only  $\tilde{O}(k(m + n))$  observations while simultaneously guaranteeing high accuracy over the completed set. We then study the complexity of identifying the optimal completion matrix. We show that a naive mathematical programming formulation of the problem is hard. However, we propose a relaxation that gives rise to efficient algorithms, which results in a slightly worse dependence on  $k$  for the sample complexity. These guarantees are outlined both in Corollary 2 and Corollary 3 and Theorem 10 and Theorem 11 for their more general versions.
Online game formulation. Furthermore, we consider the partial matrix completion problem in the online setting, where the revealed observations are not required to follow any particular fixed distribution. The goal henceforth is to minimize regret, the gap between the algorithm's performance and the single best decision in hindsight. This is a more general setting, as when imposed with distribution assumptions, regret guarantees in online learning algorithms naturally translate to statistical guarantees.
Our proposed online partial matrix completion algorithm is derived from an online repeated game. The version space is the set of all valid completions with low generalization error. High confidence should be assigned to entries where all completions in the version space are similar. Therefore, we formulate the problem as a two-player online game. One player iteratively updates a confidence matrix, and the other learns the version space. We gave an iterative gradient-based method with provable regret guarantees and concluded with preliminary experimental evidence of the validity of our framework.
Elad Hazan acknowledges funding from the Office of Naval Research grant N000142312156, the NSF award 2134040, and Open Philanthropy. This work was done in part when Clara Mohri was visiting and supported by Princeton University.
Emmanuel Abbe, Jianqing Fan, Kaizheng Wang, and Yiqiao Zhong. Entrywise eigenvector analysis of random matrices with low expected rank. Annals of statistics, 48(3):1452, 2020.
Emmanuel J Candès and Benjamin Recht. Exact matrix completion via convex optimization. Foundations of Computational mathematics, 9(6):717-772, 2009.
Emmanuel J Candès and Terence Tao. The power of convex relaxation: Near-optimal matrix completion. IEEE Transactions on Information Theory, 56(5):2053-2080, 2010.
Yuxin Chen, Yuejie Chi, Jianqing Fan, Cong Ma, and Yuling Yan. Noisy matrix completion: Understanding statistical guarantees for convex relaxation via nonconvex optimization. SIAM journal on optimization, 30(4):3098-3121, 2020.
Chi-Keung Chow. An optimum character recognition system using decision functions. IRE Transactions on Electronic Computers, 4:247-254, 1957.
Ran El-Yaniv and Yair Wiener. On the foundations of noise-free selective classification. Journal of Machine Learning Research, 11(53):1605-1641, 2010. URL http://jmlr.org/papers/v11/el-yaniv10a.html
Yoav Freund and Robert E Schapire. Adaptive game playing using multiplicative weights. Games and Economic Behavior, 29(1-2):79-103, 1999.
Yonatan Geifman and Ran El-Yaniv. Selective classification for deep neural networks. Advances in neural information processing systems, 30, 2017.
Yonatan Geifman and Ran El-Yaniv. Selectivenet: A deep neural network with an integrated reject option. In Kamalika Chaudhuri and Ruslan Salakhutdinov, editors, Proceedings of the 36th International Conference on Machine Learning, ICML 2019, 9-15 June 2019, Long Beach, California, USA, volume 97 of Proceedings of Machine Learning Research, pages 2151-2159. PMLR, 2019.
Ohad Shamir and Shai Shalev-Shwartz. Matrix completion with the trace norm: Learning, bounding, and transducing. Journal of Machine Learning Research, 15(1):3401-3423, 2014.
Nathan Srebro. Learning with Matrix Factorizations. PhD thesis, Massachusetts Institute of Technology, 2004.
Nathan Srebro and Adi Shraibman. Rank, trace-norm and max-norm. In International Conference on Computational Learning Theory, pages 545-560. Springer, 2005.
Nathan Srebro, Noga Alon, and Tommi Jaakkola. Generalization error bounds for collaborative prediction with low-rank matrices. Advances In Neural Information Processing Systems, 17, 2004.
Manfred K Warmuth. The blessing and the curse of the multiplicative updates. In International Conference on Discovery Science, pages 382-382. Springer, 2010.