SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Idan Attias
idanatti@post.bgu.ac.il
Alkis Kalavasis
alvertos.kalavasis@yale.edu
Yale University, Google Research
Grigoris Velegkas
grigoris.velegkas@yale.edu
In this work, we aim to characterize the statistical complexity of realizable regression both in the PAC learning setting and the online learning setting.
Previous work had established the sufficiency of finiteness of the fat shattering dimension for PAC learnability and the necessity of finiteness of the scaled Natarajan dimension, but little progress had been made towards a more complete characterization since the work of Simon (SICOMP '97). To this end, we first introduce a minimax instance optimal learner for realizable regression and propose a novel dimension that both qualitatively and quantitatively characterizes which classes of real-valued predictors are learnable. We then identify a combinatorial dimension related to the Graph dimension that characterizes ERM learnability in the realizable setting. Finally, we establish a necessary condition for learnability based on a combinatorial dimension related to the DS dimension, and conjecture that it may also be sufficient in this context.
Additionally, in the context of online learning we provide a dimension that characterizes the minimax instance optimal cumulative loss up to a constant factor and design an optimal online learner for realizable regression, thus resolving an open question raised by Daskalakis and Golowich in STOC '22.
Real-valued regression is one of the most fundamental and well-studied problems in statistics and data science [Vap99, GBC16, Bac21], with numerous applications in domains such as economics and medicine [DG17]. However, despite its significance and applicability, theoretical understanding of the statistical complexity of real-valued regression is still lacking.
Perhaps surprisingly, in the fundamental realizable Probably Approximately Correct (PAC) setting [Val84] and the realizable online setting [Lit88, DG22], we do not know of any characterizing dimension or optimal learners for the regression task. This comes in sharp contrast with binary and multiclass classification, both in the offline and the online settings, where the situation is much more clear [Lit88, HLW94, BEHW89, Han16, DSS14, BCD $^{+}$ 22, Nat89, DSBDSS15, RST23]. Our goal in this work is to make progress regarding the following important question:
Which dimensions characterize PAC and online learnability for realizable real-valued regression?
Consider an instance space  $\mathcal{X}$ , label space  $\mathcal{Y} = [0,1]$  and hypothesis class  $\mathcal{H} \subseteq [0,1]^{\mathcal{X}}$ . In this work, we focus on the case of the regression framework with respect to the absolute loss  $\ell(x,y) \triangleq |x - y|$  for any  $x,y \in [0,1]$ , following previous works [BLW94, Sim97, ABDCBH97, BL98, DG22]. Our qualitative results hold for more general losses, namely any approximate pseudo-metric loss, which includes common losses such as  $\ell_p$  for any  $p \geq 1$ . See the formal statements in Appendix H.
PAC/Offline Realizable Regression. Let us first recall the definition of realizable real-valued regression in the PAC setting. Informally, the learner is given i.i.d. labeled examples drawn from an unknown distribution  $\mathcal{D}$  with the promise that there exists a target hypothesis  $h^{\star} \in \mathcal{H}$  that perfectly labels the data. The goal is to use this training set to design a predictor with small error on future examples from the same distribution. Note that given a learner  $A$  and sample  $S \sim \mathcal{D}^n$ , we let  $A(S; x)$  be its prediction on  $x \in \mathcal{X}$  when the training set is  $S$ .
Definition 1 (PAC Realizable Regression). Let  $\ell :[0,1]^2\to \mathbb{R}_{\geq 0}$  be the absolute loss function. Consider a class  $\mathcal{H}\subseteq [0,1]^{\mathcal{X}}$  for some domain  $\mathcal{X}$ . Let  $h^\star \in \mathcal{H}$  be an unknown target function and let  $\mathcal{D}_{\mathcal{X}}$  be an unknown distribution on  $\mathcal{X}$ . A random sample  $S$  of size  $n$  consists of points  $x_{1},\ldots ,x_{n}$  drawn i.i.d. from  $\mathcal{D}_{\mathcal{X}}$  and the corresponding values  $h^\star (x_1),\ldots ,h^\star (x_n)$  of the target function.
- An algorithm  $A: (\mathcal{X} \times [0,1])^n \to [0,1]^{\mathcal{X}}$  is an n-sample PAC learner for  $\mathcal{H}$  with respect to  $\ell$  if, for all  $0 < \varepsilon, \delta < 1$ , there exists  $n = n(\varepsilon, \delta) \in \mathbb{N}$  such that for any  $h^\star \in \mathcal{H}$  and any domain distribution  $\mathcal{D}_{\mathcal{X}}$ , it holds that  $\mathbf{E}_{x \sim \mathcal{D}_{\mathcal{X}}}[\ell(A(S; x), h^\star(x))] \leq \varepsilon$ , with probability at least  $1 - \delta$  over  $S \sim \mathcal{D}_{\mathcal{X}}^n$ .
- An algorithm  $A: (\mathcal{X} \times [0,1])^n \to [0,1]^{\mathcal{X}}$  is an  $n$ -sample cut-off PAC learner for  $\mathcal{H}$  with respect to  $\ell$  if, for all  $0 < \varepsilon, \delta, \gamma < 1$ , there exists  $n = n(\varepsilon, \delta, \gamma) \in \mathbb{N}$  such that for any  $h^\star \in \mathcal{H}$  and any domain distribution  $\mathcal{D}_{\mathcal{X}}$ , it holds that  $\mathbf{Pr}_{x \sim \mathcal{D}_{\mathcal{X}}}[\ell(A(S; x), h^\star(x)) > \gamma] \leq \varepsilon$ , with probability at least  $1 - \delta$  over  $S \sim \mathcal{D}_{\mathcal{X}}^n$ .
We remark that these two PAC learning definitions are qualitatively equivalent (cf. Lemma 1). We note that, throughout the paper, we implicitly assume, as e.g., in [HKS19], that all hypothesis classes are admissible in the sense that they satisfy mild measure-theoretic conditions, such as those specified in [DKLD84] (Section 10.3.1) or [Pol12] (Appendix C).
The question we would like to understand in this setting (and was raised by [Sim97]) follows:
Question 1. Can we characterize learnability and design minimax optimal PAC learners for realizable regression?
Traditionally, in the context of statistical learning a minimax optimal learner  $A^{\star}$  is one that, for every class  $\mathcal{H}$ , given an error parameter  $\varepsilon$  and a confidence parameter  $\delta$  requires  $\min_A\max_{\mathcal{D}}\mathcal{M}_A(\mathcal{H};\varepsilon ,\delta ,\mathcal{D})$  samples to achieve it, where  $\mathcal{M}_A(\mathcal{H};\varepsilon ,\delta ,\mathcal{D})$  is the number of samples that some learner  $A$  requires to achieve error  $\varepsilon$  with confidence  $\delta$  when the data-generating distribution is  $\mathcal{D}$ . There seems to be some inconsistency in the literature about realizable regression, where some works present minimax optimal learners when the maximum is also taken over the hypothesis class  $\mathcal{H}$ , i.e.,  $\min_A\max_{\mathcal{D},\mathcal{H}}\mathcal{M}_A(\mathcal{H},\varepsilon ,\delta ,\mathcal{D})$ . This is a weaker result compared to ours, since it shows that there exists some hypothesis class for which these learners are optimal.
Our main results in this setting, together with the uniform convergence results from prior work, give rise to an interesting landscape of realizable PAC learning that is depicted in Figure 1.
Online Realizable Regression. We next shift our attention to the classical setting of online learning where the learner interacts with the adversary over a sequence of  $T$  rounds: in every round the adversary presents an example  $x_{t} \in \mathcal{X}$ , the learner predicts a label  $\hat{y}_{t}$  and then the adversary reveals the correct label  $y_{t}^{\star}$ . In this context, realizability means that there always exists some function  $h_t \in \mathcal{H}$  that perfectly explains the examples and the labels that the adversary has chosen. In the agnostic setting, the goal of the learner is to compete with the performance of the best function in  $\mathcal{H}$ , i.e., achieve small regret. This setting was introduced by [Lit88] in the context of binary classification, where they also characterized learnability in the realizable setting and provided an optimal algorithm. Later, [BDPSS09] provided an almost optimal algorithm in the agnostic setting, which suffered from an additional log  $T$  factor in its regret bound. This extra factor was later shaved
Figure 1: Landscape of Realizable PAC Regression: the "deleted" arrows mean that the implication is not true. The equivalence between finite fat-shattering dimension and the uniform convergence property is known even in the realizable case (see [SSSSS10]) and the fact that PAC learnability requires finite scaled Natarajan dimension is proved in [Sim97]. The properties of the other three dimensions (scaled Graph dimension, scaled One-Inclusion-Graph (OIG) dimension, and scaled Daniely-Shalev Shwartz (DS) dimension) are shown in this work. We further conjecture that finite scaled Natarajan dimension is not sufficient for PAC learning, while finite scaled DS does suffice. Interestingly, we observe that the notions of uniform convergence, learnability by any ERM and PAC learnability are separated in realizable regression.
by  $\mathrm{[ABED^{+}21]}$ . The study of multiclass online classification was initiated by [DSBDSS15], who provided an optimal algorithm for the realizable setting and an algorithm that is suboptimal by a factor of  $\log k\cdot \log T$  in the agnostic setting, where  $k$  is the total number of labels. Recently, [RST23] shaved off the  $\log k$  factor. The problem of online regression differs significantly from that of online classification, since the loss function is not binary. The agnostic online regression setting has received a lot of attention and there is a series of works that provides optimal minimax guarantees [RST10, RS14, RST15b, RST15a, BDR21].
To the best of our knowledge, the realizable setting has received much less attention. A notable exception is the work of [DG22] that focuses on realizable online regression using proper learners. They provide an optimal regret bound with respect to the sequential fat-shattering dimension. However, as they mention in their work (cf. Examples 1 and 2), this dimension does not characterize the optimal cumulative loss bound.
Interestingly, Daskalakis and Golowich [DG22] leave the question of providing a dimension that characterizes online realizable regression open. In our work, we resolve this question by providing upper and lower bounds for the cumulative loss of the learner that are tight up to a constant factor of 2 using a novel combinatorial dimension that is related to (scaled) Littlestone trees (cf. Definition 13). Formally, the setting of realizable online regression is defined as follows:
Definition 2 (Online Realizable Regression). Let  $\ell :[0,1]^2\to \mathbb{R}_{\geq 0}$  be a loss function. Consider a class  $\mathcal{H}\subseteq [0,1]^{\mathcal{X}}$  for some domain  $\mathcal{X}$ . The realizable online regression setting over  $T$  rounds consists of the following interaction between the learner and the adversary:
- The adversary presents  $x_{t} \in \mathcal{X}$ .
- The learner predicts  $\widehat{y}_t \in [0,1]$ , possibly using randomization.
- The adversary reveals the true label  $y_{t}^{\star} \in [0,1]$  with the constraint that  $\exists h_{t}^{\star} \in \mathcal{H}, \forall \tau \leq t, h(x_{\tau}) = y_{\tau}^{\star}$ .
- The learner suffers loss  $\ell (\widehat{y}_t,y_t^\star)$
The goal of the learner is to minimize its expected cumulative loss  $\mathfrak{C}_T = \mathbf{E}\left[\sum_{t\in [T]}\ell (\widehat{y}_t,y_t^\star)\right]$ .
We remark that in the definition of the cumulative loss  $\mathfrak{C}_T$ , the expectation is over the randomness of the algorithm (which is the only stochastic aspect of the online setting). As we explained before, the main question we study in this setting is the following:
A learner is proper when the predictions  $\widehat{y}_t$  can be realized by some function  $h_t \in \mathcal{H}$ .
Question 2. Can we characterize the optimal cumulative loss and design optimal online learners for realizable regression?
Our main result in this setting provides a dimension that characterizes the optimal cumulative loss up to a factor of 2 and provides an algorithm that achieves this bound.
Our work makes progress towards the characterization of realizable regression both in the offline and online settings. Similar results are known for the more studied settings of binary and multiclass (PAC/online) classification [Lit88, HLW94, BEHW89, Han16, DSS14,  $\mathrm{BCD}^{+}22$ , Nat89, DSBDSS15, RBR09, BLM20, RST23,  $\mathrm{ABED}^{+}21$ , HKLM22, KKMV23,  $\mathrm{BGH}^{+}23$ ]. The fundamental works of [BLW94, Sim97, ABDCBH97, BL98] study the question of PAC learnability for the regression task. However, none of them characterizes learnability in the realizable setting. [Sim97] showed that finiteness of scaled Natarajan dimension is necessary for realizable PAC regression. [BL98] employed the one-inclusion graph (OIG) algorithm to get a real-valued predictor whose expected error is upper bounded by the  $V_{\gamma}$ -dimension, whose finiteness is sufficient but not necessary for realizable learnability in this setting. We refer to [KS23] for details about this dimension which was introduced in [ABDCBH97]. [ABDCBH97] showed that finiteness of the fat shattering dimension at all scales is equivalent to PAC learnability in the agnostic setting. Nevertheless, this does not hold in the realizable case as Example 1 demonstrates. Similarly, the work of [BLW94] shows that fat shattering dimension characterizes the regression task when the labels are corrupted by noise. Recently, the concurrent and independent work of [AACSZ23] provides high-probability bounds for the one-inclusion graph algorithm in the realizable PAC regression setting using the  $V_{\gamma}$ -dimension. We underline that this dimension does not characterize realizable regression. For more general losses, see [Men02, BBM05].
In the area of online regression, the work of [DG22] studies the realizable setting of online regression with the absolute loss (as we do) and presents a randomized proper learning algorithm which achieves a near-optimal cumulative loss in terms of the sequential fat-shattering dimension of the hypothesis class. We emphasize that this dimension does not tightly capture this setting and, hence, does not address the question that we study. In particular, the lower bound they provide is related to the unimprovability of a bound concerning the sequential fat-shattering dimension and does not tightly capture the complexity of the problem. In the more general agnostic case, regret bounds have been obtained in the work of [RST15a], using the sequential fat-shattering dimension and sequential covering numbers. These quantities are also not tight in the realizable setting. See also [RS14, RST15b, RST17]. Moreover, regression oracles have been used in contextual bandits problems, see [FR20, SLX22] for further details.
Our characterization (cf. Theorem 2) for the offline setting via the OIG algorithm [HLW94] is further motivated by the work of [MHS22], where they propose a dimension of similar flavor for adversarially robust PAC learnability. We mention that recently the OIG algorithm has received a lot of attention from the statistical learning theory community [AACSZ22, BHM+21, KVK22, SMB22, AHHM22, CP22]. Finally, for a small sample of works that deal with offline and online regression problems, see [BDGR22, Gol21, AH22, HKS19, She17, MKFI22] and the references therein.
2 PAC Learnability for Realizable Regression
In this section, we present various combinatorial dimensions that provide necessary or sufficient conditions for learnability of real-valued functions. All the definitions that we consider have a similar flavor. We first define what it means for a class  $\mathcal{H}$  to "shatter" a set of  $n$  points and then we define the dimension to be equal to the cardinality of the largest set that  $\mathcal{H}$  can shatter. Moreover, since we are considering real-valued learning, these dimensions are parameterized by a scaling factor  $\gamma \in (0,1)$  which should be interpreted as the distance that we can get to the optimal function. We start with the standard notion of projection of a class to a set of unlabeled examples.
Definition 3 (Projection of  $\mathcal{H}$  to  $S$ ). Given  $S = \{x_1, \ldots, x_n\} \in \mathcal{X}^n$ , the projection of  $\mathcal{H} \subseteq \mathcal{Y}^{\mathcal{X}}$  to  $S$  is  $\mathcal{H}|_S = \{(h(x_1), \ldots, h(x_n)): h \in \mathcal{H}\}$ .
Furthermore, we say that a labeled sample  $S \in (\mathcal{X} \times [0,1])^n$  is realizable with respect to  $\mathcal{H}$  if there exists  $h \in \mathcal{H}$  such that  $h(x_i) = y_i, \forall i \in [n]$ .
2.1  $\gamma$ -Fat Shattering Dimension
Perhaps the most well-known dimension in the real-valued learning setting is the fat shattering dimension that was introduced in [KS94]. Its definition is inspired by the pseudo-dimension [Pol90] and it is, essentially, a scaled version of it.
Definition 4 ( $\gamma$ -Fat Shattering Dimension [KS94]). Let  $\mathcal{H} \subseteq [0,1]^{\mathcal{X}}$ . We say that a sample  $S \in \mathcal{X}^n$  is  $\gamma$ -fat shattered by  $\mathcal{H}$  if there exist  $s_1, \ldots, s_n \in [0,1]^n$  such that for all  $b \in \{0,1\}^n$  there exists  $h_b \in \mathcal{H}$  such that:
-  $h_b(x_i) \geq s_i + \gamma, \forall i \in [n]$  such that  $b_i = 1$ .
-  $h_b(x_i) \leq s_i - \gamma, \forall i \in [n]$  such that  $b_i = 0$ .
The  $\gamma$ -fat shattering dimension  $\mathbb{D}_{\gamma}^{\mathrm{fat}}$  is defined to be the maximum size of a  $\gamma$ -fat shattered set.
In the realizable setting, finiteness of the fat-shattering dimension (at all scales) is sufficient for learnability and it is equivalent to uniform convergence [BLW94]. However, the next example shows that it is not a necessary condition for learnability. This comes in contrast to the agnostic case for real-valued functions, in which learnability and uniform convergence are equivalent for all  $\mathcal{H}$ .
Example 1 (Realizable Learnability  $\nRightarrow$  Finite Fat-Shattering Dimension, see Section 6 in [BLW94]). Consider a class  $\mathcal{H} \subseteq [0,1]^{\mathcal{X}}$  where each hypothesis is uniquely identifiable by a single example, i.e., for any  $x \in \mathcal{X}$  and any  $f, g \in \mathcal{H}$  we have that  $f(x) \neq g(x)$ , unless  $f \equiv g$ . Concretely, for every  $d \in \mathbb{N}$ , let  $\{S_j\}_{0 \leq j \leq d-1}$  be a partition of  $\mathcal{X}$  and define  $\mathcal{H}_d = \{h_{b_0, \ldots, b_{d-1}} : b_i \in \{0,1\}, 0 \leq i \leq d-1\}$ , where
$$
h _ {b _ {0}, \dots , b _ {d - 1}} (x) = \frac {3}{4} \sum_ {j = 0} ^ {d - 1} \mathbb {1} _ {S _ {j}} (x) b _ {j} + \frac {1}{8} \sum_ {k = 0} ^ {d - 1} b _ {k} 2 ^ {- k}.
$$
For any  $\gamma \leq 1/4$ ,  $\mathbb{D}_{\gamma}^{\mathrm{fat}}(\mathcal{H}_d) = d$ , since for a set of points  $x_0, \ldots, x_{d-1} \in \mathcal{X}$  such that each  $x_j$  belongs to  $S_j$ ,  $0 \leq j \leq d-1$ ,  $\mathcal{H}_d$  contains all possible patterns of values above  $3/4$  and below  $1/4$ . Indeed, it is not hard to verify that for any  $j \in \{0, \ldots, d-1\}$  if we consider any  $h' := h_{b_0, \ldots, b_{d-1}} \in \mathcal{H}_d$  with  $b_j = 1$  we have  $h'(x_j) \geq 3/4$ . Similarly, if  $b_j = 0$  then it holds that  $h'(x_j) \leq 1/4$ . Hence,  $\mathbb{D}_{\gamma}^{\mathrm{fat}}(\cup_{d \in \mathbb{N}} \mathcal{H}_d) = \infty$ . Nevertheless, by just observing one example  $(x, h^\star(x))$  any ERM learner finds the exact labeling function  $h^\star$ .
We remark that this example already shows that the PAC learnability landscape of regression is quite different from that of multiclass classification, where agnostic learning and realizable learning are characterized by the same dimension  $\left[\mathrm{BCD}^{+}22\right]$ .
To summarize this subsection, the fat-shattering dimension is a natural way to quantify how well the function class can interpolate (with gap  $\gamma$ ) some fixed function. Crucially, this interpolation contains only inequalities (see Definition 4) and hence (at least intuitively) cannot be tight for the realizable setting, where there exists some function that exactly labels the features. Example 1 gives a natural example of a class with infinite fat-shattering dimension that can, nevertheless, be learned with a single sample in the realizable setting.
2.2  $\gamma$ -Natarajan Dimension
The  $\gamma$ -Natarajan dimension was introduced by [Sim97] and is inspired by the Natarajan dimension [Nat89], which has been used to derive bounds in the multiclass classification setting. Before explaining the  $\gamma$ -Natarajan dimension, let us begin by reminding the reader the standard Natarajan dimension. We say that a set  $S = \{x_1, \ldots, x_n\}$  of size  $n$  is Natarajan-shattered by a concept class  $\mathcal{H} \subseteq \mathcal{Y}^{\mathcal{X}}$  if there exist two functions  $f, g: S \to \mathcal{Y}$  so that  $f(x_i) \neq g(x_i)$  for all  $i \in [n]$ , and for all  $b \in \{0, 1\}^n$  there exists  $h \in \mathcal{H}$  such that  $h(x_i) = f(x_i)$  if  $b_i = 1$  and  $h(x_i) = g(x_i)$  if  $b_i = 0$ . Note that here we have equalities instead of inequalities (recall the fat-shattering case Definition 4).
From a geometric perspective (see  $\left[\mathrm{BCD}^{+}22\right]$ ), this means that the space  $\mathcal{H}$  projected on the set  $S$  contains the set  $\{f(x_1), g(x_1)\} \times \ldots \times \{f(x_n), g(x_n)\}$ . This set is "isomorphic" to the Boolean
2Informally, uniform convergence means that for all distributions  $\mathcal{D}$ , with high probability over the sample, the error of all  $h\in \mathcal{H}$  on the sample is close to their true population error.
hypercube of size  $n$  by mapping  $f(x_{i})$  to 1 and  $g(x_{i})$  to 0 for all  $i \in [n]$ . This means that the Natarajan dimension is essentially the size of the largest Boolean cube contained in  $\mathcal{H}$ .
[Sim97] defines the scaled analogue of the above dimension as follows.
Definition 5 ( $\gamma$ -Natarajan Dimension [Sim97]). Let  $\mathcal{H} \subseteq [0,1]^{\mathcal{X}}$ . We say that a set  $S \in \mathcal{X}^n$  is  $\gamma$ -Natarajan-shattered by  $\mathcal{H}$  if there exist  $f, g : [n] \to [0,1]$  such that for every  $i \in [n]$  we have  $\ell(f(i), g(i)) \geq 2\gamma$ , and
$$
\mathcal {H} | _ {S} \supseteq \{f (1), g (1) \} \times \dots \times \{f (n), g (n) \}.
$$
The  $\gamma$ -Natarajan dimension  $\mathbb{D}_{\gamma}^{\mathrm{Nat}}$  is defined to be the maximum size of a  $\gamma$ -Natarajan-shattered set.
Intuitively, one should think of the  $\gamma$ -Natarajan dimension as indicating the size of the largest Boolean cube that is contained in  $\mathcal{H}$ . Essentially, every coordinate  $i \in [n]$  gets its own translation of the 0,1 labels of the Boolean cube, with the requirement that these two labels are at least  $2\gamma$  far from each other. [Sim97] showed the following result, which states that finiteness of the Natarajan dimension at all scales is a necessary condition for realizable PAC regression:
Informal Theorem 1 (Theorem 3.1 in [Sim97]).  $\mathcal{H} \subseteq [0,1]^{\mathcal{X}}$  is PAC learnable in the realizable regression setting only if  $\mathbb{D}_{\gamma}^{\mathrm{Nat}}(\mathcal{H}) < \infty$  for all  $\gamma \in (0,1)$ .
Concluding these two subsections, we have explained the main known general results about realizable offline regression: (i) finiteness of fat-shattering is sufficient but not necessary for PAC learning and (ii) finiteness of scaled Natarajan is necessary for PAC learning.
2.3  $\gamma$ -Graph Dimension
We are now ready to introduce the  $\gamma$ -graph dimension, which is a relaxation of the definition of the  $\gamma$ -Natarajan dimension. To the best of our knowledge, it has not appeared in the literature before. Its definition is inspired by its non-scaled analogue in multiclass classification [Nat89, DSS14].