SlowGuess/NeurIPS_2023_OCR huggingface dataset:
The Gain of Ordering in Online Learning
Mingchen Ma
mingchen@cs.wisc.edu
We study fixed-design online learning where the learner is allowed to choose the order of the datapoints in order to minimize their regret (aka self-directed online learning). We focus on the fundamental task of online linear regression: the learner is given a dataset  $X$  with  $n$  examples in  $d$  dimensions and at step  $t$  they select a point  $x_{t} \in X$ , predict a value  $\widetilde{y}_{t}$ , and suffer loss  $(\widetilde{y}_{t} - w^{*} \cdot x_{t})^{2}$ . The goal is to design algorithms that order the examples and achieve better regret than random-or worst-order online algorithms.
For an arbitrary dataset  $X$ , we show that, under the Exponential Time Hypothesis, no efficient algorithm can approximate the optimal (best-order) regret within a factor of  $d^{1 / \mathrm{poly}(\log \log d)}$ .
We then show that, for structured datasets, we can bypass the above hardness result and achieve nearly optimal regret. When the examples of  $X$  are drawn i.i.d. from the uniform distribution on the sphere, we present an algorithm based on the greedy heuristic of selecting "easiest" examples first that achieves a log  $d$ -approximation of the optimal regret.
In online learning [Rob51, Han57, B+54, CBL06] the learner receives an example and outputs a prediction about its label. The true label of the example is then revealed and the learner suffers loss depending on the "distance" of their prediction from the true label. The goal is to minimize the total loss over all learning rounds given the knowledge of the correct labels of previous rounds. Under worst-case assumptions, where an adversary controls the sequence of examples and labels presented to the learner, a wide range of algorithms based on exponential reweighting [Vov90, LW94, FS97, Vov95, CBL06] and online convex optimization [Haz16, Ora19] have been developed.
Beyond Worst-Case Online Learning A less adversarial setting is fixed-design (aka transductive) online learning first considered in [BDKM97] and subsequently studied in [KK05, CBS13, WHGS22]. In fixed-design online learning the pool of potential examples that the learner is going to face during the learning phase is fixed in advance and known to the learner. There are three main variants of fixed-design online learning considered in the literature (see, [BDKM97]): the worst or adversarial order setting, i.e., when an adversary controls the order of examples presented to the learner, the random order, i.e., when the examples are presented to the learner in a random order, and the best-order or self-directed setting [GS94], where the learner can choose the next example to predict its label at every round. Ordering examples during learning is common in practice: in the context of deep-learning, designing the order of examples when training a model is known as curriculum learning [BLCW09, HW19, WCZ21], where the focus is on finding ways to rank the training examples from "easy" to "hard", as well as using the right pacing function for introducing more difficult data. Another application is direct marketing [LL98, NL11], one common business intelligence task, which is a process of identifying likely buyers to market products accordingly. In particular, an agent must study customers' characteristics and needs, and select customers to market their products. For example, a
streaming service or social media platform may want to learn the content preferences of customers without making too many bad recommendations. In this application, the platform chooses the order in which to present the content (from a pool of available videos) in order to minimize the "regret" (and keep the user engaged). Finally, other non-adversarial online learning variants assume that the order of examples is chosen by a teacher who knows the ground-truth and presents the examples to help the learner [GM93, Mat97, DSZ10, MSSZ22] or make regularity assumptions about the dataset of examples [RS13, JRSS15].
Self-Directed Learning and Gain from Ordering In this work, we focus on the best-order or self-directed version of fixed-design online learning. We first formally define the self-directed online prediction model [GS94] and its random- and worst-order variants that we consider in this work.
Definition 1.1 (Self-Directed Online Learning [GS94]). Let  $f \in \mathcal{C}$  be an unknown target concept from some concept class  $\mathcal{C}$  of functions from  $\mathbb{R}^d$  to  $\mathbb{R}$  and let  $X = \{x^{(1)},\ldots ,x^{(n)}\}$  be a subset of  $n \in \mathbb{N}$  points in  $\mathbb{R}^d$ . The learner has access to the full set of (unlabeled) points  $X$ .
Until the labels of all examples of  $X$  have been predicted:
- The learning algorithm picks a point  $x \in X$  and makes a prediction  $z \in \mathbb{R}$  about its label.
- The true label  $f(x)$  of  $x$  is revealed and the learner suffers loss  $(z - f(x))^2$ .
We say that the learner suffers  $L$  loss (or regret) to label  $X$  if, with probability at least 99%, it holds that the total loss suffered by learner over all rounds is at most  $L$ . In what follows, given a self-directed learner  $\mathcal{A}^{\mathrm{self}}$  we shall denote by  $\mathcal{L}(\mathcal{A}^{\mathrm{self}}, X, f)$  its total loss  $L$ .
In this work, we investigate whether we can design efficient self-directed algorithms that exploit the power of ordering to improve the regret compared to the worst- and random-order settings.
Remark 1.2 (Random-order and Worst-order Online Learning). We shall refer to the setting where the example during the training phase is picked uniformly at random (without replacement) from the unlabeled data  $X$  as random-order learning. Moreover, we shall refer to the setting where the next example is chosen by an adversary as worst-order learning.
Before we continue, we remark that we will also consider "average-case" (Bayesian) settings where the target concept  $f$  is sampled from some prior distribution  $F$  over concepts.
Remark 1.3 (Average and Worst-Case Targets). In Definition 1.1, we will also consider the case where the ground-truth  $f$  is drawn from some distribution over targets  $F$ , in which case the total cost is defined to be the average cost over  $F$ , i.e.,  $\mathcal{L}(\mathcal{A}^{\mathrm{self}}, X, F) \coloneqq \mathbf{E}_{f \sim F} \mathcal{L}(\mathcal{A}^{\mathrm{self}}, X, f)$ . In the worst case setting we will simply write  $\mathcal{L}(\mathcal{A}^{\mathrm{self}}, X) \coloneqq \max_{f \in \mathcal{C}} \mathcal{L}(\mathcal{A}^{\mathrm{self}}, X, f)$ .
To formalize the notion of how much self-directed learners "improve over worst or random settings" in a compact way, we introduce the "Gain from Ordering" (or simply gain) that is defined to be the ratio between the minimum possible total loss of an algorithm that works in the random-order setting and that of an algorithm that can order the examples.
Definition 1.4 (Gain from Ordering). Let  $f \in \mathcal{C}$  be a target concept. Let  $X = \{x^{(1)}, \ldots, x^{(n)}\}$  be a dataset of points in  $\mathbb{R}^d$ . We define the Gain from Ordering of a self-directed learner  $\mathcal{A}^{\mathrm{self}}$  as
$$
\mathcal {G} (\mathcal {A} ^ {\text {s e l f}}, X, F) := \frac {\operatorname* {m i n} _ {\mathcal {A} ^ {\text {r a n d o m}}} \mathcal {L} (\mathcal {A} ^ {\text {r a n d o m}} , X , F)}{\mathcal {L} (\mathcal {A} ^ {\text {s e l f}} , X , F)}.
$$
Moreover, we denote by  $\mathcal{G}^{*}(X,F) := \max_{\mathcal{A}^{\mathrm{self}}} \mathcal{G}(\mathcal{A}^{\mathrm{self}},X,F)$  the maximum possible Gain from Ordering. Similarly, we define the Gain from Ordering in the worst-case (where the target  $f$  is chosen adversarially, see Remark 1.3) by setting  $\mathcal{G}(\mathcal{A}^{\mathrm{self}},X) := \frac{\min_{\mathcal{A}^{\mathrm{random}}} \mathcal{L}(\mathcal{A}^{\mathrm{random}},X)}{\mathcal{L}(\mathcal{A}^{\mathrm{self}},X)}$ , and  $\mathcal{G}^{*}(X) = \max_{\mathcal{A}^{\mathrm{self}}} \mathcal{G}(\mathcal{A}^{\mathrm{self}},X)$ .
We remark that in the above definition, we do not compare against worst-order algorithms as they always perform worse than the best random-order algorithm, and therefore, our results readily generalize when considering the Gain from Ordering with respect to worst-order algorithms.
Whether ordering the examples can improve the total loss suffered by the learner was first studied in the context of online classification, i.e., when the predictions of the algorithm and the ground truth labels are binary  $z$ ,  $f(x) \in \{\pm 1\}$ . For the class of one-dimensional thresholds over the real line, in
[GS94] it was shown that, in the best-order setting, one mistake suffices. On the other hand in the random- and worst-order settings it is known [Lit88, Lit89] that the total loss (or equivalently the number of mistakes) is  $\Theta (\log n)$ , where  $n$  is the size of the dataset. Therefore, for thresholds on the real-line the Gain from Ordering is known to be  $\Theta (\log n)$ . Positive results on the Gain from Ordering exist for other concept classes as well (e.g., for monotone monomials and axis-aligned rectangles) but there are also known concepts where ordering the examples does not help, see [GS94, BDEK95].
Online Linear Regression In this paper, we focus on perhaps the most fundamental problem in online learning, namely online linear regression. An unknown target vector  $w^{*} \in \mathbb{R}^{d}$  is picked by an adversary (or sampled from some prior distribution, see Remark 1.3). The true label of an example  $x$  corresponds to  $f(x) = w^{*} \cdot x$ . In the adversarial/worst-order setting there are numerous results studying online linear regression going back to the seminal work of Widrow and Hoff [WH60, LLW91, KW97, CBLW96, Byl97, AW01, Vov01, OCCB15]. In this setting, the optimal regret is well-understood and tight (even with respect to constant factors) upper and lower bounds exist [AW01, Vov01, CBL06]. The self-directed setting is much less understood both information-theoretically, i.e., what is the best possible Gain from Ordering, and computationally, i.e., whether there exist efficient self-directed algorithms that can achieve the optimal Gain from Ordering. In this work, we aim to make progress in this direction and ask the following fundamental question.
Is there an efficient, self-directed learning algorithm for linear regression that achieves (approximately) optimal Gain from Ordering?
Our first result is an impossibility result showing that for unstructured datasets it is computationally intractable to compute an ordering that approximates the optimal Gain from Ordering under the Exponential Time Hypothesis (ETH). More precisely, we show that, even for the "easier" version of the problem where the target vector is drawn uniformly at random from the unit sphere, no efficient self-directed learner can achieve better than  $d^{1 / \mathrm{poly}(\log \log d)}$ -approximation of the optimal Gain from Ordering.
Theorem 1.5 (Hardness of Approximation for the Optimal Gain from Ordering). Let  $X$  be an arbitrary set of  $n$  unit-norm examples and denote by  $\mathbb{S}^d$  the uniform distribution over the unit sphere. Under the Exponential Time Hypothesis (ETH), there is no polynomial time self-directed learner  $\mathcal{A}^{\mathrm{self}}$  such that  $\mathcal{G}(\mathcal{A}^{\mathrm{self}}, X, \mathbb{S}^d) \geq d^{-1 / \log \log^{c_d}} \mathcal{G}^*(X, \mathbb{S}^d)$ , where  $c > 0$  is some universal constant.
Our hardness result follows by a reduction to the Densest  $k$ -Subgraph (DkS) problem (see Definition 3.1) that was shown in [Man17] to be ETH-hard to approximate. To go from our learning problem to the combinatorial DkS problem we perform a sequence of approximation preserving reductions. At a high-level, we first show that selecting the best order for online linear regression is equivalent to an offline geometric problem where we need to sort the examples so that the sum of the distances of every example from the subspace spanned by the previous examples in the ordering is minimized, see Definition 3.3). We then show that this problem can be further reduced to a specific edge packing problem on graphs (see Definition 3.7) by only worsening the achieved approximation guarantee by a factor of 2. Finally, we show that any  $\alpha$ -approximate algorithm for the  $k$ -packing problem yields an  $O(\alpha^2)$ -approximate algorithm for the Densest  $k$ -Subgraph problem. For more details, we refer to Section 3.
As our hardness result suggests, without any assumption over the structure of the data, there is no hope to design an efficient self-directed learner with "good" approximation guarantee - especially in high-dimensional settings, i.e., when  $d$  is large. This motivates us to study self-directed learning over structured datasets. We focus on the fundamental setting where the dataset  $X$  is i.i.d. drawn uniformly from the unit sphere,  $S^d$ . In this case, we design an algorithm based on a greedy heuristic that picks the example that is "most similar" to the examples already seen achieves a log  $d$ -approximation of the optimal Gain from Ordering.
Theorem 1.6 (Efficient Self-Directed Learner under Spherical Data). Let  $X$  be a set of  $n$  examples drawn i.i.d. from  $\mathbb{S}^d$ . There exists an efficient self-directed learner  $\mathcal{A}$  such that, with probability at least  $99\%$ , it holds  $\mathcal{G}(\mathcal{A}, X, \mathbb{S}^d) \geq \Omega(1 / \log d) \mathcal{G}^*(X, \mathbb{S}^d)$ . Moreover, it holds that  $\mathcal{G}(\mathcal{A}, X, \mathbb{S}^d) \geq \min(\Omega(d), \Omega(n^{2/d}))$ .
We remark that, apart from presenting an efficient algorithm for approximating the Gain of Ordering, Theorem 1.6, gives the first information-theoretic bound for the optimal gain under spherical data, showing that  $\mathcal{G}^* (X,\mathbb{S}^d)$  is roughly  $n^{2 / d}$ . At a high level, the fact that the optimal gain increases slower as the dimension increases is explained by the fact that,  $n$  samples from the uniform distribution on the sphere will be almost orthogonal (unless  $n$  is very large). Therefore, observing the labels of a subset of them reveals little information about the remaining points. See Section 4 for more details.
Apart from showing that, under structured datasets, significantly improved approximation guarantees can be achieved our upper bound of Theorem 1.6 serves as a formalization of the popular greedy heuristic of picking the "easiest examples first" used in curriculum learning ([BLCW09]) for linear regression. In particular, assuming that we have already observed the labels of the points  $x^{(1)}, \ldots, x^{(i)}$  we pick the example that is closest to the subspace spanned by  $x^{(1)}, \ldots, x^{(i)}$ , see Algorithm 1.
As an extension, we also study the self-directed ReLU regression problem, where the concept class is  $\mathcal{C} = \{\mathrm{Relu}(w^{*}\cdot x)\mid w^{*}\in \mathbb{R}^{d}\}$ . ReLU regression is a very popular non-convex optimization task that has recently received significant attention both in online and offline settings, [KS09, Sol17, GKKT17, GKK, DGK+20, DKKZ20, DKZ20, FCG20, DKTZ22, DKR23], due to the fact that ReLU is a very common activation in deep learning models. In Section 5, we present an efficient algorithm that achieves Gain of Ordering roughly  $\min (d,n^{2 / d} / \log d + \log n / \log d)$ , see Theorem 5.1, assuming a "warm-start" labeled example. We observe that, even though ReLU regression is typically a harder task than linear regression, the Gain of Ordering that we achieve is larger (by a term of  $\log n / \log d$ ) than that of linear regression. At a high-level this interesting phenomenon has to do with the the ReLU being constant on a large region of the space: a fact that the self-directed learner can exploit to improve its gain. It is an interesting direction for future research to further investigate the properties of the activation and the dataset that allow for improved gains by self directed learners.
Related to the setting of self-directed learning is active learning [CAL94], where the learner has access to a large pool of unlabeled examples and chooses the "most informative" to ask for their labels. The goal is to find a classifier with good generalization while minimizing the number of label queries. There is a long line of research on active linear classification in the distribution-specific setting (e.g., under the uniform distribution on the unit sphere) [DKM05, Han11, BU16]. We remark that our goal of minimizing the number of mistakes is orthogonal to that of active learning: at a high-level, our algorithms pick the examples for which the current hypothesis is most confident ("easiest examples") while in active learning one typically asks for the labels of the "hardest examples", e.g., those with the smallest margin with respect to the current guess (see, e.g., [ABHU15, ABHZ16, ZSA20]).
In deep learning, stochastic gradient descent typically trains models by considering the examples in a random order. In the influential work of [BLCW09] the authors proposed curriculum learning: training machine learning models in a "meaningful order" – from easy examples to harder ones. There is a long line of research (see the surveys [HW19, WCZ21, SIRS22] and references therein) giving empirical evidence that curriculum learning provides significant benefits in convergence speed and generalization over training with random order. Our results provide theoretical evidence that ordering the examples from easier to harder significantly reduces the mistakes made by the learner.
There has been a long line of work studying online linear regression. We first review the results in the realizable setting, which means  $y(x) = w^{*} \cdot x$ . In this case, the problem is also called the adaptive filtering problem by [KWH06]. Online gradient descent was shown to be a minimax optimal method by [KWH06, CBLW96]. These works show if  $\| x \| \leq B$  and  $\| w^{*} \| \leq W$ , then online gradient descent achieves a regret bound  $O(B^{2}W^{2})$ . A matching lower bound  $\Omega(B^{2}W^{2})$  was given by [CBLW96] using a dataset that contains only one example. In the adversarial setting, online gradient descent and exponentiated gradient were studied by [CBLW96, KW97]. Given the knowledge of  $B$ , by suitable tuning the learning rate, the two algorithms achieve a regret bound  $O(L(w^{*}))$ , which grows linearly with  $T$  in the worst case. Furthermore, if the algorithms know an error bound  $E$  for  $L(w^{*})$  and a bound  $W$  for  $\| w^{*} \|$ , then these algorithms can achieve a regret bound  $O(\sqrt{E})$ , which is  $O(\sqrt{T})$  in the worst case. These regret bounds cannot be further improved for such types of algorithms. On the other hand, assuming  $y(x) \leq Y$  and  $\| x \| \leq B$  for any  $x \in X$ , [Vov01, AW01] obtained a regret bound of  $O\left(\| w^{*} \|^{2} + dY^{2}\log (TX^{2} / d)\right)$  using a so-called online nonlinear ridge regression method.
A matching lower bound of  $\Omega (\| w^{*}\|^{2} + Y^{2}d\log T)$  were given by [CL06, Vov01]. Recently better lower bound of  $\Omega (dY^2\log T)$  were given by [BKM  $+15$  , GGHS19]. Another interesting setting is the stochastic setting studied by [OMP21], where  $y(x) = w^{*}\cdot x + \xi$ $\xi$  is a zero-mean sub-gaussian noise with variance  $\sigma^2$  . In particular, in such a setting, the label  $y$  can be unbounded. Ouhamma et al. showed that the online (nonlinear) ridge regression with high probability has a regret bound of  $O(d\sigma^2\log T\log \log T)$
In this section, we introduce the notations we will use in the paper. Let  $X \subseteq \mathbb{R}^d$  be the set of  $n$  examples. Denote by  $w^{*} \in \mathbb{R}^{d}$  the target vector that labels each  $x \in X$  by  $y(x) = w^{*} \cdot x$ . Let  $\mathcal{A}$  be a self-directed learner. For  $i \in [n]$ , we use random vector  $x^{(i)} \in \mathbb{R}^d$  to denote the example in  $X$  that is selected by  $\mathcal{A}$  in the  $i$ -th round. We denote by  $L_{i}$  be subspace spanned by examples  $x^{(1)}, \ldots, x^{(i)}$  and  $L_{i}^{\perp}$  the subspace that is orthogonal to  $L_{i}$ . For every  $v \in \mathbb{R}^d$  and for every subspace  $L \subseteq \mathbb{R}^d$ , we denote by  $v_{L} = \mathrm{proj}_{L}(v)$ , the projection of  $v$  onto  $L$ . Furthermore, we will use  $S^d$  to denote the unit sphere in  $\mathbb{R}^d$  and use  $\mathbb{S}^d$  to denote the uniform distribution over  $S^d$
3 Optimal Gain from Ordering under Arbitrary Data is Hard
In this section, we show that self-directed linear regression, in general, is hard to approximate, even when  $X \subseteq S^d$  and  $w^{*}$  is drawn uniformly from  $S^d$ .
In the rest of this section, we will give a high-level overview of the proof of Theorem 1.5. The full proof of results in this section can be found in Appendix A. The key idea is that if we have a good efficient self-directed learner, then we can obtain an efficient algorithm that approximately solves the  $k$ -densest subgraph problem (DkS), which has been shown hard to approximate by [Man17].
Definition 3.1 (Densest  $k$ -Subgraph Problem (DkS)). Let  $G = (V, E)$  be an undirected graph with  $n$  vertices and  $m$  edges, and let  $k \in [n]$ . The goal is to find a subset of  $k$  vertices  $S$  such that the edge density  $\rho(S) := |E(S)| / \binom{|S|}{2}$  is maximized, where  $E(S)$  denotes the set of all edges among the vertices in  $S$ . We define opt to be the maximum density over all possible subsets of  $k$  vertices. Given  $G$  and  $k$ , an  $\alpha$ -approximate algorithm for DkS problem outputs a subset of  $k$  vertices  $S$  such that  $\alpha \rho(S) \geq \text{opt in polynomial time}$ .
The Sequential Spanning Problem (SSP) To begin with, we observe that every self-directed learner has two parts, selecting examples and making predictions. The first observation, which is stated as Lemma 3.2, is that with a prior distribution of  $w^{*}$ , it is easy to obtain a Bayesian optimal prediction in each round.
Lemma 3.2. Let  $X = \{x^{(1)},\ldots ,x^{(n)}\} \subseteq \mathbb{R}^d$  be a set of  $n$  examples and  $w^{*}$  be a target vector drawn uniformly from  $S^d$  that labels  $y^{(i)} = w^{*}\cdot x^{(i)}$  for each  $x^{(i)}\in X$ . Given given any set of labeled examples  $(x^{(1)},y^{(1)}),\dots ,(x^{(i - 1)},y^{(i - 1)})$ , denote by  $L_{i - 1}$  the subspace spanned by  $x^{(1)},\ldots ,x^{(i - 1)}$  and  $w_{L_{i - 1}}^{*}$  the projection of  $w^{*}$  onto  $L_{i - 1}$ . Let  $\mathcal{A}$  be a self-directed learner, denote by  $\hat{y}^{(i)}$  be the prediction of  $\mathcal{A}$  for the next example  $x^{(i)}$ , then we have  $\mathbf{E}_{w^*}\left((\hat{y}^{(i)} - y^{(i)})^2\mid w_{L_{i - 1}}^*\right)\geq \mathbf{E}_{w^*}\left((w_{L_{i - 1}^\perp}^*\cdot x^{(i)})^2\mid w_{L_{i - 1}}^*\right)$ . Furthermore, the inequality holds for equality if  $\hat{y}^{(i)} = w_{L_{i - 1}}^*\cdot x^{(i)}$ .
With the Bayesian optimal prediction, the problem becomes how to select a good ordering. We show in Proposition 3.4, the problem can be equivalent to formulate as the following Sequential Spanning Problem.
Definition 3.3 ( $k$ -Sequential Spanning Problem ( $k$ -SSP)). Let  $X = \{x^{(1)}, \ldots, x^{(m)}\}$  on  $\mathbb{R}^d$  be a set of  $m$  points with unit norm and let  $k \in \mathbb{N}$ . Consider an ordered sequence  $\sigma$  of  $k$  points of  $X$ , i.e.,  $\sigma = x^{(i_1)}, x^{(i_2)}, \ldots, x^{(i_k)}$  and define  $L_j$  to be the sub-space spanned by the first  $j$ -elements of  $\sigma$  i.e.,  $L_j = \operatorname{span}(x^{(i_1)}, \ldots, x^{(i_j)}) = \operatorname{span}(x^{(\sigma(1))}, \ldots, x^{(\sigma(j))})$  and  $V_0 = \emptyset$ . Define the following cost, called spanning cost,
$$
C (\sigma , k) = \sum_ {i = 1} ^ {k} \| \operatorname {p r o j} _ {L _ {i - 1} ^ {\perp}} x ^ {(\sigma (i))} \| _ {2} ^ {2}.
$$
We define opt to be the minimum cost over all sequences of  $k$ -elements of  $X$ . Given  $X$  and  $k$ , an  $\alpha$ -approximate algorithm for  $k$ -SSP Problem outputs an ordered sequence of  $k$  points from  $X$  so that  $C(\sigma, k) \leq \alpha$  opt in polynomial time.