SlowGuess/NeurIPS_2023_OCR huggingface dataset:
In the online learning with experts problem, an algorithm makes predictions about an outcome on each of  $T$  days, given a set of  $n$  experts who make predictions on each day. The algorithm is given feedback on the outcomes of each day, including the cost of its prediction and the cost of the expert predictions, and the goal is to make a prediction with the minimum cost, compared to the best expert in hindsight. However, often the predictions made by experts or algorithms at some time influence future outcomes, so that the input is adaptively generated.
In this paper, we study robust algorithms for the experts problem under memory constraints. We first give a randomized algorithm that is robust to adaptive inputs that uses  $\widetilde{O}\left(\frac{n}{R\sqrt{T}}\right)$  space for regret  $R$  when the best expert makes  $M = O\left(\frac{R^2T}{\log^2n}\right)$  mistakes, thereby showing a smooth space-regret trade-off. We then show a space lower bound of  $\widetilde{\Omega}\left(\frac{nM}{RT}\right)$  for any randomized algorithm that achieves regret  $R$  with probability  $1 - 2^{-\Omega (T)}$ . Such an algorithm is useful for adaptive inputs, as the failure probability is low enough to union bound over all computation paths. Our result implies that the natural deterministic algorithm, which iterates through pools of experts until each expert in the pool has erred, is optimal up to polylogarithmic factors. Finally, we empirically demonstrate the benefit of using robust procedures against a white-box adversary that has access to the internal state of the algorithm.
Online learning with experts is a fundamental problem in sequential prediction. On each of  $T$  days, an algorithm must make a prediction about an outcome, given a set of  $n$  experts who make predictions on the outcome. The algorithm is then given feedback on the cost of its prediction and on the expert predictions for the current day. In the discrete prediction with experts problem, the set of possible predictions is restricted to a finite set, and the cost is 0 if the prediction is correct, and 1 otherwise. More generally, we assume the costs are restricted to be in a range  $[0, \rho]$  for some fixed parameter  $\rho > 0$ , with lower costs indicating better performance. This process continues for the  $T$  days, after which the performance (total cost) of the algorithm is compared to the performance (total cost) of the best performing expert. In particular, the goal for the online learning with experts problem is to minimize the regret, which is the amortized difference between the total cost of the algorithm and the total cost of the best performing expert, i.e., the expert that incurs the least overall cost.
A well-known folklore algorithm for handling the discrete prediction with experts problem is the weighted majority algorithm [42]. The deterministic variant of the weighted majority algorithm simply initializes "weights" for all experts to 1, down-weights any incorrect expert on a given day, and selects the prediction supported by the largest weight of experts. The algorithm solves the discrete prediction with experts problem with  $O(M + \log n)$  total mistakes, where  $M$  is the number of mistakes made by the best expert, thus achieving total regret  $O(M + \log n)$ . More generally, a large
body of literature has studied optimizations to the weighted majority algorithm, such as a randomized variant where the probability of the algorithm selecting each prediction is proportional to the sum of the weights of the experts supporting the prediction. The randomized weighted majority algorithm achieves regret  $O\left(\sqrt{\log n / T}\right)$  [42], which has been shown to be information-theoretically optimal, up to a constant. There have subsequently been many follow-ups to the weighted and randomized weighted majority algorithms that achieve similar regret bounds, but improve in other areas. For example, on a variety of structured problems, such as online shortest paths, follow the perturbed leader [38] achieves the same regret bound as randomized weighted majority but uses less runtime on each day. In addition, the multiplicative weights algorithm achieves the optimal  $\sqrt{\ln n / (2T)}$  regret, with a tight leading constant [33]. However, these classic algorithms use a framework that maintains the cumulative cost of each expert, which requires the algorithm to store  $\Omega(n)$  bits of information.
Memory bounds. Recently, [49] considered the online learning with experts problem when memory is a premium for the algorithm. On the hardness side, they showed that any algorithm achieving a target average regret  $R$  requires  $\Omega \left( \frac{n}{R^2 T} \right)$  space, which implies that any algorithm achieving the information-theoretic  $O\left(\sqrt{\log n / T}\right)$  regret must use near-linear space. On the other hand, when the number of mistakes  $M$  made by the best expert is small, i.e.,  $M = O\left(R^{2}T\right)$ , [49] gave a randomized algorithm that uses  $\widetilde{O}\left(\frac{n}{RT}\right)$  space for arbitrary-order streams, thus showing that the hardness of their lower bound originates from a setting where the best expert makes a large number of mistakes.
Subsequently, [47] considered the online learning with experts problem when the algorithm is limited to memory sublinear in  $n$ . They introduced a general framework that achieves  $o(T)$  regret using  $o(n)$  memory, with a trade-off parameter between space and regret that obtains  $O_{n}\left(T^{4 / 5}\right)$  regret with  $O(\sqrt{n})$  space and  $O_{n}\left(T^{0.67}\right)$  regret with  $O\left(n^{0.99}\right)$  space.
Adaptive inputs. Up to now, the discussion has focused on an oblivious setting, where the input to the algorithm may be worst-case, but is chosen independently of the algorithm and its outputs. The online learning with experts problem is often considered in the adaptive setting, where the input to the algorithm is allowed to depend on previous outputs by the algorithm, e.g., in financial markets, future stock quotes can depend on previous investment choices. Formally, we define the adaptive setting as a two-player game between an algorithm  $\mathcal{D}$  and an adversary  $\mathcal{A}$  that adaptively creates the input stream to  $\mathcal{D}$ . The game then proceeds in days and on the  $t$ -th day:
(1) The adversary  $\mathcal{A}$  chooses the outputs of all experts on day  $t$  as well as the outcome of day  $t$ , depending on all previous stream updates and all previous outputs from the algorithm  $\mathcal{D}$ .
(2) The outputs (i.e., predictions) of all experts are simultaneously given to the algorithm  $\mathcal{D}$ , which updates its data structures, acquires a fresh batch  $R_{t}$  of random bits, and outputs a predicted outcome for day  $t$ .
(3) The outcome of day  $t$  is revealed to  $\mathcal{D}$ , while the predicted outcome for day  $t$  by  $\mathcal{D}$  is revealed to the adversary  $\mathcal{A}$ .
The goal of  $\mathcal{A}$  is to induce  $\mathcal{D}$  to make as many incorrect predictions as possible throughout the stream. It is clear that any deterministic algorithm for the online learning with experts problem will maintain the same guarantees in the adaptive model. Unfortunately, both the algorithms of [49] and [47] are randomized procedures that rely on iteratively sampling "pools" of experts, which can potentially be exploited by an adaptive adversary who learns the experts sampled in each pool. Interestingly, both the randomized weighted majority algorithm [42] and the multiplicative weights algorithm [33] are known to be robust to adaptive inputs.
In this paper, we study the capabilities and limits of sublinear space algorithms for the online learning with experts problem on adaptive inputs.
Robust algorithms. Towards adaptive robustness, it is natural to study deterministic algorithms, since they retain the same guarantee under adaptive adversaries. As a warm-up, we first provide a simple deterministic algorithm that uses space  $\widetilde{O}\left(\frac{nM}{RT}\right)$ . Consider an algorithm that iteratively
selects the next pool of  $k = \widetilde{O}\left(\frac{nM}{RT}\right)$  experts and running the deterministic majority algorithm on the experts in the pool, while removing any incorrect experts from the pool until the pool is completely depleted, at which point the next pool of  $\widetilde{O}\left(\frac{nM}{RT}\right)$  experts is selected. The main intuition is that each pool can incur at most  $O(\log n)$  mistakes before it is depleted and the best expert can only make  $M$  mistakes. By the time the pool has cycled through  $nM$  experts, i.e.,  $M$  times for each of the  $n$  experts, then the best expert no longer makes any mistakes and will be retained by the pool. Thus, the total number of mistakes made by the algorithm is  $\frac{nM}{k} \cdot O(\log n)$ . On the other hand, for a target average regret  $R$ , the mistake bound of the algorithm is required to be at most  $M + RT$ , so it suffices to set  $k = \widetilde{O}\left(\frac{nM}{RT}\right)$  to achieve regret  $R$ . Since the algorithm runs deterministic majority on a pool of  $k = \widetilde{O}\left(\frac{nM}{RT}\right)$  experts, then this algorithm uses  $\widetilde{O}\left(\frac{nM}{RT}\right)$  space. Formally, we show:
Theorem 1.1 (Simple deterministic algorithm; see Section 3.1). Suppose the best expert makes  $M$  mistakes and let  $R \geq \frac{4M\log n}{T}$ . There exists a deterministic algorithm (Algorithm 2) that uses space  $\widetilde{O}\left(\frac{nM}{RT}\right)$  and achieves an average regret of  $R$ .
The algorithm is simple, computationally efficient, and easy to implement. However, the drawback is that for  $M = \Omega(RT)$ , the algorithm requires space near-linear in the number of experts  $n$ , which is undesirable when  $n$  is large. To address this issue, we complement our deterministic algorithm with a randomized algorithm that is robust to adaptive inputs and allows for a different memory-regret trade-off:
Theorem 1.2 (Robust randomized algorithm). Let  $R > \frac{64\log^2n}{T}$ , and suppose the best expert makes at most  $M \leq \frac{R^2T}{128\log^2n}$  mistakes. Then there exists an algorithm for the discrete prediction with experts problem that uses  $\widetilde{O}\left(\frac{n}{R\sqrt{T}}\right)$  space and achieves regret at most  $R$ , with high probability.
This gives a trade-off between the space and regret, almost all the way to the information-theoretic limit of  $R = O_{n}\left(\sqrt{1 / T}\right)$  for general worst-case input. However, it incurs a multiplicative space overhead of  $\widetilde{O} (\sqrt{T})$  compared to the optimal algorithms for oblivious input. Thus we believe the complete characterization of the space complexity of the discrete prediction with experts problem with adaptive input is a natural open question resulting from our work.
Tight memory bounds for robust algorithms. It is natural to ask whether there exist robust algorithms that are more space-efficient than the straightforward deterministic approach. For example, [12] showed that any oblivious randomized algorithm with failure probability  $2^{-\Omega(nT)}$  will be robust against adaptive outputs in the discrete prediction with experts problem, so a reasonable approach would be to boost the success probability of existing oblivious algorithms to  $1 - 2^{-\Omega(nT)}$ . Unfortunately, we show this cannot work:
Theorem 1.3 (Memory lower bound for high-probability algorithms). For  $n = o(2^T)$ , any randomized algorithm algorithm that achieves  $R$  regret with probability at least  $1 - 2^{-\Omega(T)}$  for the discrete prediction with experts problem must use  $\Omega\left(\frac{nM}{RT}\right)$  space when the best expert makes  $M$  mistakes.
In particular, Theorem 1.3 shows that any deterministic algorithm must use  $\Omega\left(\frac{nM}{RT}\right)$  space, which taken together with the deterministic procedure above, resolves the deterministic streaming complexity of online learning with experts. We emphasize that Theorem 1.3 also shows that using the strategy of high-probability randomized algorithms to guarantee robustness against adaptive input does not work any better than a deterministic algorithm.
At a conceptual level, our lower bound in Theorem 1.3 shows that surprisingly, the number  $M$  of the mistakes made by the best expert is an intrinsic parameter that governs the abilities and limitations of robust algorithms in this model. Thus, even though  $M$  is not a parameter that may naturally be ascertained in practice, it nevertheless completely characterizes the complexity of the problem. On the other hand, for algorithmic purposes, it suffices to acquire a constant-factor approximation to  $M$  as an input to the algorithm.
Another reason Theorem 1.3 is somewhat surprising is because as the number of mistakes  $M$  made by the best expert increases, then the algorithm is also permitted to make more mistakes and in some sense, the problem seems "easier". However, Theorem 1.3 shows this intuition is not true—the problem actually becomes more difficult as  $M$  increases.
Moreover, we give an alternative proof in the regime when  $M = \Omega (T)$ . The proof differs from the proof of Theorem 1.3. Instead, it leverages the communication complexity of a new set disjointness problem, recently proposed by [39]. The statement is technically weaker Theorem 1.3 and appears in the appendix; see Appendix E.
Empirical evaluations. Finally, we conduct experimental evaluations in Section 5 by comparing the natural deterministic algorithm to the randomized algorithm of [49] against a white-box adversary who has access to the internal state of the algorithm, including any experts sampled and maintained by the algorithm. The deterministic algorithm iteratively selects pools of  $k = \widetilde{O}\left(\frac{nM}{RT}\right)$  experts, discarding any expert that has erred, and refreshing the pool with the next batch of  $k$  experts once the pool is emptied. The randomized algorithm similarly discards erroneous experts from a pool of  $k$  experts, but it repeatedly samples pools of  $k$  experts rather than selecting the next pool of  $k$  experts. On average across the multiple trials for each setting, the randomized algorithm made several times more mistakes than the deterministic algorithm, ranging from 1.98x times more mistakes to 3.29x times more mistakes than the deterministic algorithm, thus demonstrating the importance of robust algorithms against adversarial inputs.
The experts problem and memory bounds. The experts problem has been extensively studied [17], both in the discrete decision setting [42] and in the setting where costs are determined by various loss functions [35, 52-55]. Hence, the experts problem can be applied to many different applications, such as portfolio optimization [24, 23], ensemble boosting [32], and forecasting [37]. Given certain assumptions on the expert, such as assuming the experts are decisions trees [36, 50], threshold functions [43], or have nice linear structures [38], additional optimizations have been made to improve the algorithmic runtimes for the experts problem and more generally, existing work has largely ignored optimizing for memory constraints in favor of focusing on time complexity or regret guarantees, thus frequently using  $\Omega(n)$  memory to track the performance of each expert.
Recently, [49] introduced the study of memory-regret trade-offs for the experts problem. For  $n \gg T$ , [49] showed that the space complexity of the problem is  $\tilde{\Theta}\left(\frac{n}{R^2T}\right)$  in the random-order streams, but also gave a randomized algorithm that uses  $\tilde{O}\left(\frac{n}{RT}\right)$  space for arbitrary-order streams when the number of mistakes  $M$  made by the best expert is "small". Subsequently, [47] considered the online learning with experts problem for  $T \gg n$ , introducing a general space-regret trade-off framework that achieves  $o(T)$  regret using  $o(n)$  memory, including  $O_{n}(T^{4/5})$  regret with  $O(\sqrt{n})$  space and  $O_{n}(T^{0.67})$  regret with  $O(n^{0.99})$  space.
Concurrent and independent work. Concurrent to our work, [46] considered a variant of the problem where at each time, the algorithm selects an expert instead of a prediction. They then introduce an algorithm robust against an adaptive adversary who observes the specific expert chosen by the algorithm at each time, as well as lower bounds for any algorithm robust to such an adversary.
One way to ensure adversarial robustness is through deterministic algorithms. On that end, we achieve stronger lower bounds for deterministic algorithms, showing that there must be a dependency on the number  $M$  of mistakes made by the best expert, i.e., any deterministic algorithm achieving amortized regret  $R$  must use  $\widetilde{\Omega}\left(\frac{nM}{RT}\right)$  space. In fact, when the number of mistakes  $M$  made by the best expert is sufficiently small, i.e.,  $M = O\left(\frac{R^2T}{\log^2n}\right)$  for amortized regret  $R$ , we give a randomized upper bound that uses less space than this lower bound. By comparison, the lower bound of [46] shows that any algorithm achieving  $R$  amortized regret must use  $\widetilde{\Omega}\left(\sqrt{\frac{n}{R}}\right)$  space, though their lower bound also applies to randomized algorithms.
Due to the difference in setting, our algorithmic techniques are quite different from those of [46]. We use a recent idea of [34, 4, 10] to hide the internal randomness of our algorithm from the adversary whereas [46] rotates between groups of experts to prevent an adversary from inducing high regret by making a specific expert bad immediately after it is selected.
For any  $t \leq n$  and vector  $(X_1, X_2, \dots, X_n)$ , we let  $X_{<t}$  denote  $(X_1, \dots, X_{t-1})$ ,  $X_{\leq t} = (X_1, \dots, X_t)$ , and  $X_{-t} = (X_1, \dots, X_{t-1}, X_{t+1}, \dots, X_n)$ . Also,  $X_{>t}$  and  $X_{\geq t}$  are defined similarly. Let  $e_i$  denote the  $i$ th standard basis vector, and for any  $S$ ,  $e_S$  the vector that has a 1 at index  $i \in S$  and 0 everywhere else. For a random variable  $X$ , let  $H(X)$  denote its entropy.
We write  $[n]$  for an integer  $n > 0$  to denote the set  $\{1, \dots, n\}$ . We write  $\mathrm{poly}(n)$  to denote a fixed polynomial in  $n$ . If an event occurs with probability at least  $1 - \frac{1}{\mathrm{poly}(n,T)}$ , we say the event occurs with high probability. We give additional technical preliminaries in Appendix B.
Formal problem statement. In the online learning with experts problem, there are  $n$  experts that each make predictions on each of  $T$  days. The prediction are in  $\{0, 1\}$ . An algorithm uses the experts to output a prediction for each day  $t \in [T]$ . The actual outcome of the day  $t$  is then revealed, at which point the algorithm is penalized with a cost that is 0 if the prediction is correct, and 1 otherwise.
This process continues for the  $T$  days. At the end, suppose that the best expert has incurred cost  $M$ , while the algorithm has incurred  $C$ . Then the performance of the algorithm is measured by the (average) regret  $R = \max \left(\frac{C - M}{T}, 0\right)$ .
Differential privacy. We use tools from differential privacy.
Definition 2.1 (Differential privacy, [30]). Given a privacy parameter  $\varepsilon > 0$  and a failure parameter  $\delta \in (0,1)$ , a randomized algorithm  $\mathcal{A}:\mathcal{X}^*\to \mathcal{Y}$  is  $(\varepsilon ,\delta)$ -differentially private if, for every pair of neighboring streams  $S$  and  $S^{\prime}$  and for all  $E\subseteq \mathcal{V}$ ,
$$
\Pr \left[ \mathcal {A} (S) \in E \right] \leq e ^ {\varepsilon} \cdot \Pr \left[ \mathcal {A} \left(S ^ {\prime}\right) \in E \right] + \delta .
$$
Theorem 2.2 (Private median, e.g., [34]). Given a database  $\mathcal{D} \in X^*$ , a privacy parameter  $\varepsilon > 0$  and a failure parameter  $\delta \in (0,1)$ , there exists an  $(\varepsilon,0)$ -differentially private algorithm PRIVMED that outputs an element  $x \in X$  such that with probability at least  $1 - \delta$ , there are at least  $\frac{|S|}{2} - m$  elements in  $S$  that are at least  $x$ , and at least  $\frac{|S|}{2} - m$  elements in  $S$  that are at most  $x$ , for  $m = O\left(\frac{1}{\varepsilon} \log \frac{|X|}{\delta}\right)$ .
Theorem 2.3 (Advanced composition, e.g., [31]). Let  $\varepsilon, \delta' \in (0,1]$  and let  $\delta \in [0,1]$ . Any mechanism that permits  $k$  adaptive interactions with mechanisms that preserve  $(\varepsilon, \delta)$ -differential privacy guarantees  $(\varepsilon', k\delta + \delta')$ -differential privacy, where  $\varepsilon' = \sqrt{2k\ln\frac{1}{\delta'}} \cdot \varepsilon + 2k\varepsilon^2$ .
Theorem 2.4 (Generalization of DP, e.g., [29, 9]). Let  $\varepsilon \in (0,1/3)$ ,  $\delta \in (0,\varepsilon/4)$ , and  $n \geq \frac{1}{\varepsilon^2} \log \frac{2\varepsilon}{\delta}$ . Suppose  $\mathcal{A}: X^n \to 2^X$  is an  $(\varepsilon, \delta)$ -differentially private algorithm that curates a database of size  $n$  and produces a function  $h: X \to \{0,1\}$ . Suppose  $\mathcal{D}$  is a distribution over  $X$  and  $S$  is a set of  $n$  elements drawn independently and identically distributed from  $\mathcal{D}$ . Then
$$
\operatorname * {P r} _ {S \sim \mathcal {D}, h \leftarrow \mathcal {A} (S)} \left[ \left| \frac {1}{| S |} \sum_ {x \in S} h (x) - \underset {x \sim \mathcal {D}} {\mathbb {E}} [ h (x) ] \right| \geq 1 0 \varepsilon \right] <   \frac {\delta}{\varepsilon}.
$$
3 Algorithms Against Adaptive Adversaries
In this section, we show that there exists algorithms for the discrete prediction with experts problem that is robust to adaptive outputs.
3.1 A Near-Optimal Deterministic Algorithm
We first present a simple deterministic algorithm for arbitrary-order streams. The algorithm repeatedly selects pools of the next  $k = \widetilde{O}\left(\frac{nM}{RT}\right)$  experts. While the pool is non-empty, the algorithm runs the deterministic majority algorithm on the algorithm and removes any incorrect experts from the pool. Once the pool is empty, the next  $\widetilde{O}\left(\frac{nM}{RT}\right)$  experts are added to the pool, possibly cycling through all  $n$  experts multiple times if necessary, where an expert can be added to the pool again even if it has been previously deleted from the pool. We give the formal algorithm and analysis in Appendix C.
Theorem 3.1 (Deterministic algorithm). Among  $n$  experts in a stream of length  $T$ , suppose the best expert makes  $M$  mistakes and let  $R \geq \frac{4M\log n}{T}$ . There exists a deterministic algorithm (Algorithm 2) that uses space  $\widetilde{O}\left(\frac{nM}{RT}\right)$  and achieves an average regret of  $R$ .
In light of lower bound Theorem 1.3, it is evident that Theorem 3.1 is nearly optimal, up to polylogarithmic factors, for deterministic algorithms, which are automatically adversarially robust. On the other hand, it does not seem necessary that any adversarially robust algorithm must be deterministic. Indeed, we now give a randomized adversarially robust algorithm with better space guarantees.
3.2 A Randomized Robust Streaming Algorithm
We first recall the following randomized algorithm for arbitrary-order streams with oblivious input, i.e., non-adaptive input:
Lemma 3.2 (Algorithm for oblivious inputs; [49]). Let  $R > \sqrt{\frac{128\log^2n}{T}}$ , and suppose the best expert makes at most  $M \leq \frac{R^2T}{1280\log^2n}$  mistakes. Then there exists an algorithm DISCPRED for the discrete prediction with experts problem that uses  $\widetilde{O}\left(\frac{n}{RT}\right)$  space and achieves regret at most  $R$ , with high probability, i.e., probability at least  $1 - \frac{1}{\mathrm{poly}(n,T)}$ .