SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Algorithm 2 Deterministic algorithm for the experts problem
Input: A stream of length  $T$  with  $n$  experts, upper bound  $M$  on the number of mistakes made by the best expert, and target regret  $R$   
Output: A sequence of predictions with regret  $R$   
1:  $k \gets \frac{4nM}{RT} \log n$   
2:  $S \gets \emptyset$   
3: while the stream persists do  
4: if  $S$  is empty then  
5:  $S \gets [n]$   
6: Let  $P$  be the first  $k$  indices of  $S$   
7:  $S \gets S \setminus P$   
8: while  $P \neq \emptyset$  do  
9: For each following day, choose the outcome output by the majority of the experts in  $P$   
10: Delete the incorrect experts on that day from  $P$
We now prove the correctness and space complexity of Algorithm 2.
Proof of Theorem 3.1. We first remark that the algorithm can make at most  $\log k \leq \log n$  mistakes over the lifespan of each pool of size  $k \coloneqq \frac{2nM}{RT} \log n$  because each time the algorithm makes a mistake, at least half of the pool must be incorrect and deleted, so the size of the pool decreases by at least half with each mistake the algorithm mistakes. Note that  $k \leq n$  for  $R \geq \frac{4M \log n}{T}$ , so the algorithm is well-defined.
Since each pool  $P$  has size  $k$  and there are  $n$  experts, then there are at most  $\frac{4n}{k}$  pools before the entire set  $S$ , which is initialized to  $n$ , is depleted. Thus, there are at most  $\frac{4n}{k}$  pools to iterate through the entire set of experts. Moreover, each time the algorithm has iterated through the entire set of experts, each expert must have made at least one mistake. This is because an expert is only deleted from the pool  $P$  when it has made a mistake and since all experts have been deleted from  $P$ , then all experts have made at least one mistake.
Since the best expert makes at most  $M$  mistakes, then the best expert can be deleted from the pool  $P$  at most  $M$  times. In other words, the algorithm can cycle through the entire set of  $n$  experts at most  $M + 1$  times.
Hence, the total number of mistakes by the algorithm is at most
$$
\frac {2 n}{k} \cdot \log n \cdot (M + 1) \leq \frac {4 n}{k} \cdot \log n \cdot M \leq \frac {4 n R T}{4 n M \log n} \cdot \log n \cdot M = R T,
$$
so the algorithm achieves regret at most  $R$ . Since the algorithm selects a subset of  $k = \frac{4nM}{RT} \log n$  experts, then the space complexity follows.
D Proofs of the Lower Bounds for Arbitrary-Order Streams
In this section, we give space lower bounds for the experts problem on arbitrary-order streams. As a warm-up, we first show in Section D.1 a general space lower bound for randomized algorithms when the best expert makes a "small" number of mistakes. We then give our main lower bound result in Section D.2, showing that any deterministic algorithm achieving regret  $R$  must use space  $\Omega \left( \frac{nM}{RT} \right)$  when the best expert makes  $M$  mistakes.
D.1 Warm-up: Lower Bound for Accurate Best Expert
In this section, we show that any randomized algorithm that achieves regret  $R$  must use  $\Omega\left(\frac{n}{RT}\right)$  space, even when the best expert makes  $\Theta(RT)$  mistakes. In contrast, [49] give an  $\Omega\left(\frac{n}{R^2T}\right)$  space lower bound:
Theorem D.1 (Memory lower bound; Theorem 1 of [49]). Let  $R > 0$ ,  $p < \frac{1}{2}$  be fixed constants, i.e., independent of other input parameters. Any algorithm that achieves  $R$  regret for the experts problem with probability at least  $1 - p$  must use at least  $\Omega \left( \frac{n}{R^2 T} \right)$  space.
Furthermore, this lower bound holds even when the costs are binary, and expert predictions, as well as the correct answers, are constrained to be i.i.d. across the days, albeit with different distributions across the experts.
The proof of this lower bound exploits a construction where the best expert makes  $\Theta (T)$  mistakes. Thus, it is not clear how the space complexity of the problem behaves when the best expert makes a smaller number of mistakes. In fact, [49] also give an algorithm that uses  $\widetilde{O}\left(\frac{n}{RT}\right)$  space when the best expert makes  $O(RT)$  mistakes, bypassing the aforementioned lower bound.
We now prove that in this small mistake regime, this algorithm is tight. Towards this goal, we first define the  $\varepsilon$ -DIFFDIST problem that reduces to the experts problem. It was proposed by [49] to prove memory lower bounds for the expert problem in random order stream.
Definition D.2 (The  $\varepsilon$ -DIFFDIST Problem). We have  $T$  players, each of whom holds  $n$  bits, indexed from 1 to  $n$ . We must distinguish between two cases, which we refer to as " $V = 0"$  and " $V = 1"$ . Let  $\mu_0$  be a Bernoulli distribution with parameter  $\frac{1}{2}$ , i.e., a fair coin, and let  $\mu_1$  be a Bernoulli distribution with parameter  $\frac{1}{2} + \varepsilon$ .
- (NO Case, "  $V = 0$ ") Every index for every player is drawn i.i.d. from a fair coin, i.e.,  $\mu_0$ .
- (YES Case, " $V = 1$ ") An index  $L \in [n]$  is selected arbitrarily—the  $L$ -th bit of each player is chosen i.i.d. from  $\mu_1$ . All other bits for every player are chosen i.i.d. from  $\mu_0$ .
Any protocol that successfully solves the  $\varepsilon$ -DIFFDIST problem with a constant probability greater than  $\frac{1}{2}$  must use at least  $\Omega\left(\frac{n}{\varepsilon^2}\right)$  communication, a result due to [49]:
Lemma D.3 (Communication complexity of  $\varepsilon$ -DIFFDIST; Lemma 3 of [49]). The communication complexity of solving the  $\varepsilon$ -DIFFDIST problem with a constant  $1 - p$  probability, for any  $p \in [0,0.5)$ , is  $\Omega\left(\frac{n}{\varepsilon^2}\right)$ .
The proof of Theorem D.1 by [49] uses  $n$  coin flips across each of the  $T$  players to form the  $n$  expert predictions over each of the  $T$  days. In the NO case, each expert will be correct on roughly  $\frac{T}{2}$  days, while in the YES case, a single expert will be correct on roughly  $\frac{T}{2} + \varepsilon T$  days, so that an algorithm with regret  $R = O(\varepsilon)$  will be able to distinguish between the two cases. There is a slight subtlety in the proof that uses a masking argument to avoid "trivial" algorithms that happen to succeed on a "lucky" input, but for the purposes of our proof in this section, the masking argument is not needed. It then follows that the total communication is  $\Omega\left(\frac{n}{R^2}\right)$  across the  $T$  players, so that any streaming algorithm must use at least  $\Omega\left(\frac{n}{R^2T}\right)$  bits of space.
Suppose we instead consider the  $\varepsilon$ -DIFFDIST problem over  $RT$  players, representing  $RT$  days in the experts problem. Moreover, suppose we set  $\varepsilon = \Theta(1)$  in the  $\varepsilon$ -DIFFDIST problem, so that in the NO case, each of the experts will be correct on roughly  $\frac{RT}{2}$  days, while in the YES case, a single expert will be correct on roughly  $\frac{RT}{2} + CRT$  days, for some constant  $C > 0$ . Suppose we further pad all of the experts with incorrect predictions across an additional  $T - RT$  days, so that the total number of days is  $T$ , but the number of correct expert predictions remains the same. Then an algorithm achieving regret  $O(R)$  will be able to distinguish between the two cases, so that the total communication is  $\Omega\left(\frac{n}{R}\right)$ , so that any streaming algorithm must use at least  $\Omega\left(\frac{n}{RT}\right)$  bits of space.
Corollary D.4. Let  $R, p < \frac{1}{2}$  be fixed constants, i.e., independent of other input parameters. Any algorithm that achieves  $R$  regret for the experts problem with probability at least  $1 - p$  must use at least  $\Omega \left( \frac{n}{RT} \right)$  space even when the best expert makes as few as  $\Theta(RT)$  mistakes. This lower bound holds even when the costs are binary and expert predictions, as well as the correct answer, are constrained to be i.i.d. across the days, albeit with different distributions across the experts.
Proof. The claim follows from setting  $T = RT$  and  $R = \Theta(1)$  in the proof of Theorem D.1.
D.2 Lower Bound for Deterministic Algorithms
We now prove our main space lower bound for deterministic algorithms (Theorem 1.3). We first set up some basic notations and introduce a hard distribution.
Let  $T$  be any fixed positive integer. Let  $\mathcal{D}_{\mathrm{NO}}^{(n)}$  be the distribution over matrices  $A$  with size  $T\times n$  such that all entries of the matrix are i.i.d. Bernoulli with parameter  $\frac{1}{2}$ , i.e., each entry of  $A$  is 0 with probability  $\frac{1}{2}$  and 1 with probability  $\frac{1}{2}$ . Let  $\mathcal{D}_{\mathrm{YES}}^{(n)}$  be the distribution over matrices  $M$  with size  $T\times n$  such that there is a randomly chosen column  $L\in [n]$ , which is i.i.d. Bernoulli with parameter  $(1 - \frac{M}{T})$  and all other columns are i.i.d. Bernoulli with parameter  $\frac{1}{2}$ . Let  $\mathrm{BIASDETECT}_n$  be the problem of detecting whether  $A$  is drawn from  $\mathcal{D}_{\mathrm{YES}}^{(n)}$  or  $\mathcal{D}_{\mathrm{NO}}^{(n)}$ , so that  $\mathrm{BIASDETECT}_n$  is simply the  $\varepsilon$ -DIFFDIST problem with  $\varepsilon = \frac{1}{2} -\frac{M}{T}$ .
Let  $\Pi$  be a communication protocol for  $\mathrm{BIASDETECT}_n$  that is correct with probability at least  $1 - \exp(-\Theta(T))$ . Since  $\mathcal{D}_{\mathrm{NO}}^{(n)}$  is a product distribution across columns, then it can be written as  $\zeta^n$ , where  $\zeta$  is the distribution over a single column such that all entries of the column are i.i.d. Bernoulli with parameter  $\frac{1}{2}$ . Let  $\mathrm{BIASDETECT}_1$  denote the problem of distinguishing between  $\mathcal{D}_{\mathrm{NO}}^{(1)}$  and  $\mathcal{D}_{\mathrm{YES}}^{(1)}$  on a single column, i.e.,  $n = 1$ . Using  $\mathcal{D}_{\mathrm{NO}}^{(n)}$  as the hard distribution, we have the following direct sum theorem.
Lemma D.5 (Direct sum for BIASDETECT). The information complexity of  $\mathrm{BIASDETECT}_n$  satisfies
$$
\mathsf {I C} _ {\mathcal {D} _ {\mathsf {N O}} ^ {(n)}, 2 ^ {- \Theta (T)}} \left(\mathrm {B I A S D e t e c t} _ {n}\right) \geq n \cdot \mathsf {I C} _ {\mathcal {D} _ {\mathsf {N O}} ^ {(1)}, 2 ^ {- \Theta (T)}} \left(\mathrm {B I A S D e t e c t} _ {1}\right).
$$
Proof. By definition,  $\mathcal{D}_{\mathrm{NO}}^{(n)} = \zeta^n$  is a product distribution over  $n$  columns. The lemma follows from the standard direct sum lemma of information cost (Lemma B.5).
With the above direct sum theorem for  $\mathrm{BIASDETECT}_n$ , it now suffices to provide a single-coordinate information cost lower bound against  $\mathrm{BIASDETECT}_1$ . The proof is delayed to Section D.3.
Lemma D.6 (Single-coordinate information cost lower bound). Let  $c \in (0,1)$  and  $\Pi$  be any protocol with error  $\delta = 2^{-\Theta(T)}$  for BIASDETECT $_1$ . We have that the information cost of  $\Pi$  with respect to  $\zeta$  is at least
$$
I \left(\Pi \left(C _ {1}, C _ {2}, \dots , C _ {T}\right); C _ {1}, C _ {2}, \dots , C _ {T}\right) \geq \Omega (M), \tag {D.1}
$$
where the bits  $C_i \sim \zeta$  are i.i.d. single coordinates.
Combining Lemma D.6 with the direct sum theorem (Lemma D.5), we immediately get the following information complexity lower bound for  $\mathrm{BIASDETECT}_n$ :
Theorem D.7 (n-Coordinate information complexity lower bound). Let  $c \in (0,1)$ . Then
$$
\mathsf {I C} _ {\mathcal {D} _ {\mathsf {N O}} ^ {(n)}, 2 ^ {- \Theta (T)}} (\mathrm {B I A S D E T E C T} _ {n}) = \Omega (n M).
$$
Proof. This follows by applying the direct sum theorem (Lemma D.5) to the single-coordinate bound Lemma D.6.  $\square$
This implies that any algorithm with  $R$  regret and success rate at least  $1 - 2^{-\Theta (T)}$  requires  $\Omega \left(\frac{nM}{RT}\right)$  memory, where  $M$  is the mistake bound on the best expert.
Theorem D.8 (Memory lower bound for expert learning). Let  $R, M$  be fixed and independent of other input parameters. Any streaming algorithm that achieves  $R$  regret for the experts problem with probability at least  $1 - 2^{-\Theta(T)}$  must use at least  $\Omega\left(\frac{nM}{RT}\right)$  space, for  $n = o(2^T)$ , where the best expert makes  $M$  mistakes.
Proof. We now consider the problem  $\mathrm{BIASDETECT}_n$  on a matrix of size  $RT \times n$ . Note that in the NO case, at any fixed column  $i \in [n]$ , the probability that there are more than  $\frac{3RT}{5} - \frac{M}{2}$  instances of 0, for  $M \leq \frac{RT}{8}$ , is at most  $2\exp(-c_1RT)$ , for a sufficiently small constant  $c_1 \in (0,1)$ . Thus, by a union bound, the probability that there exists an index  $i \in [n]$  with more than  $\frac{3RT}{4} - \frac{M}{2}$  instances of 0 is at most  $2n\exp(-c_1RT)$ .
Similarly in the YES case, the probability that there are fewer than  $\frac{4RT}{5} - \frac{M}{2}$  instances of 0 for a fixed  $i \in [n]$  and for  $M \leq \frac{RT}{8}$  is at most  $2\exp(-c_2RT)$ , for a sufficiently small constant  $c_2 \in (0,1)$  and so by a union bound, the probability that there exists an index  $i \in [n]$  with fewer than  $\frac{3RT}{4} - \frac{M}{2}$  instances of 0 is at most  $2n\exp(-c_2RT)$ . Hence, for  $n = o(2^T)$ , there exists a constant  $c \in (0,1)$  such that any algorithm that achieves total regret at most  $\frac{RT}{5}$  with probability at least  $1 - \exp(-cT)$  can distinguish between the YES and NO cases with probability  $1 - \exp(-\Theta(T))$ .
By Theorem D.7 and Lemma B.6, the total communication across the  $RT$  players must be at least  $\Omega(nM)$ . Therefore, any streaming algorithm that achieves average  $R$  regret for the experts problem with probability at least  $1 - 2^{-\Theta(T)}$  must use at least  $\Omega\left(\frac{nM}{RT}\right)$  space.
D.3 Proof of the Single-Coordinate Information Cost Lower Bound
We now show the single-coordinate lower bound of Lemma D.6.
Proof of Lemma D.6. Consider a protocol that is correct with probability  $1 - 2^{-\Theta (T)}$  and let  $(C_1,C_2,\ldots ,C_T)\sim \zeta^T$  be a single column drawn from the NO case, where each coordinate is i.i.d. Bernoulli with parameter  $1 / 2$ . For notational convenience, let  $\Pi = \Pi (C_1,\dots ,C_T)$  denote the transcript given the input  $(C_1,C_2,\dots ,C_T)$ . We consider the one-way message-passing model, where each player  $P_{i}$  holds the input  $C_i$ . For all  $i < T$ , let  $M_{i}$  denote the message sent from player  $P_{i}$  to player  $P_{i + 1}$ .
By the chain rule of mutual information, the information cost of the transcript, the left-side of Equation D.1 that we need to bound, can be written as
$$
I \left(\Pi ; C _ {1}, C _ {2}, \dots , C _ {T}\right) = \sum_ {j = 1} ^ {T} I \left(M _ {j}; C _ {1}, C _ {2}, \dots , C _ {T} \mid M _ {<   j}\right). \tag {D.2}
$$
By the independence of one-way communication, we have
$$
I \left(M _ {j}; C _ {1}, C _ {2}, \dots , C _ {T} \mid M _ {<   j}\right) = I \left(M _ {j}; C _ {j} \mid M _ {<   j}\right). \tag {D.3}
$$
Combining the two equalities above, the information cost equals
$$
I \left(\Pi ; C _ {1}, C _ {2}, \dots , C _ {T}\right) = \sum_ {j = 1} ^ {T} I \left(M _ {j}; C _ {j} \mid M _ {<   j}\right). \tag {D.4}
$$
We now lower bound the right-side. First, we make the following definition. For any  $i \in [T]$ , we say that  $(M_i, M_{<i})$  is informative for  $i$  with respect to the input  $C$  and the transcript  $\Pi = (M_1, \dots, M_T)$  if
$$
\left| \Pr \left(C _ {i} = 0 \mid M _ {i}, M _ {<   i}\right) - \Pr \left(C _ {i} = 1 \mid M _ {i}, M _ {<   i}\right) \right| \geq c \tag {D.5}
$$
for some constant  $c > 0$ ; and uninformative otherwise. Intuitively, an informative index  $i$  with respect to  $(M_i, M_{<i})$  means that conditional on the past messages  $M_{<i}$ , the message  $M_i$  reveals much information about  $C_i$ . Hence, in this case,  $I(M_i, C_i \mid M_{<i})$  would be large. Now for all  $i \in [T]$ , let  $p_i$  be the probability that  $(M_i, M_{<i})$  is informative (for  $i$  with respect to  $C$  and  $\Pi$ ).
Conceptually, we need to show that  $\sum_{i}p_{i}$  is large, since then there would be sufficiently many informative messages, and so the information cost in the left-side of Equation D.4 is high. We formalize this idea in the following lemma.
Lemma D.9. In the setting above, where  $c > 0$  is a constant, the information cost can be lower bounded by
$$
I \left(\Pi ; C _ {1}, C _ {2}, \dots , C _ {T}\right) = \sum_ {j = 1} ^ {T} I \left(M _ {j}; C _ {j} \mid M _ {<   j}\right) \geq \Omega \left(\sum_ {j = 1} ^ {T} p _ {j}\right) \tag {D.6}
$$
Proof. We start by expanding the definition of the mutual information terms. For each  $j \in T$ , we have
$$
I \left(M _ {j}; C _ {j} \mid M _ {<   j}\right) = H \left(C _ {j} \mid M _ {<   j}\right) - H \left(C _ {j} \mid M _ {j}, M _ {<   j}\right) \tag {D.7}
$$
For the first term, notice that  $C_j$  and  $M_{<j}$  are independent by one-way communication. Moreover, by definition  $C_j$  is Bernoulli with parameter  $1/2$ . Therefore,
$$
H \left(C _ {j} \mid M _ {<   j}\right) = H \left(C _ {j}\right) = H (1 / 2) = 1.
$$
- either  $(M_j, M_{<j})$  is informative, which holds with probability  $p_j$ , and in this case, the conditional entropy is upper bounded by  $H(C_j \mid M_j, M_{<j}) \leq H(1/2 + c/2)$ ;
- or  $(M_j, M_{<j})$  is uninformative, and in this case, we trivially upper bound the conditional entropy by  $H(C_j \mid M_j, M_{<j}) \leq 1$ ;
Putting the observations together and using Equation D.7, it follows that
$$
\begin{array}{l} I \left(M _ {j}; C _ {j} \mid M _ {<   j}\right) = H \left(C _ {j} \mid M _ {<   j}\right) - H \left(C _ {j} \mid M _ {j}, M _ {<   j}\right) \\ \geq 1 - \left(p _ {j} \cdot H \left(1 / 2 + c / 2\right) + \left(1 - p _ {j}\right) \cdot 1\right) \\ = p _ {j} - p _ {j} \cdot H (1 / 2 + c / 2) \\ \geq p _ {j} \left(1 - (1 - c ^ {2}) ^ {1 / \ln 4}\right). \\ \end{array}
$$
where the last step uses the upper bound of Lemma B.1. Then we have
$$
\begin{array}{l} I \left(M _ {j}; C _ {j} \mid M _ {<   j}\right) \geq p _ {j} \left(1 - (1 - c ^ {2}) ^ {1 / \ln 4}\right) \\ \geq c ^ {3} \cdot \Omega (p _ {j}), \\ \end{array}
$$
where the last step follows since  $1 - (1 - x^{2})^{1 / \ln 4} \geq x^{3} / 100$  for  $x \in [0,1]$ . Summing over  $j = 1,2,\ldots,T$  in Equation D.6 finishes the proof.
To prove the claimed information cost inequality Equation D.1, we show that  $\sum_{i}p_{i} = \Omega (M)$
Lemma D.10. There exists a constant  $\gamma > 0$  such that
$$
\sum_ {j = 1} ^ {T} p _ {j} > \gamma \cdot M.
$$
Proof. Suppose by way of contradiction that  $\sum_{j=1}^{T} p_{j} = o(M)$ . Let  $A$  be a protocol that sends (possibly random) messages  $M_{1}, \ldots, M_{T}$  on a random input  $C \in \{0,1\}^{T} \sim \zeta^{T}$  drawn uniformly from the NO distribution, i.e., each coordinate of  $C := C_{1}, \ldots, C_{T}$  is picked to be 0 with probability  $\frac{1}{2}$  and 1 with probability  $\frac{1}{2}$ . Moreover, suppose  $A$  is a protocol that distinguishes between a YES instance and a NO instance with probability at least  $1 - \frac{e^{-cT}2^{-T}}{8}$ , for some constant  $c > 0$ .
Since  $p_i$  is the probability that  $M_i$  is informative, then by assumption, the expected number of informative indices  $i$  over the messages  $M_1, \ldots, M_T$  is  $f(M)$  for some  $f(M) = o(M)$ . Thus by Markov's inequality, the probability that the number of informative indices is at most  $10f(M) = o(M)$  with probability at least  $\frac{9}{10}$ . Let  $S$  be the set of the uninformative indices so that  $|S| = T - 10f(M) = T - o(M)$ . Let  $C'$  be an input that agrees with  $C$  on the informative indices  $[T] \setminus S$  and is chosen arbitrarily on uninformative indices  $S$ , so that  $C_i' = C_i$  for  $i \in [T] \setminus S$ .
By definition, each uninformative index only changes the distribution of the output by a  $(1 \pm c)$  factor. In particular, for  $c \in (0, 1/2)$ , the probability that the protocol  $A$  generates  $\Pi$  on input  $C'$  is at least  $(1 - c)^T \geq e^{-2cT}$  times the probability that the protocol  $A$  generates  $\Pi$  on input  $C$ . However, since  $C$  can differ from  $C'$  on  $S$ , then  $C$  can differ from  $C'$  on  $|S| = T - 10f(M) = T - o(M)$  indices.
Now since each coordinate of  $C$  is picked to be 0 with probability  $\frac{1}{2}$  and 1 with probability  $\frac{1}{2}$ , then the probability that  $C$  contains more than  $T - M$  zeros is at least  $1 - T^M \cdot \frac{1}{2^T} \geq 1 - 2^{T/2}$  for sufficiently large  $T$ . But then there exists a choice of  $C'$  that contains fewer than  $\frac{M}{2}$  zeros such that  $A$  will also output  $\Pi$  with probability at least  $\frac{e^{-cT}}{2}$ . Since  $C'$  contains fewer than  $\frac{M}{2}$ , then  $C'$  is more likely to generate from a YES instance and indeed a YES instance will generate  $C$  with probability
$2^{-T}$ . On the other hand, since  $\Pi$  corresponds to a transcript for which  $A$  will output NO, then the probability that  $A$  is incorrect on  $C'$  is at least  $\frac{e^{-cT}}{4}$ , which contradicts the claim that  $A$  succeeds with probability  $1 - \frac{e^{-cT}2^{-T}}{8}$ . Thus it follows that  $\sum_{j=1}^{T} p_j = \Omega(M)$ , as desired.
Now we combine Lemma D.9 and Lemma D.10. This implies that the information cost can be lower bounded by
$$
I \left(\Pi ; C _ {1}, C _ {2}, \dots , C _ {T}\right) \geq \Omega \left(\sum_ {j = 1} ^ {T} p _ {j}\right) \geq \gamma M, \tag {D.8}
$$
for a constant  $\gamma > 0$ . This completes the proof.
E An Alternative Proof in the Large Mistake Regime
We give another analysis of the information cost when  $M = \Omega(T)$ , where  $M$  is the number of mistakes of the best expert.
Lemma E.1 (Single-Coordinate Information Cost Lower Bound). Let  $c \in (0, 1)$  and  $\Pi$  be any protocol with error  $\delta = 2^{-T}$  for  $\mathrm{BIASDETECT}_1$ . Suppose that the best expert makes  $M = c' T$  mistakes for some constant  $c'$ . We have that the information cost of  $\Pi$  with respect to  $\zeta$  is at least
$$
I \left(\Pi \left(C _ {1}, \dots , C _ {T}\right); C _ {1}, \dots , C _ {T}\right) \geq \Omega \left((1 - c) ^ {2} T\right), \tag {E.1}
$$
where  $C_i \sim \zeta$  are i.i.d. single coordinates.
Applying direct sum theorem (Lemma D.5), we get the following information complexity lower bound for  $\mathrm{BIASDETECT}_n$ :
Theorem E.2 (n-Coordinate Information Complexity Lower Bound). Let  $c \in (0,1)$  and assume  $M = c^{\prime}T$  for some constant  $c^\prime$ . Then
$$
\left. \mathrm {I C} _ {\mathcal {D} ^ {(1)}, 2 ^ {- \Theta (T)}} \left(\mathrm {B I A S D E T E C T} _ {n}\right) = \Omega \left((1 - c) ^ {2} n T\right). \right.
$$
By an argument similar to Theorem D.8, we have:
Theorem E.3 (Memory lower bound for expert learning). Let  $M = c' T$  for some constant  $c'$ . Any streaming algorithm that achieves constant regret for the experts problem with probability at least  $1 - 2^{-\Theta(T)}$  must use at least  $\Omega(n)$  space, where the best expert makes  $M$  mistakes.
For the purpose of proving Lemma E.1, we need some technical lemmas.
Lemma E.4 (Lemma 3.5 of [39]). Consider any communication protocol  $\Pi$  where each player receives one bit and condition on any fixed input  $b\in \{0,1\} ^T$ . Each player  $i$  can be implemented such that, if the other players receive input  $b_{-i}$ , player  $i$  only observes their input with probability  $d_{TV}(\Pi_b,\Pi_{b\oplus e_i})$ .
Lemma E.5 (Lemma 3.6 of [39]). Let  $c \in (0,1)$ ,  $p \in (0,\frac{1 - c}{2})$  and  $\gamma_c = \frac{1}{c\log(e / c)}$ . For a set of binary random variables  $Y_1, Y_2, \dots, Y_k$  such that  $\mathbb{E}[\sum_i Y_i] = pk$ , there exists a set  $S \subset [n]$  of size  $ck$  such that  $\operatorname*{Pr}(Y_j = 0, \forall j \in S) > e^{-k / \gamma_c - 1}$ .
Proof of Lemma E.1. Let  $(C_1, C_2, \dots, C_n) \sim \zeta^n$  be a single column drawn from the NO case, where each coordinate is i.i.d. Bernoulli with parameter  $1/2$ . Let  $M = c' T$  for some constant  $c'$ . We consider the one-way message-passing model, where for all  $i < T$ ,  $M_i$  denotes the message sent from player  $P_i$  to player  $P_{i+1}$ . It suffices to lower bound