SlowGuess/NeurIPS_2023_OCR huggingface dataset:
3. Feed  $S_{T} = \left\{(\pmb{r}^{(1)}, \omega^{(1)}), \dots, (\pmb{r}^{(T)}, \omega^{(T)})\right\}$  to the forecast aggregation problem. Obtain solution  $\hat{f}$ .
4. Convert  $\hat{f}$  into  $\hat{D}$  according to Claim E.2.
Output:  $\hat{D}$
We remark that, in the second step of the reduction, the report  $r_i^{(t)}$  can be computed even if  $D$  is unknown, because the  $D(s_i^{(t)})$  is the same for all  $D \in \mathcal{D}$  and hence known.
Using the above reduction, we show that the sample complexity of  $\varepsilon$ -optimal forecast aggregation for  $\mathcal{P}$  cannot be smaller than the sample complexity of learning  $\mathcal{D}$  within total variation distance  $(1 + B)^2\sqrt{\varepsilon}$ , which will prove Lemma 4.3:
Proof of Lemma 4.3: First, we verify that the distribution of samples  $S_T$  in the above reduction is exactly the distribution of  $T$  samples  $\{(\pmb{r}^{(1)},\omega^{(1)}),\dots ,(\pmb{r}^{(T)},\omega^{(T)})\}$  from  $P_D$ . This is because: (1) the distribution of  $\omega^{(t)}$  is Uniform  $\{0,1\}$ , as defined in  $P_D$ ; (2) given  $\omega^{(t)} = 0$ , the distribution of  $s^{(t)}$  is Uniform  $(\mathcal{S})$ , as defined in  $P_D$ ; (3) given  $\omega^{(t)} = 1$ , the distribution of  $s^{(t)}$  is the same as the distribution of  $x^{(t)}$ , which is  $D$ , because the random draws of  $\omega^{(t)}$  and  $x^{(t)}$  are independent; (4) according to Fact E.1, the report  $r_i^{(t)} = \frac{D(s_i^{(t)})}{\frac{1}{m} + D(s_i^{(t)})} = P_D(\omega = 1|s_i^{(t)})$ , as desired.
Then, by the definition of sample complexity of forecast aggregation, if we are given  $T = T_{\mathcal{P}}(\varepsilon, \delta)$  samples  $S_T$  for the forecast aggregation problem, then with probability at least  $1 - \delta$  we should be able to find an  $\varepsilon$ -optimal aggregator  $\hat{f}$  with respect to  $P_D$ . According to Claim E.2, we can convert  $\hat{f}$  into a  $\hat{D}$  such that
$$
d _ {\mathrm {T V}} (\hat {D}, D) \leq (1 + B) ^ {2} \sqrt {\varepsilon}.
$$
By the definition of sample complexity  $T_{\mathcal{D}}^{\mathrm{TV}}(\cdot, \delta)$  of distribution learning,  $T$  must be at least
$$
T \geq T _ {\mathcal {D}} ^ {\mathrm {T V}} \big ((1 + B) ^ {2} \sqrt {\varepsilon}, \delta \big),
$$
E.2 Proof of Proposition 4.4
To prove Proposition 4.4 we will construct a family of distributions  $\mathcal{D}$  that satisfies the three properties in Definition 4.2 and requires  $T_{\mathcal{D}}^{\mathrm{TV}}(\varepsilon_{\mathrm{TV}},\delta) = \Omega \left(\frac{m^{n - 2} + \log(1 / \delta)}{\varepsilon_{\mathrm{TV}}^2}\right)$  samples to learn. For simplicity, we write  $\varepsilon = \varepsilon_{\mathrm{TV}}$ . For technical convenience, we assume  $\varepsilon < \frac{1}{40}$ ,  $\delta < 0.01$ .
E.2.1 Part 1: Constructing  $\mathcal{D}$
We index the distributions  $D_z \in \mathcal{D}$  by  $z$ ; the meaning of  $z$  will be defined later. Without loss of generality, we assume  $|\mathcal{S}_i| = m$  to be an even integer, and denote  $\mathcal{S}_i = \{1, \dots, m\} =: S$ . We will define  $D_z$  to be a distribution over the joint signal space  $\pmb{S} = \mathcal{S}_1 \times \dots \times \mathcal{S}_n = S^n = \{1, \dots, m\}^n$ . In the following we will call a joint signal  $s \in S^n$  simply a signal. We write a signal  $s \in S^n$  as  $s = (b, x, y)$ , where  $b \in S^{n-2}$  and  $x, y \in S$ . We sort all the  $m^n$  signals in  $S^n$  by the lexicographical
order, from  $(1,\ldots ,1,1)$ ,  $(1,\ldots ,1,2)$ , ..., to  $(m,\dots ,m,m)$ . We number the signals from 1 to  $m^n$ , using
$$
\operatorname {n u m} (\boldsymbol {s}) = \operatorname {n u m} (\boldsymbol {b}, x, y) \in \{1, \dots , m ^ {n} \}.
$$
to denote their numbers. We divide the whole signal space  $S^n$  into  $|S^{n - 2}| = m^{n - 2}$  " buckets", each of size  $m^2$  and denoted by  $B_{b}$ :
$$
B _ {\boldsymbol {b}} = \left\{\left(\boldsymbol {b}, x, y\right): x, y \in S \right\}, \quad \boldsymbol {b} \in S ^ {n - 2}.
$$
We first define a "base" distribution  $D_{\mathrm{base}}$ , then construct the distributions  $D_z$ 's by modifying the probabilities of the base distribution within each bucket  $B_b$ . Let  $\gamma = 1 + \frac{1}{m^n}$ . The base distribution is defined as follows:
$$
D _ {\text {b a s e}} (s) = \frac {\gamma^ {\operatorname {n u m} (s)}}{W}, \quad W = \sum_ {s \in S ^ {n}} \gamma^ {\operatorname {n u m} (s)} = \sum_ {\ell = 1} ^ {m ^ {n}} \gamma^ {\ell}.
$$
Because  $1 \leq \gamma^{\ell} \leq \gamma^{m^n} \leq \left(1 + \frac{1}{m^n}\right)^{m^n} \leq e$ , we have
$$
m ^ {n} \leq W \leq e m ^ {n}. \tag {26}
$$
We assign a sign  $z_{b} \in \{+1, -1\}$  to each bucket  $B_{b}$ , and let  $z$  be a vector of length  $m^{n-2}$  that includes the signs of all buckets:
$$
z = \left(z _ {\boldsymbol {b}}\right) _ {\boldsymbol {b} \in S ^ {n - 2}}, \quad z _ {\boldsymbol {b}} \in \{+ 1, - 1 \}.
$$
We have  $2^{m^{n-2}}$  different  $z$ 's in total, and hence  $2^{m^{n-2}}$  different distributions  $D_z$ 's in  $\mathcal{D}$  in total. Let  $c = 20$ , so that  $c\varepsilon \leq 1/2$ . For each  $z$ , we define  $D_z$  as follows: within each bucket  $B_b$ , for each element  $(b, x, y) \in B_b$  let
$$
D _ {z} (\boldsymbol {b}, x, y) = \left\{ \begin{array}{l l} D _ {\text {b a s e}} (\boldsymbol {b}, x, y) + \frac {z _ {\boldsymbol {b}} c \varepsilon}{W} & \text {i f} x \leq \frac {m}{2}, y \leq \frac {m}{2} \\ D _ {\text {b a s e}} (\boldsymbol {b}, x, y) - \frac {z _ {\boldsymbol {b}} c \varepsilon}{W} & \text {i f} x \leq \frac {m}{2}, y > \frac {m}{2} \\ D _ {\text {b a s e}} (\boldsymbol {b}, x, y) - \frac {z _ {\boldsymbol {b}} c \varepsilon}{W} & \text {i f} x > \frac {m}{2}, y \leq \frac {m}{2} \\ D _ {\text {b a s e}} (\boldsymbol {b}, x, y) + \frac {z _ {\boldsymbol {b}} c \varepsilon}{W} & \text {i f} x > \frac {m}{2}, y > \frac {m}{2} \end{array} . \right. \tag {27}
$$
Claim E.3. The family of distributions  $\mathcal{D} = \{D_z\}_{z}$  defined above satisfies the three properties in Definition 4.2:  $B$ -uniformly bounded with  $B = e + 1/2$ , same marginal across distributions, distinct marginals across signals.
Proof.  $B$ -uniformly bounded: For any  $s$ , any  $D_z$ , by definition,
$$
D _ {z} (\boldsymbol {s}) \leq D _ {\text {b a s e}} (\boldsymbol {s}) + \frac {c \varepsilon}{W} \leq \frac {\gamma^ {m ^ {n}}}{W} + \frac {c \varepsilon}{W} \stackrel {(2 6)} {\leq} \frac {e}{m ^ {n}} + \frac {1 / 2}{m ^ {n}}.
$$
So, the distribution is  $B$ -uniformly bounded with  $B = e + 1/2$ .
Same marginal across distributions: Consider each  $D_z(s_i)$ . We want to show that  $D_z(s_i)$  does not depend on  $z$ , and in fact,  $D_z(s_i) = D_{\mathrm{base}}(s_i)$ . If  $i \in \{1, \dots, n-2\}$ , namely,  $s_i$  is a component of the vector  $b$ , then we have
$$
D _ {z} (s _ {i}) = \sum_ {\boldsymbol {s} _ {- i} \in \boldsymbol {\mathcal {S}} _ {- i}} D _ {z} (s _ {i}, \boldsymbol {s} _ {- i}) = \sum_ {\boldsymbol {b} \in S ^ {n - 2}; b _ {i} = s _ {i}} \sum_ {x = 1} ^ {m} \sum_ {y = 1} ^ {m} D _ {z} (\boldsymbol {b}, x, y).
$$
We notice that, fixing any  $\pmb{b}$ , the numbers of  $\frac{+z_b c \varepsilon}{W}$  and  $\frac{-z_b c \varepsilon}{W}$  in the summation  $\sum_{x=1}^{m} \sum_{y=1}^{m} D_z(\pmb{b}, x, y)$  are the same. So, they cancel out, and we obtain
$$
D _ {z} (s _ {i}) = \sum_ {\pmb {b} \in S ^ {n - 2}: b _ {i} = s _ {i}} \sum_ {x = 1} ^ {m} \sum_ {y = 1} ^ {m} D _ {\mathrm {b a s e}} (\pmb {b}, x, y) = D _ {\mathrm {b a s e}} (s _ {i}).
$$
If  $i = n - 1$ , namely  $s_i = x$ , then we have:
$$
D _ {z} (s _ {i}) = \sum_ {\pmb {s} _ {- i} \in \pmb {\mathcal {S}} _ {- i}} D _ {z} (s _ {i}, \pmb {s} _ {- i}) = \sum_ {\pmb {b} \in S ^ {n - 2}} \sum_ {y = 1} ^ {m} D _ {z} (\pmb {b}, x, y).
$$
Fixing any  $\pmb{b}$ , the numbers of  $\frac{\pm z_b c\varepsilon}{W}$  and  $\frac{-z_b c\varepsilon}{W}$  in the summation  $\sum_{y=1}^{m} D_z(\pmb{b}, x, y)$  are the same. So, they cancel out, and we obtain
$$
D _ {z} (s _ {i}) = \sum_ {\boldsymbol {b} \in S ^ {n - 2}} \sum_ {y = 1} ^ {m} D _ {\mathrm {b a s e}} (\boldsymbol {b}, x, y) = D _ {\mathrm {b a s e}} (s _ {i}).
$$
Finally, if  $i = n$ , namely  $s_i = y$ , then by a similar argument as above we have
$$
D _ {z} (s _ {i}) = \sum_ {\boldsymbol {b} \in S ^ {n - 2}} \sum_ {x = 1} ^ {m} D _ {\mathrm {b a s e}} (\boldsymbol {b}, x, y) = D _ {\mathrm {b a s e}} (s _ {i}).
$$
Distinct marginals across signals: By the "same marginal across distributions" property above we have  $D_z(s_i) = D_{\mathrm{base}}(s_i)$ . So, to prove "distinct marginals across signals" we only need to prove  $D_{\mathrm{base}}(s_i) \neq D_{\mathrm{base}}(s_i')$  for  $s_i \neq s_i'$ . Without loss of generality assume  $s_i < s_i'$ . By the definition  $D_{\mathrm{base}}(\boldsymbol{s}) = \frac{\gamma^{\mathrm{num}(\boldsymbol{s})}}{W}$  and the fact that  $\mathrm{num}(s_i, s_{-i}) < \mathrm{num}(s_i', s_{-i})$  for any  $s_{-i} \in S_{-i}$ , we have
$$
D _ {\mathrm {b a s e}} (s _ {i}) = \sum_ {\pmb {s} _ {- i} \in \pmb {\mathcal {S}} _ {- i}} D _ {\mathrm {b a s e}} (s _ {i}, \pmb {s} _ {- i}) <   \sum_ {\pmb {s} _ {- i} \in \pmb {\mathcal {S}} _ {- i}} D _ {\mathrm {b a s e}} (s _ {i} ^ {\prime}, \pmb {s} _ {- i}) = D _ {\mathrm {b a s e}} (s _ {i} ^ {\prime}),
$$
so  $D_{\mathrm{base}}(s_i)\neq D_{\mathrm{base}}(s_i')$
E.2.2 Part 2: Sample Complexity Lower Bound of Learning  $\mathcal{D}$
Overview We want to prove the proposition that the sample complexity of learning the family of distributions  $D = \{D_z\}_{z}$  defined above is at least  $T_{\mathcal{D}}^{\mathrm{TV}}(\varepsilon, \delta) = \Omega \left( \frac{m^{n-2} + \log(1/\delta)}{\varepsilon^2} \right)$ . This proof is analogous to a textbook proof of Proposition 3.3 (the sample complexity for learning all distributions), which uses reductions from the distinguishing distributions problem. Roughly speaking, if one can learn the unknown distribution  $D_z$  well then one must be able to guess most of the components of the sign vector  $z = (z_b)_{b \in S^{n-2}}$  correctly, meaning that one can distinguish whether the distribution  $D_{z_b}$  on bucket  $B_b$  is  $D_{z_b=+1}$  or  $D_{z_b=-1}$ . However, since  $D_{z_b=+1}$  and  $D_{z_b=-1}$  are "  $O(\varepsilon)$ -close" to each other, distinguishing them requires  $\Omega \left( \frac{1}{\varepsilon^2} \right)$  samples. In average, there are only  $O\left( \frac{T}{m^{n-2}} \right)$  samples falling into a bucket (because there are  $m^{n-2}$  buckets in total and the distribution  $D_z$  is close to uniform). We thus need  $O\left( \frac{T}{m^{n-2}} \right) = \Omega \left( \frac{1}{\varepsilon^2} \right)$ , which gives  $T = \Omega \left( \frac{m^{n-2}}{\varepsilon^2} \right)$ . Ignoring logarithmic terms, this proves the proposition.
Formal argument First, we note that if we can learn  $D_z$  very well, then we can guess the vector  $z$  correctly for a large fraction of its  $m^{n-2}$  components. Formally, suppose we obtain from  $T$  samples a distribution  $\hat{D}$  such that  $d_{\mathrm{TV}}(\hat{D}, D_z) \leq \varepsilon$ . We find the distribution  $D_w$  in  $\mathcal{D}$ ,  $w = (w_b)_{b \in S^{n-2}} \in \{+1, -1\}^{m^{n-2}}$ , that is closest to  $\hat{D}$  in total variation distance. By definition, we have  $d_{\mathrm{TV}}(D_w, \hat{D}) \leq d_{\mathrm{TV}}(D_z, \hat{D}) \leq \varepsilon$ . Hence, by triangle inequality,
$$
d _ {\mathrm {T V}} \left(D _ {w}, D _ {z}\right) \leq d _ {\mathrm {T V}} \left(D _ {w}, \hat {D}\right) + d _ {\mathrm {T V}} \left(\hat {D}, D _ {z}\right) \leq 2 \varepsilon .
$$
$$
\# = \left| \left\{\boldsymbol {b} \in S ^ {n - 2} \mid w _ {\boldsymbol {b}} \neq z _ {\boldsymbol {b}} \right\} \right|
$$
be the number of different components of  $w$  and  $z$ . We claim that:
Claim E.4.  $d_{\mathrm{TV}}(D_w, D_z) \leq 2\varepsilon$  implies  $\# \leq \frac{2W}{cm^2}$ .
Proof. Whenever we have a different component  $w_{b} \neq z_{b}$ , this different component contributes the following to the total variation distance between  $D_{w}$  and  $D_{z}$ :
$$
\frac {1}{2} \sum_ {(\boldsymbol {b}, x, y) \in B _ {\boldsymbol {b}}} \left| D _ {w} (\boldsymbol {b}, x, y) - D _ {z} (\boldsymbol {b}, x, y) \right| = \frac {1}{2} \sum_ {(\boldsymbol {b}, x, y) \in B _ {\boldsymbol {b}}} \frac {2 c \varepsilon}{W} = \frac {m ^ {2} c \varepsilon}{W}. \tag {28}
$$
So, the number of different components of  $w$  and  $z$  is at most  $\frac{d_{\mathrm{TV}}(D_w,D_z)}{\frac{m^2c\varepsilon}{W}} \leq \frac{2W}{cm^2}$ .
We first show the  $\Omega\left(\frac{\log(1 / \delta)}{\varepsilon^2}\right)$  part in the sample complexity lower bound, and then show the  $\Omega\left(\frac{m^{n-2}}{\varepsilon^2}\right)$  part. Together, they give the lower bound  $\max\left\{\Omega\left(\frac{\log(1 / \delta)}{\varepsilon^2}\right), \Omega\left(\frac{m^{n-2}}{\varepsilon^2}\right)\right\} = \Omega\left(\frac{m^{n-2} + \log(1 / \delta)}{\varepsilon^2}\right)$ . Consider the distribution  $D_{+1} \in \mathcal{D}$  whose index is the all “+1” vector and the distribution  $D_{-1} \in \mathcal{D}$  whose index is the all “-1” vector. According to (28), the total variation distance between  $D_{+1}$  and  $D_{-1}$  is  $d_{\mathrm{TV}}(D_{+1}, D_{-1}) = m^{n-2} \cdot \frac{m^2c\varepsilon}{W} = \frac{m^n c\varepsilon}{W}$  because +1 and -1 have  $m^{n-2}$  different components. Since  $W \leq em^n$  (26), we have  $d_{\mathrm{TV}}(D_{+1}, D_{-1}) \geq \frac{c\varepsilon}{e} > 2\varepsilon$ . Consider the distinguishing distributions problem (defined in Section 3) where we want to distinguish  $D_{+1}$  and  $D_{-1}$ . If we can learn from samples a distribution  $\hat{D}$  that is  $\varepsilon$ -close in total variation distance to the unknown distribution  $D_{+1}$  or  $D_{-1}$ , then we can perfectly tell whether the unknown distribution is  $D_{+1}$  or  $D_{-1}$  because the two distributions are more than  $2\varepsilon$ -away from each other in total variation distance. Lemma A.5 implies that, to distinguish  $D_{+1}$  and  $D_{-1}$  with probability  $1 - \delta$ , the number of samples must be at least  $\Omega\left(\frac{\log(1 / \delta)}{d_{\mathrm{H}}^2(D_{+1}, D_{-1})}\right) = \Omega\left(\frac{\log(1 / \delta)}{\varepsilon^2}\right)$ . This proves the  $\Omega\left(\frac{\log(1 / \delta)}{\varepsilon^2}\right)$  part.
We then prove the  $\Omega\left(\frac{m^{n-2}}{\varepsilon^2}\right)$  part. Suppose we first draw the vector  $z$  from  $\{+1, -1\}^{m^{n-2}}$  uniformly at random, then draw  $T$  samples from  $D_z$ . We obtain the  $D_w$  as above. Let's consider the expected number of different components of  $w$  and  $z$  in this two-step random draw procedure:
$$
\underset {z, T \text {s a m p l e s}} {\mathbb {E}} [ \# ] = \mathbb {E} \left[ \sum_ {\boldsymbol {b} \in S ^ {n - 2}} \mathbb {1} \left\{z _ {\boldsymbol {b}} \neq w _ {\boldsymbol {b}} \right\} \right] = \sum_ {\boldsymbol {b} \in S ^ {n - 2}} \mathbb {E} \left[ \mathbb {1} \left\{z _ {\boldsymbol {b}} \neq w _ {\boldsymbol {b}} \right\} \right]. \tag {29}
$$
We consider each component  $\mathbb{E}\big[\mathbb{1}\{z_b\neq w_b\} \big]$  in the above summation. Suppose that, within the  $T$  samples drawn from  $D_{z},T_{b}$  of them fall into the bucket  $B_{b}$ . So,  $T_{b}$  follows the Binomial  $(T,D(B_{b}))$  distribution with
$$
D \left(B _ {b}\right) = \sum_ {\left(b, x, y\right) \in B _ {b}} D _ {z} (b, x, y) = \frac {1}{W} \sum_ {\left(b, x, y\right) \in B _ {b}} \gamma^ {\operatorname {n u m} \left(b, x, y\right)}.
$$
(Notice that the  $+\frac{z_b c \varepsilon}{W}$  and  $-\frac{z_b c \varepsilon}{W}$  cancel out in the summation and hence  $D(B_b)$  does not depend on  $z$ .) Let  $D_{z_b}$  denote the "  $B_b$ -part" of distribution  $D_z$ , namely,  $D_z$  conditioning on  $B_b$ :
$$
D _ {z _ {b}} (\boldsymbol {s}) = \frac {D _ {z} (\boldsymbol {s})}{D (B _ {b})}, \forall \boldsymbol {s} \in B _ {b}.
$$
We think of the random draw of the vector  $z$  and the  $T$  samples as follows: first, we draw  $T_{b}$  from Binomial  $(T, D(B_{b}))$ ; second, we draw  $z_{b} \in \{+1, -1\}$  uniformly at random; third, we draw  $T_{b}$  samples from the conditional distribution  $D_{z_{b}}$ ; forth, we draw the remaining vector  $z_{-b}$  and the remaining  $T - T_{b}$  samples (which are samples outside of  $B_{b}$ ). Only writing the first two steps explicitly, we have
$$
\begin{array}{l} \mathbb {E} \left[ \mathbb {1} \left\{z _ {\boldsymbol {b}} \neq w _ {\boldsymbol {b}} \right\} \right] = \mathbb {E} _ {T _ {\boldsymbol {b}}} \left[ \mathbb {E} _ {z _ {\boldsymbol {b}}} \left[ \mathbb {E} \left[ \mathbb {1} \left\{z _ {\boldsymbol {b}} \neq w _ {\boldsymbol {b}} \right\} \mid z _ {\boldsymbol {b}}, T _ {\boldsymbol {b}} \right] \right] \right] \\ = \mathbb {E} _ {T _ {b}} \left[ \frac {1}{2} \mathbb {E} \left[ \mathbb {1} \left\{z _ {b} \neq w _ {b} \right\} \mid z _ {b} = + 1, T _ {b} \right] + \frac {1}{2} \mathbb {E} \left[ \mathbb {1} \left\{z _ {b} \neq w _ {b} \right\} \mid z _ {b} = - 1, T _ {b} \right] \right] \\ = \underset {T _ {b}} {\mathbb {E}} \left[ \frac {1}{2} \Pr \left[ z _ {b} \neq w _ {b} \mid z _ {b} = + 1, T _ {b} \right] + \frac {1}{2} \Pr \left[ z _ {b} \neq w _ {b} \mid z _ {b} = - 1, T _ {b} \right] \right]. \tag {30} \\ \end{array}
$$
Claim E.5. For any  $T_{b}$ ,  $\frac{1}{2} \operatorname*{Pr}\left[z_{b} \neq w_{b} \mid z_{b} = +1, T_{b}\right] + \frac{1}{2} \operatorname*{Pr}\left[z_{b} \neq w_{b} \mid z_{b} = -1, T_{b}\right] \geq \frac{1}{2} - 2c\varepsilon \sqrt{T_{b}}$ .
Proof. We notice that  $\frac{1}{2}\operatorname*{Pr}\left[z_{b} \neq w_{b} \mid z_{b} = +1, T_{b}\right] + \frac{1}{2}\operatorname*{Pr}\left[z_{b} \neq w_{b} \mid z_{b} = -1, T_{b}\right]$  is the probability that we make a mistake when guessing the sign  $z_{b}$  using  $w_{b}$ , if (1)  $z_{b}$  is chosen from  $\{-1, +1\}$  uniformly at random; (2) we are given  $T_{b}$  samples from  $D_{z_{b}}$ ; (3) we then draw the remaining vector  $z_{-b}$  and the remaining samples; (4) finally, we use the  $D_{w}$  computed from all samples to get  $w_{b}$ . The steps (3) and (4) define a randomized function that maps the  $T_{b}$  samples of  $D_{z_{b}}$  to  $w_{b} \in \{0, 1\}$ , and therefore, according to the first item of Lemma A.5, we have
$$
\frac {1}{2} \Pr \left[ z _ {\boldsymbol {b}} \neq w _ {\boldsymbol {b}} \mid z _ {\boldsymbol {b}} = + 1, T _ {\boldsymbol {b}} \right] + \frac {1}{2} \Pr \left[ z _ {\boldsymbol {b}} \neq w _ {\boldsymbol {b}} \mid z _ {\boldsymbol {b}} = - 1, T _ {\boldsymbol {b}} \right] \geq \frac {1}{2} - \sqrt {\frac {T _ {\boldsymbol {b}}}{2}} d _ {\mathrm {H}} \left(D _ {z _ {\boldsymbol {b}} = + 1}, D _ {z _ {\boldsymbol {b}} = - 1}\right). \tag {31}
$$
Then we consider  $d_{\mathrm{H}}(D_{z_b = +1}, D_{z_b = -1})$ . We use Lemma A.2 to do so. For any  $s \in B_b$ , we have, on the one hand,
$$
\frac {D _ {z _ {b} = + 1} (\pmb {s})}{D _ {z _ {b} = - 1} (\pmb {s})} = \frac {\gamma^ {\mathrm {n u m} (\pmb {s})} \pm c \varepsilon}{\gamma^ {\mathrm {n u m} (\pmb {s})} \mp c \varepsilon} \geq \frac {\gamma^ {\mathrm {n u m} (\pmb {s})} - c \varepsilon}{\gamma^ {\mathrm {n u m} (\pmb {s})} + c \varepsilon} \geq 1 - 2 c \varepsilon ,
$$
because  $\frac{a - b}{a + b} = 1 - \frac{2b}{a + b} \geq 1 - 2b$  for  $a = \gamma^{\mathrm{num}(s)} \geq 1$ . On the other hand,
$$
\frac {D _ {z _ {b} = + 1} (\boldsymbol {s})}{D _ {z _ {b} = - 1} (\boldsymbol {s})} \leq \frac {\gamma^ {\mathrm {n u m} (\boldsymbol {s})} + c \varepsilon}{\gamma^ {\mathrm {n u m} (\boldsymbol {s})} - c \varepsilon} \leq 1 + 4 c \varepsilon ,
$$
because  $\frac{a + b}{a - b} = 1 + \frac{2b}{a - b} \leq 1 + 4b$  for  $a = \gamma^{\mathrm{num}(s)} \geq 1$  and  $b = c\varepsilon \leq \frac{1}{2}$ . Therefore, by Lemma A.2 we have
$$
d _ {\mathrm {H}} ^ {2} \left(D _ {z _ {b} = + 1}, D _ {z _ {b} = - 1}\right) \leq \frac {1}{2} (4 c \varepsilon) ^ {2} = 8 c ^ {2} \varepsilon^ {2}. \tag {32}
$$
Combining (31) and (32) proves our claim.
By (30) and Claim E.5, we have  $\mathbb{E}\big[\mathbb{1}\{z_{\pmb{b}}\neq w_{\pmb{b}}\} \big]\geq \mathbb{E}_{T_{\pmb{b}}}\big[\frac{1}{2} -2c\varepsilon \sqrt{T_{\pmb{b}}} \big]$ . Summing over  $\pmb {b}\in S^{n - 2}$ , (29) becomes
$$
\begin{array}{l} \sum_ {\boldsymbol {b} \in S ^ {n - 2}} \mathbb {E} \left[ \mathbb {1} \left\{z _ {\boldsymbol {b}} \neq w _ {\boldsymbol {b}} \right\} \right] \geq \sum_ {\boldsymbol {b} \in S ^ {n - 2}} \mathbb {E} _ {T _ {\boldsymbol {b}}} \left[ \frac {1}{2} - 2 c \varepsilon \sqrt {T _ {\boldsymbol {b}}} \right] = \frac {m ^ {n - 2}}{2} - 2 c \varepsilon \sum_ {\boldsymbol {b} \in S ^ {n - 2}} \mathbb {E} [ \sqrt {T _ {\boldsymbol {b}}} ] \\ \text {(b y)} \mathbb {E} [ \sqrt {X} ] \leq \sqrt {\mathbb {E} [ X ]}) \geq \frac {m ^ {n - 2}}{2} - 2 c \varepsilon \sum_ {\boldsymbol {b} \in S ^ {n - 2}} \sqrt {\mathbb {E} [ T _ {\boldsymbol {b}} ]}. \\ \end{array}
$$
Because  $\mathbb{E}[T_{\pmb{b}}] = T\cdot D(B_{\pmb{b}}) = \frac{T}{W}\sum_{(\pmb {b},x,y)\in B_{\pmb{b}}}\gamma^{\mathrm{num}(\pmb {b},x,y)}\leq \frac{T}{W} m^{2}\gamma^{m^{n}}$  , we have
$$
\sum_ {\boldsymbol {b} \in S ^ {n - 2}} \mathbb {E} \left[ \mathbb {1} \left\{z _ {\boldsymbol {b}} \neq w _ {\boldsymbol {b}} \right\} \right] \geq \frac {m ^ {n - 2}}{2} - 2 c \varepsilon \sum_ {\boldsymbol {b} \in S ^ {n - 2}} \sqrt {\frac {T}{W} m ^ {2} \gamma^ {m n}} = \frac {m ^ {n - 2}}{2} - 2 c \varepsilon m ^ {n - 2} \sqrt {\frac {T}{W} m ^ {2} \gamma^ {m n}}. \tag {33}
$$
Now, let's consider the probability with which we can obtain  $\hat{D}$  such that  $d_{\mathrm{TV}}(\hat{D}, D_z) \leq \varepsilon$ . We will show that this probability is at most  $0.99 < 1 - \delta$  if  $T$  is less than  $10^{-5} \cdot \frac{m^{n-2}}{\varepsilon^2}$ . Recall from Claim E.4 that  $d_{\mathrm{TV}}(\hat{D}, D_z) \leq \varepsilon$  implies  $\# = \sum_{b \in S^{n-2}} \mathbb{1}\{z_b \neq w_b\} \leq \frac{2W}{cm^2}$ . So, the probability is at most
$$
\Pr \left[ \sum_ {\boldsymbol {b} \in S ^ {n - 2}} \mathbb {1} \left\{z _ {\boldsymbol {b}} \neq w _ {\boldsymbol {b}} \right\} \leq \frac {2 W}{c m ^ {2}} \right] = \Pr \left[ \sum_ {\boldsymbol {b} \in S ^ {n - 2}} \mathbb {1} \left\{z _ {\boldsymbol {b}} = w _ {\boldsymbol {b}} \right\} \geq m ^ {n - 2} - \frac {2 W}{c m ^ {2}} \right]
$$
$$
\text {(b y} \quad \text {M a r k o v ’ s i n e q u a l i t y)} \leq \frac {\mathbb {E} \left[ \sum_ {\boldsymbol {b} \in S ^ {n - 2}} \mathbb {1} \left\{z _ {\boldsymbol {b}} = w _ {\boldsymbol {b}} \right\} \right]}{m ^ {n - 2} - \frac {2 W}{c m ^ {2}}}
$$
$$
\text {(b y (3 3))} \leq \frac {\frac {m ^ {n - 2}}{2} + 2 c \varepsilon m ^ {n - 2} \sqrt {\frac {T}{W} m ^ {2} \gamma^ {m n}}}{m ^ {n - 2} - \frac {2 W}{c m ^ {2}}}
$$
$$
\begin{array}{l} \left(\gamma^ {m ^ {n}} \leq e \text {a n d} m ^ {n} \leq W \leq e m ^ {n} \text {b y (2 6)}\right) \leq \frac {\frac {m ^ {n - 2}}{2} + 2 c \varepsilon m ^ {n - 2} \sqrt {\frac {e T}{m ^ {n - 2}}}}{m ^ {n - 2} - \frac {2 e m ^ {n}}{c m ^ {2}}} \\ = \frac {\frac {1}{2} + 2 c \sqrt {\frac {e \varepsilon^ {2}}{m ^ {n - 2}} T}}{1 - \frac {2 e}{c}} <   0. 9 9 <   1 - \delta , \\ \end{array}
$$
when  $c = 20, T < 10^{-5} \cdot \frac{m^{n - 2}}{\varepsilon^2}$ , and  $\delta < 0.01$ . This means that, in order to obtain such  $\hat{D}$  with probability at least  $1 - \delta$ , at least  $10^{-5} \cdot \frac{m^{n - 2}}{\varepsilon^2}$  samples are needed.
F Missing Proofs from Section 5
F.1 Proof of Theorem 5.2: the  $\Omega\left(\frac{1}{\varepsilon} \log \frac{1}{\delta}\right)$  Lower Bound
The proof uses a reduction from the distinguishing distributions problem (defined in Section 3). We construct two conditionally independent distributions  $P^1, P^2$  over the space  $\Omega \times S_1 \times \dots \times S_n$  with each  $|S_i| = 2$ ,  $S_i = \{a, b\}$ . Given  $T$  samples from either  $P^1$  or  $P^2$ , we want to tell which distribution the samples are coming from. We will show that, if we can solve the forecast aggregation problem, then we can distinguish the two distributions (with high probability), which requires  $T = \Omega \left( \frac{1}{d_{\mathrm{H}}^2(P_1, P_2)} \log \frac{1}{\delta} \right) = \Omega \left( \frac{1}{\varepsilon} \log \frac{1}{\delta} \right)$  samples according to Lemma A.5.