SlowGuess/NeurIPS_2023_OCR huggingface dataset:
In this subsection we prove the  $O\left(\frac{m^n + \log(1 / \delta)}{\varepsilon^2}\right)$  upper bound in Theorem 4.1. This is a direct corollary of the distribution learning result introduced in Section 3.
We regard  $P$  as a distribution over  $\mathbf{r}$  and  $\omega$  instead of over  $s$  and  $\omega$ . Then  $P$  is a discrete distribution with support size at most  $2m^n$  because each possible report  $r_i \in [0,1]$  corresponds to some discrete signal  $s_i$  in  $S_i$ . Let  $\hat{P}_{\mathrm{emp}}$  be the empirical distribution of reports and event realizations:  $\hat{P}_{\mathrm{emp}} = \mathrm{Uniform}\bigl\{(\pmb{r}^{(1)},\omega^{(1)}),\dots ,(\pmb{r}^{(T)},\omega^{(T)})\bigr\}$ . By Proposition 3.3, with probability at least  $1 - \delta$  over the random draw of  $T = O\big(\frac{2m^n + \log(1 / \delta)}{\varepsilon^2}\big)$  samples, we have  $d_{\mathrm{TV}}(\hat{P}_{\mathrm{emp}},P) \leq \varepsilon$ . According to Fact 3.2, and by the definition of  $L_P(f)$ , we have: for any aggregator  $f:[0,1]^n \to [0,1]$ ,
$$
\left| L _ {\hat {P} _ {\mathrm {e m p}}} (f) - L _ {P} (f) \right| = \left| \mathbb {E} _ {\hat {P} _ {\mathrm {e m p}}} \left[ | f (\boldsymbol {r}) - \omega | ^ {2} \right] - \mathbb {E} _ {P} \left[ | f (\boldsymbol {r}) - \omega | ^ {2} \right] \right| \leq d _ {\mathrm {T V}} (\hat {P} _ {\mathrm {e m p}}, P) \leq \varepsilon .
$$
Therefore, if we pick the empirically optimal aggregator  $\hat{f}_{\mathrm{emp}} = \mathrm{argmin}_f L_{\hat{P}_{\mathrm{emp}}}(\bar{f})$ , we get
$$
L _ {P} (\hat {f} _ {\mathrm {e m p}}) \leq L _ {\hat {P} _ {\mathrm {e m p}}} (\hat {f} _ {\mathrm {e m p}}) + \varepsilon \leq L _ {\hat {P} _ {\mathrm {e m p}}} (f ^ {*}) + \varepsilon \leq L _ {P} (f ^ {*}) + 2 \varepsilon ,
$$
which means that  $\hat{f}_{\mathrm{emp}}$  is a  $2\varepsilon$ -optimal aggregator for  $P$ .
4.2 Proof of the Lower Bound
In this subsection we prove the  $\Omega\left(\frac{m^{n-2} + \log(1/\delta)}{\varepsilon}\right)$  lower bound in Theorem 4.1. The main idea is a reduction from the distribution learning problem (defined in Section 3) for a specific family  $\mathcal{D}$  of distributions over the joint signal space  $\pmb{S} = S_1 \times \dots \times S_n$ . We construct a corresponding family of distributions  $\mathcal{P} = \{P_D : D \in \mathcal{D}\}$  for the forecast aggregation problem, such that, if we can obtain an  $\varepsilon$ -optimal aggregator  $\hat{f}$  for  $P_D$ , then we can convert  $\hat{f}$  into a distribution  $\hat{D}$  such that  $d_{\mathrm{TV}}(\hat{D}, D) \leq O(\sqrt{\varepsilon})$ . We then prove that learning  $\mathcal{D}$  within total variation distance  $\varepsilon_{\mathrm{TV}} = O(\sqrt{\varepsilon})$  requires  $\Omega\left(\frac{m^{n-2} + \log(1/\delta)}{\varepsilon_{\mathrm{TV}}^2}\right) = \Omega\left(\frac{m^{n-2} + \log(1/\delta)}{\varepsilon}\right)$  samples. This gives the sample complexity lower bound for the forecast aggregation problem for  $\mathcal{P}$  (and hence  $\mathcal{P}_{\mathrm{all}}$ ).
We will need a family of distributions  $\mathcal{D}$  that satisfies the following three properties:
Definition 4.2. We say a family of distributions  $\mathcal{D}$  satisfies
1.  $B$ -uniformly bounded, if:  $D(\boldsymbol{s}) \leq \frac{B}{|\boldsymbol{S}|} = \frac{B}{m^n}, \forall \boldsymbol{s} \in \boldsymbol{S}, \forall D \in \mathcal{D}$ , where  $B \geq 1$  is a constant.
2. same marginal across distributions, if: for any  $D$ ,  $D' \in \mathcal{D}$ , any  $i$ , any  $s_i \in S_i$ ,  $D(s_i) = D'(s_i)$ .
3. distinct marginals across signals, if: for any  $D \in \mathcal{D}$ , any  $i$ , any  $s_i \neq s_i' \in S_i$ ,  $D(s_i) \neq D(s_i')$ .
How do we construct the family  $\mathcal{P}$ ? For each distribution  $D\in \mathcal{D}$ , we construct distribution  $P_{D}$  as follows: the marginal distribution of  $\omega$  is Uniform  $\{0,1\}$ , i.e.,  $P_D(\omega = 0) = P_D(\omega = 1) = \frac{1}{2}$ ; conditioning on  $\omega = 0$ , the joint signal  $s$  is uniformly distributed:  $P_{D}(s\mid \omega = 0) = \frac{1}{|\pmb{S}|} = \frac{1}{m^{n}}$ ,  $\forall s\in S$ ; conditioning on  $\omega = 1$ , the joint signal is distributed according to  $D$ :  $P_{D}(s\mid \omega = 1) = D(s),\forall s\in S$ . The family  $\mathcal{P}$  is  $\{P_D:D\in \mathcal{D}\}$ .
We show that if we can obtain  $\varepsilon$ -optimal aggregators for distributions in  $\mathcal{P}$ , then we can learn the distributions in  $\mathcal{D}$  within total variation distance  $(1 + B)^2\sqrt{\varepsilon}$ , and thus the sample complexity of the former is lower bounded by the sample complexity of the latter:
Lemma 4.3. Let  $\mathcal{D}$  be a family of distributions that satisfies the three properties in Definition 4.2. Let  $\mathcal{P} = \{P_{\mathcal{D}}: D \in \mathcal{D}\}$  be defined above. Then,  $T_{\mathcal{P}}(\varepsilon, \delta) \geq T_{\mathcal{D}}^{\mathrm{TV}}((1 + B)^2 \sqrt{\varepsilon}, \delta)$ .
Proof sketch of Lemma 4.3. The full proof is in Appendix E.1. We give a sketch here. According to the definition of  $P_D$ , by Bayes' rule, we have
$$
P _ {D} (\omega = 1 \mid \boldsymbol {s}) = \frac {\frac {1}{2} P _ {D} (\boldsymbol {s} \mid \omega = 1)}{\frac {1}{2} P _ {D} (\boldsymbol {s} \mid \omega = 0) + \frac {1}{2} P _ {D} (\boldsymbol {s} \mid \omega = 1)} = \frac {D (\boldsymbol {s})}{\frac {1}{m ^ {n}} + D (\boldsymbol {s})}. \tag {6}
$$
The "distinct marginals across signals" property in Definition 4.2 ensures that there is a one-to-one mapping between signal  $s_i$  and report  $r_i = P_D(\omega = 1|s_i)$ , and hence a one-to-one mapping between joint signal  $s$  and joint report  $r$ . So, the Bayesian aggregator  $f^*$  satisfies  $f^*(\boldsymbol{r}) = P_D(\omega = 1|\boldsymbol{r}) = P_D(\omega = 1|\boldsymbol{s}) = \frac{D(\boldsymbol{s})}{1/m^n + D(\boldsymbol{s})}$ . This gives
$$
D (\boldsymbol {s}) = \frac {1}{m ^ {n}} \frac {f ^ {*} (\boldsymbol {r})}{1 - f ^ {*} (\boldsymbol {r})}. \tag {7}
$$
Suppose we have obtained an  $\varepsilon$ -optimal aggregator  $\hat{f}$  for  $P_{D}, \mathbb{E}_{P_{D}}\left[|\hat{f}(\boldsymbol{r}) - f^{*}(\boldsymbol{r})|^{2}\right] \leq \varepsilon$ , then we convert  $\hat{f}$  into  $\hat{D}$  by letting  $\hat{D}(s) = \frac{1}{m^{n}} \frac{\hat{f}(\boldsymbol{r})}{1 - \hat{f}(\boldsymbol{r})}$ . The total variation distance between  $\hat{D}$  and  $D$  is:
$$
d _ {\mathrm {T V}} (\hat {D}, D) = \frac {1}{2} \sum_ {\boldsymbol {s} \in \boldsymbol {S}} | \hat {D} (\boldsymbol {s}) - D (\boldsymbol {s}) | \overset {(7)} {=} \frac {1}{2} \sum_ {\boldsymbol {s} \in \boldsymbol {S}} \frac {1}{m ^ {n}} \left| \frac {\hat {f} (\boldsymbol {r})}{1 - \hat {f} (\boldsymbol {r})} - \frac {f ^ {*} (\boldsymbol {r})}{1 - f ^ {*} (\boldsymbol {r})} \right|. \tag {8}
$$
The "  $B$ -uniformly bounded" property in Definition 4.2 ensures  $D(\pmb{s}) = O(\frac{1}{m^n})$ , which has two consequences: (1)  $P_{D}(\pmb{s}) = O(\frac{1}{m^{n}})$ ; (2)  $f^{*}(\pmb{r}) = O(1)$ , which implies  $\left|\frac{\hat{f}(\pmb{r})}{1 - \hat{f}(\pmb{r})} - \frac{f^{*}(\pmb{r})}{1 - f^{*}(\pmb{r})}\right| = O\big(|\hat{f}(\pmb{r}) - f^{*}(\pmb{r})|\big)$  due to Lipschitz property of the function  $\frac{x}{1 - x}$  for  $x = O(1)$ . These two consequences imply
$$
\begin{array}{l} d _ {\mathrm {T V}} (\hat {D}, D) = O \left(\sum_ {\boldsymbol {s} \in \boldsymbol {\mathcal {S}}} \frac {1}{m ^ {n}} | \hat {f} (\boldsymbol {r}) - f ^ {*} (\boldsymbol {r}) |\right) = O \left(\sum_ {\boldsymbol {s} \in \boldsymbol {\mathcal {S}}} P _ {D} (\boldsymbol {s}) | \hat {f} (\boldsymbol {r}) - f ^ {*} (\boldsymbol {r}) |\right) \\ = O \left(\mathbb {E} \left[ | \hat {f} (\boldsymbol {r}) - f ^ {*} (\boldsymbol {r}) | \right]\right) \overset {\text {J e n s e n ' s i n e q u a l i t y}} {\leq} O \left(\sqrt {\mathbb {E} \left[ | \hat {f} (\boldsymbol {r}) - f ^ {*} (\boldsymbol {r}) | ^ {2} \right]}\right) = O (\sqrt {\varepsilon}). \\ \end{array}
$$
So, we obtain  $d_{\mathrm{TV}}(\hat{D},D)\leq O(\sqrt{\varepsilon}) = (1 + B)^2\sqrt{\varepsilon}$
There is a subtlety, however: In the distribution learning problem we are given samples from  $D$ , which are signals  $\{s^{(t)}\}_{t=1}^T$ . We need to convert them into the corresponding reports  $\{r^{(t)}\}_{t=1}^T$  as the samples for the forecast aggregation problem. To do this we need to know  $P_D$  or  $D$ , which we do not know. So, we make use of the "same marginal across distributions" property in Definition 4.2 here: because all the distributions  $D \in \mathcal{D}$  have the same marginal probability  $D(s_i) = D'(s_i)$  (but possibly different joint probabilities  $D(s) \neq D'(s)$ ), we are able to compute the report
$$
r _ {i} ^ {(t)} = P _ {D} (\omega = 1 \mid s _ {i} ^ {(t)}) = \frac {\frac {1}{2} P _ {D} (s _ {i} ^ {(t)} \mid \omega = 1)}{\frac {1}{2} P _ {D} (s _ {i} ^ {(t)} \mid \omega = 0) + \frac {1}{2} P _ {D} (s _ {i} ^ {(t)} \mid \omega = 1)} = \frac {D (s _ {i} ^ {(t)})}{\frac {1}{m} + D (s _ {i} ^ {(t)})},
$$
separately for each expert  $i$  without knowing  $D$ , since we know what the family  $\mathcal{D}$  is. This allows us to reduce the distribution learning problem for  $\mathcal{D}$  to the forecast aggregation problem for  $\mathcal{P}$ .
Then, we find a family of distributions  $\mathcal{D}$  that satisfies the three properties in Definition 4.2 and requires many samples to learn.
Proposition 4.4. There exists a family of distributions  $\mathcal{D}$  that satisfies the three properties in Definition 4.2 (with  $B = e + \frac{1}{2}$ ) and requires  $T_{\mathcal{D}}^{\mathrm{TV}}(\varepsilon_{\mathrm{TV}},\delta) = \Omega \left(\frac{m^{n - 2} + \log(1 / \delta)}{\varepsilon_{\mathrm{TV}}^2}\right)$  samples to learn.
The above sample complexity is smaller than the  $\Omega\left(\frac{|\mathcal{S}| + \log(1 / \delta)}{\varepsilon_{\mathrm{TV}}^2}\right)$  lower bound in Proposition 3.3 because we are restricting to a smaller set of distributions than the set of all distributions over  $\mathcal{S}$ . The proof of Proposition 4.4 is analogous to a textbook proof of Proposition 3.3, which uses reductions from the distinguishing distributions problem. See details in Appendix E.2.
Finishing the proof of Theorem 4.1: By Lemma 4.3 and Proposition 4.4, plugging in  $\varepsilon_{\mathrm{TV}} = (1 + B)^2\sqrt{\varepsilon}$  with  $B = e + \frac{1}{2}$ , we obtain the lower bound on the sample complexity of forecast aggregation for  $\mathcal{P}$  (and hence for  $\mathcal{P}_{\mathrm{all}}$ ):
$$
T _ {\mathcal {P}} (\varepsilon , \delta) \geq T _ {\mathcal {D}} ^ {\mathrm {T V}} \big ((1 + B) ^ {2} \sqrt {\varepsilon}, \delta \big) = \Omega \Big (\frac {m ^ {n - 2} + \log (1 / \delta)}{\big ((1 + B) ^ {2} \sqrt {\varepsilon} \big) ^ {2}} \Big) = \Omega \Big (\frac {m ^ {n - 2} + \log (1 / \delta)}{\varepsilon} \Big).
$$
5 Sample Complexity for Conditionally Independent Distributions
Section 4 proved that learning  $\varepsilon$ -optimal aggregators for all discrete distributions needs exponentially many samples. As shown in the proof, this large sample complexity is because the experts' signals can be arbitrarily correlated conditioned on the event  $\omega$ . Accurate estimation of such correlation requires many samples. So, in this section we restrict attention to the case where the experts' signals are conditionally independent. It turns out that an  $\varepsilon$ -optimal aggregator can be learned using only  $O\left(\frac{1}{\varepsilon^2} \log \frac{1}{\varepsilon \delta}\right)$  samples in this case, which does not depend on  $n$ . The assumption of discrete signal space can be relaxed here. We also investigate two special and interesting families of conditionally independent distributions that admit an even smaller sample complexity of  $O\left(\frac{1}{\varepsilon} \log \frac{1}{\delta}\right)$ .
5.1 General Conditionally Independent Distributions
Let  $P$  be a conditionally independent distribution over  $S \times \Omega$ , namely,  $P(s \mid \omega) = \prod_{i=1}^{n} P(s_i \mid \omega)$  for all  $s \in S$ , for  $\omega \in \{0,1\}$ . Here,  $S_i$  can be a continuous space, in which case,  $P(\cdot \mid \omega)$  represents a density function. We introduce some additional notations. Let  $p = P(\omega = 1)$  be the prior probability of  $\omega = 1$ . For technical convenience we assume  $p \in (0,1)$ . Define
$$
\rho = \frac {p}{1 - p} = \frac {P (\omega = 1)}{P (\omega = 0)} \in (0, + \infty). \tag {9}
$$
We will be working with ratios like  $\frac{r_i}{1 - r_i}$  and  $\frac{p}{1 - p}$  a lot in this section. We will use the following characterization of the optimal aggregator  $f^*$  for conditionally independent distributions:
Lemma 5.1 ([8]). For conditionally independent distribution  $P$ , given signals  $\boldsymbol{s} = (s_i)_{i=1}^n$ , with corresponding reports  $\boldsymbol{r} = (r_i)_{i=1}^n$  where  $r_i = P(\omega = 1 | s_i)$ , the posterior probability of  $\omega = 1$  is:
$$
f ^ {*} (\boldsymbol {r}) = P (\omega = 1 \mid \boldsymbol {r}) = P (\omega = 1 \mid \boldsymbol {s}) = \frac {1}{1 + \rho^ {n - 1} \prod_ {i = 1} ^ {n} \frac {1 - r _ {i}}{r _ {i}}}. \tag {10}
$$
(Define  $f^{*}(\boldsymbol{r}) = 0$  if  $\rho^{n-1} \prod_{i=1}^{n} \frac{1 - r_i}{r_i} = +\infty$ .)
Lemma 5.1 implies that one way to learn  $f^{*}$  is to simply learn the value of  $\rho$ . If we can learn  $\rho$  with accuracy  $\frac{\sqrt{\varepsilon}}{n}$ , then we can learn  $\rho^{n-1}$  with accuracy  $\sqrt{\varepsilon}$  and obtain an  $\hat{f}$  that is  $\varepsilon$ -close to  $f^{*}$  for every possible input  $\boldsymbol{r} \in [0,1]^n$ . However, by standard concentration inequalities, learning  $\rho$  with accuracy  $\frac{\sqrt{\varepsilon}}{n}$  requires  $\tilde{O}\left(\frac{n^2}{\rho\varepsilon}\right)$  samples, which is larger than the  $\tilde{O}\left(\frac{1}{\varepsilon^2}\right)$  bound we will prove. The key is that we do not actually need  $\hat{f}(\boldsymbol{r})$  to be close to  $f^{*}(\boldsymbol{r})$  for every  $\boldsymbol{r} \in [0,1]_+^n$ ; we only need the expectation  $\mathbb{E}\left[|\hat{f}(\boldsymbol{r}) - f^{*}(\boldsymbol{r})|^2\right] \leq \varepsilon$ . This allows us to prove a smaller sample complexity bound, using a pseudo-dimension argument.
The main result of this section is that the sample complexity of forecast aggregation with respect to all conditionally independent distributions is between  $\Omega\left(\frac{1}{\varepsilon} \log \frac{1}{\delta}\right)$  and  $O\left(\frac{1}{\varepsilon^2} \log \frac{1}{\varepsilon \delta}\right)$ :
Theorem 5.2. Let  $\mathcal{P}_{\mathrm{ind}}$  be the set of all conditionally independent distributions over  $S\times \Omega$ . Suppose  $n\geq 2$ . The sample complexity of forecast aggregation with respect to  $\mathcal{P}_{\mathrm{ind}}$  is
$$
O \left(\frac {1}{\varepsilon^ {2}} \log \frac {1}{\varepsilon \delta}\right) \geq T _ {\mathcal {P} _ {\text {i n d}}} (\varepsilon , \delta) \geq \Omega \left(\frac {1}{\varepsilon} \log \frac {1}{\delta}\right). \tag {11}
$$
We provide the main ideas of the proof of Theorem 5.2 here. The upper bound  $O\left(\frac{1}{\varepsilon^2}\log \frac{1}{\varepsilon\delta}\right)$  is a corollary of our theorem for multi-outcome events (Theorem C.1), so we only give a sketch here. We note that, according to Lemma 5.1, the optimal aggregator has the form  $f^{*}(\boldsymbol {r}) = \frac{1}{1 + \rho^{n - 1}\prod_{i = 1}^{n}\frac{1 - r_{i}}{r_{i}}}$ .
We consider the class of aggregators  $\mathcal{F} = \left\{f^{\theta}:f^{\theta}(\boldsymbol {r}) = \frac{1}{1 + \theta^{n - 1}\prod_{i = 1}^{n}\frac{1 - r_i}{r_i}}\right\}$  parameterized by  $\theta \in (0, + \infty)$ . The class of loss functions  $\mathcal{G} = \left\{g^{\theta}:g^{\theta}(\boldsymbol {r},\omega) = |f^{\theta}(\boldsymbol {r}) - \omega |^{2}\right\}$  associated with  $\mathcal{F}$  has pseudo-dimension  $\mathrm{Pdim}(\mathcal{G}) = O(1)$ . By the known result (e.g., [1]) that the pseudo-dimension gives a sample complexity upper bound on the uniform convergence of a class of functions, we conclude that the empirically optimal aggregator in  $\mathcal{F}$  must be  $O(\varepsilon)$ -optimal on the true distribution (with probability at least  $1 - \delta$ ), given  $O\big(\frac{1}{\varepsilon^2} (\mathrm{Pdim}(\mathcal{G})\log \frac{1}{\varepsilon} +\log \frac{1}{\delta})\big) = O\big(\frac{1}{\varepsilon^2}\log \frac{1}{\varepsilon\delta}\big)$  samples.
We prove the  $\Omega\left(\frac{1}{\varepsilon}\log \frac{1}{\delta}\right)$  lower bound by a reduction from the distinguishing distributions problem (introduced in Section 3). We construct two conditionally independent distributions  $P^1, P^2$  over
the space  $\Omega \times \mathcal{S}$  that differ by  $d_{\mathrm{H}}^2 (P^1,P^2) = O(\varepsilon)$  in squared Hellinger distance. Specifically,  $P^1$  has prior  $P^{1}(\omega = 1) = 0.5 - O(\frac{1}{n}) + O(\frac{\sqrt{\varepsilon}}{n})$  and  $P^2$  has prior  $P^2 (\omega = 1) = 0.5 - O(\frac{1}{n}) - O(\frac{\sqrt{\varepsilon}}{n})$ ; the conditional distributions of each signal,  $P^1 (s_i|\omega)$  and  $P^2 (s_i|\omega)$ , differ by  $O(\frac{\varepsilon}{n})$  in squared Hellinger distance; taking the product of  $n$  signals,  $P^1 (s|\omega)$  and  $P^2 (s|\omega)$  differ by  $O(\varepsilon)$ . The distinguishing distributions problem asks: given  $T$  samples from either  $P^1$  or  $P^2$ , tell which distribution the samples come from. We show that, if we can solve the forecast aggregation problem, namely,  $\varepsilon$ -approximate  $f^{*}(\boldsymbol {r}) = \frac{1}{1 + \rho^{n - 1}\prod_{i = 1}^{n}\frac{1 - r_{i}}{r_{i}}}$ , then we can estimate  $\rho$  with accuracy  $O(\frac{\sqrt{\varepsilon}}{n})$ , and hence distinguish  $P^1$  and  $P^2$ . But distinguishing  $P^1$  and  $P^2$  requires  $\Omega (\frac{1}{d_{\mathrm{H}}^2(P^1,P^2)}\log \frac{1}{\delta}) = \Omega (\frac{1}{\varepsilon}\log \frac{1}{\delta})$  samples. This gives the lower bound we claimed. See details in Appendix F.1.
5.2 Strongly and Weakly Informative Experts
While the sample complexity of forecast aggregation for general conditionally independent distributions is  $O\left(\frac{1}{\varepsilon^2}\log \frac{1}{\varepsilon\delta}\right)$ , under further assumptions this bound can be improved. In particular, we find two special yet natural families of conditionally independent distributions that admit  $O\left(\frac{1}{\varepsilon}\log \frac{1}{\delta}\right)$  sample complexity. In these two cases, the experts are either "very informative" or "very noninformative". Roughly speaking, an expert is "very informative" if the conditional distributions of the expert's signal under event  $\omega = 0$  and  $\omega = 1$  are significantly different, so the expert's prediction  $r_i$  is away from the prior  $p$ . An expert is "very non-informative" if the opposite is true. Intuitively, an expert being informative should help aggregation and hence reduce the sample complexity. Interestingly though, we show that even if the experts are non-informative the sample complexity of forecast aggregation can also be reduced. See details in Appendix B.
6 Extension: Multi-Outcome Events
Our main results regarding the sample complexity of forecast aggregation for binary events (Theorems 4.1 and 5.2) can be generalized to multi-outcome events with  $|\Omega| > 2$ . We prove that: for general distributions, the sample complexity is  $\tilde{\Omega}\left(\frac{m^{n-2}}{\varepsilon}\right) = \Omega\left(\frac{m^{n-2} + \log(1/\delta)}{\varepsilon}\right) \leq T_{\mathcal{P}}(\varepsilon, \delta) \leq O\left(\frac{|\Omega|m^n + \log(1/\delta)}{\varepsilon^2}\right) = \tilde{O}\left(\frac{|\Omega|m^n}{\varepsilon^2}\right)$ ; for conditionally independent distributions, the sample complexity is  $\tilde{\Omega}\left(\frac{1}{\varepsilon}\right) = \Omega\left(\frac{1}{\varepsilon}\log \frac{1}{\delta}\right) \leq T_{\mathcal{P}_{\mathrm{ind}}}(\varepsilon, \delta) \leq O\left(\frac{|\Omega| \log |\Omega|}{\varepsilon^2}\log \frac{1}{\varepsilon} + \frac{1}{\varepsilon^2}\log \frac{1}{\delta}\right) = \tilde{O}\left(\frac{|\Omega|}{\varepsilon^2}\right)$ . See Appendix C for details.
In this work, we showed an exponential gap between the sample complexity of forecast aggregation for general distributions,  $\tilde{\Omega}\left(\frac{m^{n-2}}{\varepsilon}\right)$ , and conditionally independent distributions,  $\tilde{O}\left(\frac{1}{\varepsilon^2}\right)$ . This gap is due to the need of estimating the conditional correlation between experts in the general case, which is not needed in the conditional independence case. Notably, the bound  $\tilde{O}\left(\frac{1}{\varepsilon^2}\right)$  for conditionally independent distributions does not depend on the number of experts.
We discuss the dependency of the sample complexity on  $\varepsilon$  and other directions for future works:
The dependency on  $\varepsilon$  An open question left by our work is the dependency of the sample complexity on the parameter  $\varepsilon$ . We conjecture that the tight dependency should be  $\frac{1}{\varepsilon}$  (so our lower bounds are tight). This is supported by the following evidence:
Theorem 7.1. For the case of  $|\Omega| = 2$  and for general distributions, if the distribution  $P$  has a minimum joint probability  $\min_{(s,\omega)\in S\times \Omega}P(s,\omega) > \frac{c}{m^n}$  for some  $c > 0$ , then the sample complexity of forecast aggregation is at most  $O\Big(\frac{m^n}{c\varepsilon} (n\log m + \log \frac{1}{\delta})\Big) = \tilde{O}\Big(\frac{nm^n}{c\varepsilon}\Big).5$
In particular, this theorem can be applied to distributions that are close to uniform, where  $P(s,\omega) \approx \frac{1}{|\pmb{S} \times \Omega|} = \frac{1}{2m^n}$ , giving a bound of  $\tilde{O}\left(\frac{nm^n}{\varepsilon}\right)$ . Notably, the set of distributions we constructed in the proof of the  $\tilde{\Omega}\left(\frac{m^{n-2}}{\varepsilon}\right)$  lower bound in Theorem 4.1 are also close to uniform. This means that
This bound has a better dependency on  $\varepsilon$  but worse on  $n$  than the  $O\left(\frac{m^n}{\varepsilon^2}\right)$  bound in Theorem 4.1.
close-to-uniform distributions have a tight sample complexity bound of the form  $\Theta\left(\frac{f(n,m)}{\varepsilon}\right)$ , not  $\Theta\left(\frac{f(n,m)}{\varepsilon^2}\right)$ . Moreover, since close-to-uniform distributions are the "most difficult" distributions to learn in the distribution learning problem, it is likely that they are also the most difficult distributions for the forecast aggregation problem, and therefore the tight sample complexity of forecast aggregation should be determined by the sample complexity for those distributions, which is  $\Theta\left(\frac{f(n,m)}{\varepsilon}\right)$ .
Other future directions