SlowGuess/NeurIPS_2023_OCR huggingface dataset:
- The middle ground between fully correlated experts and conditionally independent experts: An example is the partial evidence model in [4]. Applying [4]'s results, one can show that the sample complexity of forecast aggregation in the partial evidence model is at most  $\tilde{O}\left(\frac{n^2}{\varepsilon^2}\right)$ .<sup>6</sup> Giving a lower bound for the partial evidence model and exploring other intermediate models is open.
- Weaker benchmark: Since the Bayesian aggregator needs exponentially many samples to approximate, can we find a weaker yet meaningful benchmark with a small sample complexity?
- Samples vs experts: In reality, obtaining samples of experts' historical forecasts can be difficult, while recruiting experts is easy. Can we achieve better aggregation by recruiting more experts instead of collecting more samples? How many experts do we need?
- Eliciting more information: Previous works on information elicitation and aggregation have noticed that better aggregation can be achieved by eliciting more information than agents' own predictions, for example, also eliciting each agent's prediction about other agents' predictions (e.g., [41, 47, 31, 14]). One can ask whether and how eliciting more information can help to reduce the sample complexity of information aggregation.
- Continuous distributions: In our model the random variable  $\omega$  to be predicted is discrete. One can study a setting where  $\omega$  is a continuous random variable and the experts report, e.g., the means of their posterior beliefs about  $\omega$ . The results for continuous random variables might be very different from the results in this work.
- Other loss functions: We focused on the squared loss  $\mathbb{E}\big[|f(\boldsymbol{r}) - \omega|^2\big]$  due to its popularity in machine learning problems and its useful property that the difference between the squared losses of any aggregator and the optimal aggregator is equal to their expected squared difference (Lemma 2.1). Alternatively, one can consider other loss functions like the logarithmic loss  $\mathbb{E}[\omega \log(f(\boldsymbol{r})) + (1 - \omega)\log(1 - f(\boldsymbol{r}))]$  and the absolute loss  $\mathbb{E}[|f(\boldsymbol{r}) - \omega|]$ . There might be some technical challenges in the analysis of sample complexity for those loss functions, though: e.g., the logarithmic loss can be unbounded [4, 40] and the absolute loss does not enjoy a property like Lemma 2.1.
We would like to thank Yannai Gonczarowski, Ariel Procaccia, Milind Tambe, David Parkes, Bo Waggoner, Rafael Frongillo, Grant Schoenebeck, Yuqing Kong, and anonymous reviewers for their helpful comments. This research is based upon work supported in part by the National Science Foundation under grants no. IIS-2007887 and no. IIS-2147187.
[1] Martin Anthony and Peter L. Bartlett. Neural Network Learning: Theoretical Foundations. Cambridge University Press, 1 edition, November 1999.
[2] Itai Arieli, Yakov Babichenko, and Rann Smorodinsky. Robust forecast aggregation. Proceedings of the National Academy of Sciences, 115(52), December 2018.
[3] Itai Arieli, Yakov Babichenko, Inbal Talgam-Cohen, and Konstantin Zabarnyi. Universally Robust Information Aggregation for Binary Decisions. In Proceedings of the 24th ACM Conference on Economics and Computation, pages 118-118, London United Kingdom, July 2023. ACM.
[4] Yakov Babichenko and Dan Garber. Learning Optimal Forecast Aggregation in Partial Evidence Environments. Mathematics of Operations Research, 46(2):628-641, May 2021.
In particular, [4] studies an online learning setting with logarithmic loss. Their regret bound can be converted to our sample complexity bound by an online-to-batch conversion and the Pinsker's inequality.
[48] Robert L. Winkler. Combining Probability Distributions from Dependent Information Sources. Management Science, 27(4):479-488, 1981. Publisher: INFORMS.
[49] Robert L. Winkler and Spyros Makridakis. The Combination of Forecasts. Journal of the Royal Statistical Society. Series A (General), 146(2):150, 1983.
[50] Chunxue Yang and Xiaohui Bei. Learning Optimal Auctions with Correlated Valuations from Samples. In Proceedings of the 38th International Conference on Machine Learning, volume 139 of Proceedings of Machine Learning Research, pages 11716-11726. PMLR, July 2021.
[51] Yuhong Yang. Combining Forecasting Procedures: Some Theoretical Results. Econometric Theory, 20(1):176-222, 2004. Publisher: Cambridge University Press.
A.1 Properties of Hellinger Distance
We review some useful properties of the Hellinger distance. First, the Hellinger distance gives upper bounds on the total variation distance:
Fact A.1 (e.g., Lemma 2.3 in [46]).
$d_{\mathrm{TV}}(D_1,D_2)\leq \sqrt{2} d_{\mathrm{H}}(D_1,D_2).$
-  $1 - d_{\mathrm{TV}}^2 (D_1,D_2)\geq \left(1 - d_{\mathrm{H}}^2 (D_1,D_2)\right)^2$  . (This inequality implies the first one.)
Second, we will use the following lemma to upper bound the squared Hellinger distance between two distributions that are close to each other:
Lemma A.2. Let  $D_{1}$  and  $D_{2}$  be two distributions on  $\mathcal{X}$  satisfying  $1 - \varepsilon \leq \frac{D_2(x)}{D_1(x)}\leq 1 + \varepsilon$ $\forall x\in \mathcal{X}$ . Then,  $d_{\mathrm{H}}^2 (D_1,D_2)\leq \frac{1}{2}\varepsilon^2$
$$
d _ {\mathrm {H}} ^ {2} (D _ {1}, D _ {2}) = \frac {1}{2} \sum_ {x \in \mathcal {X}} \left(\sqrt {D _ {1} (x)} - \sqrt {D _ {2} (x)}\right) ^ {2} = \frac {1}{2} \sum_ {x \in \mathcal {X}} D _ {1} (x) \left(1 - \sqrt {\frac {D _ {2} (x)}{D _ {1} (x)}}\right) ^ {2}.
$$
If  $1 - \varepsilon \leq \frac{D_2(x)}{D_1(x)} < 1$ , then we have  $\left(1 - \sqrt{\frac{D_2(x)}{D_1(x)}}\right)^2 \leq \left(1 - \sqrt{1 - \varepsilon}\right)^2 \leq \left(1 - (1 - \varepsilon)\right)^2 = \varepsilon^2$ . If  $1 + \varepsilon \geq \frac{D_2(x)}{D_1(x)} > 1$ , then we have  $\left(1 - \sqrt{\frac{D_2(x)}{D_1(x)}}\right)^2 \leq \left(\sqrt{1 + \varepsilon} - 1\right)^2 \leq \left((1 + \varepsilon) - 1\right)^2 = \varepsilon^2$ . These two cases together imply
$$
d _ {\mathrm {H}} ^ {2} (D _ {1}, D _ {2}) \leq \frac {1}{2} \sum_ {x \in \mathcal {X}} D _ {1} (x) \cdot \varepsilon^ {2} = \frac {1}{2} \varepsilon^ {2}.
$$
Third, we will use a property of the Hellinger distance between distributions defined on a product space. Suppose  $D_{1}$  and  $D_{2}$  are two distributions over a product space  $\mathcal{X} \times \mathcal{Y}$ . They can be decomposed into the marginal distribution of  $x \in \mathcal{X}$  and the conditional distribution of  $y \in \mathcal{Y}$  given  $x$ , namely,  $D_{1}(x,y) = D_{1,x}(x) \cdot D_{1,y|x}(y|x)$  and  $D_{2}(x,y) = D_{2,x}(x) \cdot D_{2,y|x}(y|x)$ . Then, the squared Hellinger distance between  $D_{1}$  and  $D_{2}$  satisfies the following:
Lemma A.3.  $d_{\mathrm{H}}^{2}(D_{1},D_{2})\leq d_{\mathrm{H}}^{2}(D_{1,x},D_{2,x}) + \max_{x\in \mathcal{X}}d_{\mathrm{H}}^{2}(D_{1,y|x},D_{2,y|x})$  . In particular, if  $x$  and  $y$  are independent, then  $d_{\mathrm{H}}^2 (D_1,D_2)\leq d_{\mathrm{H}}^2 (D_{1,x},D_{2,x}) + d_{\mathrm{H}}^2 (D_{1,y},D_{2,y})$
$$
\begin{array}{l} d _ {\mathrm {H}} ^ {2} (D _ {1}, D _ {2}) = 1 - \sum_ {x \in \mathcal {X}} \sum_ {y \in \mathcal {Y}} \sqrt {D _ {1} (x , y) D _ {2} (x , y)} \\ = 1 - \sum_ {x \in \mathcal {X}} \sqrt {D _ {1 , x} (x) D _ {2 , x} (x)} \sum_ {y \in \mathcal {Y}} \sqrt {D _ {1 , y | x} (y | x) D _ {2 , y | x} (y | x)} \\ = 1 - \sum_ {x \in \mathcal {X}} \sqrt {D _ {1 , x} (x) D _ {2 , x} (x)} + \sum_ {x \in \mathcal {X}} \sqrt {D _ {1 , x} (x) D _ {2 , x} (x)} \Big (1 - \sum_ {y \in \mathcal {Y}} \sqrt {D _ {1 , y | x} (y | x) D _ {2 , y | x} (y | x)} \Big) \\ = d _ {\mathrm {H}} ^ {2} \left(D _ {1, x}, D _ {2, x}\right) + \sum_ {x \in \mathcal {X}} \sqrt {D _ {1 , x} (x) D _ {2 , x} (x)} \cdot d _ {\mathrm {H}} ^ {2} \left(D _ {1, y | x}, D _ {2, y | x}\right) \\ \leq d _ {\mathrm {H}} ^ {2} \left(D _ {1, x}, D _ {2, x}\right) + \sum_ {x \in \mathcal {X}} \sqrt {D _ {1 , x} (x) D _ {2 , x} (x)} \cdot \max  _ {x \in \mathcal {X}} d _ {\mathrm {H}} ^ {2} \left(D _ {1, y | x}, D _ {2, y | x}\right) \\ \leq d _ {\mathrm {H}} ^ {2} (D _ {1, x}, D _ {2, x}) + 1 \cdot \max _ {x \in \mathcal {X}} d _ {\mathrm {H}} ^ {2} (D _ {1, y | x}, D _ {2, y | x}), \\ \end{array}
$$
where the last inequality is because  $\sum_{x\in \mathcal{X}}\sqrt{D_{1,x}(x)D_{2,x}(x)} = 1 - d_{\mathrm{H}}^{2}(D_{1,x},D_{2,x})\leq 1$
Finally, let  $D^{\otimes T}$  denote the distribution of  $T$  i.i.d. samples from  $D$ , namely, the product of  $T$  independent  $D$ 's. We have the following lemma that relates  $d_{\mathrm{H}}^{2}(D_{1}^{\otimes T}, D_{2}^{\otimes T})$  with  $d_{\mathrm{H}}^{2}(D_{1}, D_{2})$ :
Lemma A.4 (e.g., [32]).  $d_{\mathrm{H}}^{2}(D_{1}^{\otimes T},D_{2}^{\otimes T}) = 1 - \left(1 - d_{\mathrm{H}}^{2}(D_{1},D_{2})\right)^{T}\leq T\cdot d_{\mathrm{H}}^{2}(D_{1},D_{2}).$
A.2 Distinguishing Distributions
Let  $D_{1}, D_{2}$  be two distributions over a discrete space  $\mathcal{X}$ . A distribution  $D_{i}$  is chosen uniformly at random from  $\{D_{1}, D_{2}\}$ . Then, we are given  $T$  samples from  $D_{i}$  and want to guess whether the distribution is  $D_{1}$  or  $D_{2}$ . It is known that at least  $T = \Omega\left(\frac{1}{d_{\mathrm{H}}^{2}(D_{1}, D_{2})} \log \frac{1}{\delta}\right)$  samples are needed to guess correctly with probability at least  $1 - \delta$ , no matter how we guess. Formally:
Lemma A.5 (e.g., [32]). Let  $j \in \{1, 2\}$  be the index of the distribution we guess based on the samples. The probability of making a mistake when distinguishing  $D_{1}$  and  $D_{2}$  using  $T$  samples, namely  $\operatorname*{Pr}[j \neq i] = \frac{1}{2} \operatorname*{Pr}[j \neq i \mid i = 1] + \frac{1}{2} \operatorname*{Pr}[j \neq i \mid i = 2]$ , is at least:
-  $\operatorname*{Pr}[j\neq i]\geq \frac{1}{2} -\sqrt{\frac{T}{2}} d_{\mathrm{H}}(D_1,D_2).$
-  $\operatorname*{Pr}[j\neq i]\geq \frac{1}{4}\big(1 - d_{\mathrm{H}}^2 (D_1,D_2)\big)^{2T}\geq \frac{1}{4} e^{-4Td_{\mathrm{H}}^2 (D_1,D_2)}$  , assuming  $d_{\mathrm{H}}^{2}(D_{1},D_{2})\leq \frac{1}{2}$
The second item implies that, in order to achieve  $\operatorname*{Pr}[j \neq i] \leq \delta$ , we must have  $T \geq \frac{1}{4d_{\mathrm{H}}^2(D_1,D_2)} \log \frac{1}{4\delta}$ .
We provide a proof of this lemma for completeness:
Proof. Let  $D_1^{\otimes T}$  and  $D_2^{\otimes T}$  denote the distributions of  $T$  i.i.d. samples from  $D_1$  and  $D_2$ , respectively. The draw of  $T$  samples from  $D_1$  or  $D_2$  is equivalent to the draw of one sample from  $D_1^{\otimes T}$  or  $D_2^{\otimes T}$ . Given one sample from  $D_1^{\otimes T}$  or  $D_2^{\otimes T}$ , the probability of making a mistake when guessing the distribution is at least:
$$
\begin{array}{l} \Pr [ j \neq i ] = \frac {1}{2} \left(\Pr [ j = 2 \mid i = 1 ] + \Pr [ j = 1 \mid i = 2 ]\right) \\ = \frac {1}{2} \left(1 - \Pr [ j = 1 \mid i = 1 ] + \Pr [ j = 1 \mid i = 2 ]\right) \\ = \frac {1}{2} - \frac {1}{2} \left(\Pr [ j = 1 \mid i = 1 ] - \Pr [ j = 1 \mid i = 2 ]\right) \\ = \frac {1}{2} - \frac {1}{2} \left(\mathbb {E} _ {D _ {1} ^ {\otimes T}} [ \mathbb {1} \{j = 1 \} ] - \mathbb {E} _ {D _ {2} ^ {\otimes T}} [ \mathbb {1} \{j = 1 \} ]\right) \\ \text {b y} \quad 3. 2 \geq \frac {1}{2} - \frac {1}{2} d _ {\mathrm {T V}} \left(D _ {1} ^ {\otimes T}, D _ {2} ^ {\otimes T}\right). \tag {12} \\ \end{array}
$$
We then upper bound  $d_{\mathrm{TV}}(D_1^{\otimes T}, D_2^{\otimes T})$  in two ways, which will prove the two items of the lemma, respectively. First, according to first item of Fact A.1, we have
$$
d _ {\mathrm {T V}} \left(D _ {1} ^ {\otimes T}, D _ {2} ^ {\otimes T}\right) \leq \sqrt {2} d _ {\mathrm {H}} \left(D _ {1} ^ {\otimes T}, D _ {2} ^ {\otimes T}\right).
$$
By Lemma A.4,
$$
d _ {\mathrm {H}} \left(D _ {1} ^ {\otimes T}, D _ {2} ^ {\otimes T}\right) \leq \sqrt {T} d _ {\mathrm {H}} \left(D _ {1}, D _ {2}\right).
$$
The above two inequalities give  $d_{\mathrm{TV}}(D_1^{\otimes T}, D_2^{\otimes T}) \leq \sqrt{2T} d_{\mathrm{H}}(D_1, D_2)$ . This proves the first item of the lemma.
Second, according to the second item of Fact A.1 and Lemma A.4, we have
$$
1 - d _ {\mathrm {T V}} ^ {2} \left(D _ {1} ^ {\otimes T}, D _ {2} ^ {\otimes T}\right) \geq \left(1 - d _ {\mathrm {H}} ^ {2} \left(D _ {1} ^ {\otimes T}, D _ {2} ^ {\otimes T}\right)\right) ^ {2} = \left(1 - d _ {\mathrm {H}} ^ {2} \left(D _ {1}, D _ {2}\right)\right) ^ {2 T}
$$
Since  $1 - d_{\mathrm{TV}}^{2}(D_{1}^{\otimes T}, D_{2}^{\otimes T}) = \big(1 + d_{\mathrm{TV}}(D_{1}^{\otimes T}, D_{2}^{\otimes T})\big)\big(1 - d_{\mathrm{TV}}(D_{1}^{\otimes T}, D_{2}^{\otimes T})\big) \leq 2\big(1 - d_{\mathrm{TV}}(D_{1}^{\otimes T}, D_{2}^{\otimes T})\big)$ , we have
$$
1 - d _ {\mathrm {T V}} \left(D _ {1} ^ {\otimes T}, D _ {2} ^ {\otimes T}\right) \geq \frac {1}{2} \left(1 - d _ {\mathrm {H}} ^ {2} \left(D _ {1}, D _ {2}\right)\right) ^ {2 T}.
$$
Plugging into (12), we get
$$
\Pr [ j \neq i ] \geq \frac {1}{4} \left(1 - d _ {\mathrm {H}} ^ {2} (D _ {1}, D _ {2})\right) ^ {2 T}.
$$
When  $d_{\mathrm{H}}^2 (D_1,D_2) < \frac{1}{2}$ , we use the inequality  $1 - x\geq e^{-2x}$  for  $0 < x < \frac{1}{2}$  to conclude that
$$
\Pr [ j \neq i ] \geq \frac {1}{4} \left(e ^ {- 2 d _ {\mathrm {H}} ^ {2} (D _ {1}, D _ {2})}\right) ^ {2 T} = \frac {1}{4} e ^ {- 4 T d _ {\mathrm {H}} ^ {2} (D _ {1}, D _ {2})}.
$$
B Special Cases: Strongly and Weakly Informative Experts
In this section we investigate two special families of conditionally independent distributions that admit  $O\left(\frac{1}{\varepsilon}\log \frac{1}{\delta}\right)$  sample complexity, which is smaller than the  $O\left(\frac{1}{\varepsilon^2}\log \frac{1}{\varepsilon\delta}\right)$  bound for general conditionally independent distributions. In these two cases, the (signals of) experts are either "very informative" or "very non-informative".
Definition B.1. Let  $\gamma \in [0,\infty ]$  be a parameter. For an expert  $i$ , we say its signal  $s_i\in S_i$  is
-  $\gamma$ -strongly informative if either  $\frac{P(s_i|\omega = 1)}{P(s_i|\omega = 0)} \geq 1 + \gamma$  or  $\frac{P(s_i|\omega = 1)}{P(s_i|\omega = 0)} \leq \frac{1}{1 + \gamma}$  holds.
-  $\gamma$ -weakly informative if  $\frac{1}{1 + \gamma} \leq \frac{P(s_i|\omega = 1)}{P(s_i|\omega = 0)} \leq 1 + \gamma$ .
An expert  $i$  is  $\gamma$ -strongly (or  $\gamma$ -weakly) informative if all of its signals in  $\mathcal{S}_i$  are  $\gamma$ -strongly (or  $\gamma$ -weakly) informative.
A signal  $s_i$  being  $\gamma$ -strongly (or  $\gamma$ -weakly) informative implies that its corresponding report  $r_i$  will be "  $\gamma$ -away from" (or "  $\gamma$ -close to") the prior  $p = P(\omega = 1)$ , in the "  $\frac{r_i}{1 - r_i}$  and  $\frac{p}{1 - p}$ " ratio form. Specifically, if  $s_i$  is  $\gamma$ -strongly informative, then from Equation (1) we have
$$
\frac {r _ {i}}{1 - r _ {i}} = \frac {P \left(s _ {i} \mid \omega = 1\right)}{P \left(s _ {i} \mid \omega = 0\right)} \frac {p}{1 - p} \geq (1 + \gamma) \rho \text {o r} \leq \frac {1}{1 + \gamma} \rho . \tag {13}
$$
As  $\gamma$  gets larger, a  $\gamma$ -strongly informative signal (expert) is more informative for predicting whether  $\omega = 1$  or  $0$ . This would make aggregation easier. If  $s_i$  is  $\gamma$ -weakly informative, then:
$$
\frac {1}{1 + \gamma} \rho \leq \frac {r _ {i}}{1 - r _ {i}} \leq (1 + \gamma) \rho . \tag {14}
$$
As  $\gamma$  gets smaller, a  $\gamma$ -weakly informative signal (expert) is less informative for predicting  $\omega$ , but in this case their report  $r_i$  will be close to the prior  $p$ , which allows us to estimate the  $\rho^{n-1}$  term in the optimal aggregator  $f^*(\boldsymbol{r}) = \frac{1}{1 + \rho^{n-1} \prod_{i=1}^{n} \frac{1 - r_i}{r_i}}$  better. Those are some intuitions why both strongly and weakly informative signals can reduce the sample complexity of forecast aggregation.
Formally, for  $\gamma$ -strongly informative experts with not-too-small  $\gamma$ , we have the following result:
Theorem B.2. If  $n \geq 32\log \frac{2}{\varepsilon}$  and all experts are  $\gamma$ -strongly informative with  $\frac{\gamma}{1 + \gamma} \geq 8\sqrt{\frac{2}{n}\log\frac{2}{\varepsilon}}$ , then the sample complexity of forecast aggregation is  $\leq O\left(\frac{1}{\varepsilon n\left(\frac{\gamma}{1 + \gamma}\right)^2}\log \frac{1}{\delta} + \frac{1}{\varepsilon}\log \frac{1}{\delta}\right) = O\left(\frac{1}{\varepsilon}\log \frac{1}{\delta}\right)$ .
We remark that the conditions of the theorem,  $n \geq 32\log \frac{2}{\varepsilon}$  and  $\frac{\gamma}{1 + \gamma} \geq 8\sqrt{\frac{2}{n}\log\frac{2}{\varepsilon}}$ , are easier to be satisfied when the number of experts  $n$  increases, if the informativeness parameter  $\gamma$  of each expert is a constant or does not decrease with  $n$  (which we believe is a reasonable assumption given that experts are independent of each other). Also, if  $\gamma$  is fixed or increasing, then as  $n$  increases the sample complexity decreases.
The proof of Theorem B.2 is in Appendix G.1. Roughly speaking, we divide each expert's signal set into two sets,  $S_{i}^{1}$  and  $S_{i}^{0}$ : signals that are more likely to occur under  $\omega = 1$  (i.e.,  $\frac{P(s_i|\omega = 1)}{P(s_i|\omega = 0)} \geq 1 + \gamma$ ) and under  $\omega = 0$  (i.e.,  $\frac{P(s_i|\omega = 1)}{P(s_i|\omega = 0)} \leq \frac{1}{1 + \gamma}$ ). If the realized  $\omega$  is 1, then one may expect to see  $\Omega\left(\frac{\gamma}{1 + \gamma} n\right)$  more  $S_{i}^{1}$  signals than the  $S_{i}^{0}$  signals from the  $n$  experts, because the probabilities of these two types of signals differ by  $\Omega\left(\frac{\gamma}{1 + \gamma}\right)$  for each expert. If  $\omega$  is 0 then one may expect to see more  $S_{i}^{0}$  signals than the  $S_{i}^{1}$  signals. So, by checking which type of signals are more we can tell whether  $\omega = 0$  or
<sup>7</sup>We note that an expert can be neither  $\gamma$ -strongly informative nor  $\gamma$ -weakly informative for any  $\gamma$ .
1. To tell whether a signal belongs to  $S_i^1$  or  $S_i^0$ , we compare the corresponding report  $\frac{r_i}{1 - r_i}$  with  $\rho$  (namely,  $\frac{r_i}{1 - r_i} \geq (1 + \gamma)\rho \Longleftrightarrow s_i \in S_i^1$  and  $\frac{r_i}{1 - r_i} \leq \frac{1}{1 + \gamma}\rho \Longleftrightarrow s_i \in S_i^0$ ), where  $\rho$  is estimated with accuracy  $\frac{\gamma}{1 + \gamma}$  using  $O\left(\frac{1}{\rho n\left(\frac{\gamma}{1 + \gamma}\right)^2}\log \frac{1}{\delta}\right)$  samples. Dealing with the case of  $\rho < \varepsilon$  separately, we obtain the bound  $O\left(\frac{1}{\varepsilon n\left(\frac{\gamma}{1 + \gamma}\right)^2}\log \frac{1}{\delta} +\frac{1}{\varepsilon}\log \frac{1}{\delta}\right)$ .
For  $\gamma$ -weakly informative experts with small  $\gamma$ , we have:
Theorem B.3. If all experts are  $\gamma$ -weakly informative with  $\gamma \leq 1$ , then the sample complexity of forecast aggregation is  $\leq \min \left\{O\left(\frac{\gamma n}{\varepsilon} \log \frac{1}{\delta}\right), O\left(\frac{1}{\varepsilon^2} \log \frac{1}{\varepsilon \delta}\right)\right\}$ , which is  $O\left(\frac{1}{\varepsilon} \log \frac{1}{\delta}\right)$  if  $\gamma = O\left(\frac{1}{n}\right)$ .
The  $O\left(\frac{1}{\varepsilon^2}\log \frac{1}{\varepsilon\delta}\right)$  term in the sample complexity follows from the result for general conditionally independent distributions (Theorem 5.2). The  $O\left(\frac{\gamma n}{\varepsilon}\log \frac{1}{\delta}\right)$  term is proved in Appendix G.2. We give the rough idea here using  $\gamma = \frac{1}{n}$  as an example. The proof relies on the observation that  $\mathbb{E}\left[\prod_{i=1}^{n}\frac{r_i}{1-r_i} \mid \omega = 0\right] = \rho^n$ . Since experts are weakly informative, each of their reports  $\frac{r_i}{1-r_i}$  is around  $\rho$  in the range  $[\frac{1}{1+\gamma}\rho, (1+\gamma)\rho] \subseteq [\frac{1}{1+\frac{1}{n}}\rho, (1+\frac{1}{n})\rho]$ . Taking the product, we have  $\prod_{i=1}^{n}\frac{r_i}{1-r_i} \in [\frac{\rho^n}{e}, e\rho^n]$ , which is in a bounded range. This allows us to use Chernoff bound to argue that the  $\rho^n$  (or  $\rho^{n-1}$ ) term in the optimal aggregator  $f^*(\boldsymbol{r}) = \frac{1}{1+\rho^{n-1}\prod_{i=1}^{n}\frac{1-r_i}{r_i}}$  can be estimated, with  $O(\sqrt{\varepsilon})$  accuracy, using only  $O\left(\frac{e}{(\sqrt{\varepsilon})^2}\log \frac{1}{\delta}\right) = O\left(\frac{1}{\varepsilon}\log \frac{1}{\delta}\right)$  samples of  $\prod_{i=1}^{n}\frac{r_i}{1-r_i}$ . The aggregator using the estimate  $\hat{\rho}^{n-1}$ ,  $\hat{f}(\boldsymbol{r}) = \frac{1}{1+\hat{\rho}^{n-1}\prod_{i=1}^{n}\frac{1-r_i}{r_i}}$ , turns out to be  $O(\varepsilon)$ -optimal.
C Extension: Multi-Outcome Events
In this section we generalize our model from binary events to multi-outcome events. The event space now becomes  $\Omega = \{1,2,\ldots ,|\Omega |\}$  with  $|\Omega | > 2$ . The joint distribution of event  $\omega \in \Omega$  and experts' signals  $\pmb {s} = (s_i)_i = 1$  is still denoted by  $P$ , which belongs to some class of distributions  $\mathcal{P}$ . The size of each expert's signal space is still assumed to be  $|S_{i}| = m < + \infty$  for general distributions and can be infinite for conditionally independent distributions (where  $s_1,\dots ,s_n$  are independent conditioned on  $\omega$ ). After observing signal  $s_i$ , expert  $i$  reports its posterior belief of the event given  $s_i$ , which is  $\pmb {r}_i = (r_{ij})_{j\in \Omega}$  where  $r_{ij} = P(\omega = j|s_i)$ . An aggregator now is a vector-valued function  $\pmb {f} = (f_j)_{j\in \Omega}$  that maps the joint report  $\pmb {r} = (\pmb {r}_i)_i^n = (r_{ij})_{ij}$  to a probability distribution  $\pmb {f}(\pmb {r})$  over  $\Omega$ , where  $f_{j}(\pmb {r})$  is the aggregated predicted probability for  $\omega = j$ . We assume  $f_{j}(\pmb {r})\geq 0$  and  $\sum_{j\in \Omega}f_j(\pmb {r}) = 1$ . The definition of the (expected) loss of an aggregator  $\pmb{f}$  becomes:
$$
L _ {P} (\boldsymbol {f}) = \mathbb {E} _ {P} \left[ \sum_ {j \in \Omega} \left| f _ {j} (\boldsymbol {r}) - \mathbb {1} [ \omega = j ] \right| ^ {2} \right]. \tag {15}
$$
It is easy to see that the optimal aggregator  $\pmb{f}^{*} = \operatorname{argmin}_{\pmb{f}} L_{P}(\pmb{f})$  in the multi-outcome case is still the Bayesian aggregator (this is a generalization of Lemma 2.1):
$$
\boldsymbol {f} ^ {*} = \left(f _ {j} ^ {*}\right) _ {j \in \Omega}, \quad f _ {j} ^ {*} (\boldsymbol {r}) = P (\omega = j \mid \boldsymbol {r}) \tag {16}
$$
and the difference between the losses of  $\pmb{f}$  and  $\pmb{f}^{*}$  satisfies
$$
L _ {P} (\boldsymbol {f}) - L _ {P} \left(\boldsymbol {f} ^ {*}\right) = \mathbb {E} _ {P} \left[ \sum_ {j \in \Omega} \left| f _ {j} (\boldsymbol {r}) - f _ {j} ^ {*} (\boldsymbol {r}) \right| ^ {2} \right]. \tag {17}
$$
So, an  $\varepsilon$ -optimal aggregator  $f$  is an aggregator that satisfies  $\mathbb{E}_P\left[\sum_{j\in \Omega}|f_j(\boldsymbol {r}) - f_j^* (\boldsymbol {r})|^2\right]\leq \varepsilon$
Using  $T = T_{\mathcal{P}}(\varepsilon, \delta)$  samples  $\{(r^{(t)}, \omega^{(t)})\}_{t=1}^{T}$  from  $P$ , the principal wants to find an  $\varepsilon$ -optimal aggregator  $\hat{\pmb{f}}$  with probability at least  $1 - \delta$ , for any distribution  $P$  in a class  $\mathcal{P}$ . We give lower bounds and upper bounds on the sample complexity:
Theorem C.1. The sample complexity of forecast aggregation for multi-outcome events is:
- For the class  $\mathcal{P}$  of general distributions:  $\Omega\left(\frac{m^{n-2} + \log(1/\delta)}{\varepsilon}\right) \leq T_{\mathcal{P}}(\varepsilon, \delta) \leq O\left(\frac{|\Omega|m^n + \log(1/\delta)}{\varepsilon^2}\right)$ .
- For the class  $\mathcal{P}_{\mathrm{ind}}$  of conditionally independent distributions:  $\Omega \left(\frac{1}{\varepsilon} \log \frac{1}{\delta}\right) \leq T_{\mathcal{P}_{\mathrm{ind}}}(\varepsilon, \delta) \leq O\left(\frac{|\Omega| \log |\Omega|}{\varepsilon^2} \log \frac{1}{\varepsilon} + \frac{1}{\varepsilon^2} \log \frac{1}{\delta}\right)$ .
C.1 Proof of Theorem C.1
Before proving the theorem, we note that the loss of any aggregator  $\pmb{f}$  satisfying  $f_{j}(\pmb{r}) \geq 0$  and  $\sum_{j \in \Omega} f_{j}(\pmb{r}) = 1$  is bounded by [0, 2]: