SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Tao Lin
tlin@g.harvard.edu
Yiling Chen
yiling@seas.harvard.edu
We consider a Bayesian forecast aggregation model where  $n$  experts, after observing private signals about an unknown binary event, report their posterior beliefs about the event to a principal, who then aggregates the reports into a single prediction for the event. The signals of the experts and the outcome of the event follow a joint distribution that is unknown to the principal, but the principal has access to i.i.d. "samples" from the distribution, where each sample is a tuple of the experts' reports (not signals) and the realization of the event. Using these samples, the principal aims to find an  $\varepsilon$ -approximately optimal aggregator, where optimality is measured in terms of the expected squared distance between the aggregated prediction and the realization of the event. We show that the sample complexity of this problem is at least  $\tilde{\Omega}(m^{n-2}/\varepsilon)$  for arbitrary discrete distributions, where  $m$  is the size of each expert's signal space. This sample complexity grows exponentially in the number of experts  $n$ . But, if the experts' signals are independent conditioned on the realization of the event, then the sample complexity is significantly reduced, to  $\tilde{O}(1/\varepsilon^2)$ , which does not depend on  $n$ . Our results can be generalized to non-binary events. The proof of our results uses a reduction from the distribution learning problem and reveals the fact that forecast aggregation is almost as difficult as distribution learning.
Suppose you want to know whether it will rain tomorrow. A Google search on "weather" returns  $40\%$  probability of raining. The weather forecasting app on your phone shows  $85\%$ . And one of your friends, who is an expert in meteorology, predicts  $65\%$ . How do you aggregate these different predictions into a single, accurate prediction? This problem is called forecast aggregation or forecast combination [7, 16, 45]. It has innumerable applications in fields ranging from economics, statistics, operations research, to machine learning, decision theory, and of course, climate science.
A straightforward solution to forecast aggregation is to take the (unweighted) average of all experts' forecasts. Simple as it is, unweighted average performs surprisingly well in practice, as observed by many forecast aggregation works from 1960s to 2020s (e.g., [7, 38, 35, 30, 16, 45, 34]). Naturally, when past data of expert forecasts and event outcomes (e.g., historical weather forecasts and outcomes) are available, one may hope to leverage on such data to learn more accurate aggregators. While adjusting the weights of experts in averaging according to their individual historical accuracy has led to improved accuracy for the aggregated prediction [49], interestingly, more sophisticated data-driven methods [23, 35, 49] were often outperformed by the unweighted average. The dominance of simple aggregators over more sophisticated data-driven methods is observed so often in empirical applications that it is termed "the forecast combination puzzle" [44, p.428].
There are many potential explanations for the forecast combination puzzle [43, 23, 15]. In some scenarios, the past events are different from the future events in fundamental ways (e.g., geopolitical
forecasting) and hence past data may not be informative in predicting the future. Another widely accepted conjecture is that the amount of past data is not large enough for a data-intensive method to perform well. Indeed, the sample sizes in many empirical forecast aggregation works are small under today's "big-data" standard (24 in [27], 69 in [42], 87 in [6], 100 in [30]). However, there are aggregation settings where future events are arguably similar to past events and we do have abundant data — for instance, the forecasting of weather, stock prices [21], and some forecasting competitions on Kaggle<sup>1</sup>. Such settings are well-suited for data-driven methods. It is hence tempting to ask how many data are needed for data-driven aggregators to achieve high accuracy in these settings.
In this paper, we initiate the study of sample complexity of forecast aggregation, building upon a standard Bayesian forecasting model [37, 48, 24]. In this model, there are  $n$  experts who observe private signals about an unknown binary event and make posterior predictions about the event. The experts' signals and the event are jointly distributed according to some underlying unknown distribution (information structure)  $P$ , which determines the optimal way to aggregate the experts' predictions. With sample access to the unknown distribution  $P$ , we ask the following question:
How many samples do we need to approximately learn the optimal aggregator?
Our model favors the use of data-driven aggregation methods because historical tasks are i.i.d. as future tasks. We show that however, even in this benign model, optimal aggregation in general needs exponentially many samples. One may thus expect that data-driven methods can hardly perform well in more realistic and non-i.i.d. scenarios. Nevertheless, for some special, yet interesting, families of information structures, the sample complexity of forecast aggregation is significantly lower.
Main results (1) If  $P$  can be an arbitrary discrete distribution, then at least  $\tilde{\Omega}(m^{n-2}/\varepsilon)$  samples are needed, and  $\tilde{O}(m^n/\varepsilon^2)$  samples are sufficient, to learn an  $\varepsilon$ -optimal aggregator with high probability, where  $m$  is the number of signals an expert can possibly observe. (2) If the experts' signals are conditionally independent, then the sample complexity is exponentially reduced and surprisingly does not depend on the number of experts or the number of signals:  $\tilde{O}(1/\varepsilon^2)$  samples are sufficient, and  $\tilde{\Omega}(1/\varepsilon)$  samples are necessary, to learn an  $\varepsilon$ -optimal aggregator with high probability.
Main techniques The main technical part of our paper is to prove the  $\tilde{\Omega}(m^{n-2}/\varepsilon)$  lower bound on the sample complexity of forecast aggregation for the general case, via a reduction from the distribution learning problem. It is known that learning a discrete distribution with support size  $|S|$  in total variation distance  $\varepsilon$  requires  $\tilde{\Omega}(|S|/\varepsilon^2)$  samples. By reducing distribution learning to forecast aggregation, we obtain the lower bound on the sample complexity of the latter problem. This reduction is highly nontrivial. To do this reduction we define a new distribution learning problem that is different from the one in the literature, which is of independent interest. This reduction also reveals an interesting fact: learning to aggregate optimally on some distribution is almost as difficult as learning the distribution itself. This is a little surprising because one might initially think that aggregation should be easier than distribution learning – we show that this is not the case.
Structure of the paper We discuss related works in Section 1.1. Section 2 introduces our model. Section 3 includes some preliminaries, including the distribution learning problem, which we will use in the proof. Section 4 studies the sample complexity for general distributions. Section 5 focuses on the conditional independence case. Section 6 summarizes how our results can be generalized to non-binary (multi-outcome) events. Section 7 concludes and discusses future directions.
Data-driven aggregation Data-driven approaches to forecast aggregation date back to perhaps the first paper on forecast aggregation [7] and have been a standard practice in the literature (see, e.g., surveys [16, 45] and more recent works [51, 42, 4, 47, 22, 40]). Many of these works focus on specific weighted average aggregators like linear pooling (weighted arithmetic mean) [51] and logarithmic pooling (normalized weighted geometric mean) [42, 40], and the goal is to estimate the optimal weights from data. However, those weighted average aggregators are not necessarily the
<https://www.kaggle.com/c/m5-forecasting-accuracy>
2The  $\tilde{O} (\cdot)$  and  $\tilde{\Omega} (\cdot)$  notations omit logarithmic factors.
optimal (Bayesian) aggregator unless the underlying information structure satisfies some strict conditions (e.g., experts' forecasts are equal to the true probability of the event plus normally distributed noises [48]). Our work aims to understand the sample complexity of Bayesian aggregation, namely how many samples are needed to approximate the Bayesian aggregator.
Our work is closely related to a recent work [4] on Bayesian aggregation via online learning. For the case of conditionally independent experts, [4] shows that the Bayesian aggregator can be approximated with  $\varepsilon = \tilde{O}\left(\frac{n}{\sqrt{T}}\right)$  regret. By an online-to-batch conversion, this implies a  $T = \tilde{O}\left(\frac{n^2}{\varepsilon^2}\right)$  sample complexity upper bound for the batch learning setting. Our paper studies the batch learning setting. For the conditional independence case, we obtain an improved bound of  $\tilde{O}\left(\frac{1}{\varepsilon^2}\right)$ .
Robust forecast aggregation Recent works on "robust forecast aggregation" [2, 39, 18, 3] also study information aggregation problems where the principal does not know the underlying information structure. They take a worst-case approach, assuming that the information structure is chosen adversarially. This often leads to negative results: e.g., a bad approximation ratio [2, 39] or a degenerate maximin aggregator that solely relies on a random expert's opinion [18, 3]. In contrast, we assume sample access to the unknown information structure. Our sample complexity approach is orthogonal and complementary to the robust forecast aggregation approach.
Sample complexity of mechanism design Our work may remind the reader of a long line of research on the sample complexity of revenue maximization in mechanism design (e.g., [17, 20, 36, 19, 5, 11, 29, 10, 25, 50, 28]). In particular, [28] gives a general framework to bound the sample complexity for mechanism design problems that satisfy a "strong monotonicity" property, but this property is not satisfied by our forecast aggregation problem. A key observation from this line of works is that the number of samples needed to learn an  $\varepsilon$ -optimal auction for  $n \geq 1$  independent bidders is increasing in  $n$ , because when there are more bidders, although we can obtain higher revenue, the optimal auction benchmark is also improved. We see a similar phenomenon that the sample complexity of forecast aggregation increases with the number of experts in the general case, but not in the case where experts are conditionally independent.
2.1 Forecast Aggregation
There are  $n \geq 2$  experts and one principal. The principal wants to predict the probability that a binary event  $\omega \in \Omega = \{0,1\}$  happens ( $\omega = 1$ ), based on information provided by the experts. For example,  $\omega$  may represent whether it will rain tomorrow. We present binary events to simplify notations. All our results can be generalized to multi-outcome events with  $|\Omega| > 2$  (see Section 6). We also refer to  $\omega$  as "the state of the world". Each expert  $i = 1, \ldots, n$  observes some private signal  $s_i \in S_i$  that is correlated with  $\omega$ , where  $S_i$  denotes the space of all possible signals of expert  $i$ . We assume for now that  $S_i$  is finite, with size  $|S_i| = m$ . We relax this assumption in Section 5 where we consider conditionally independent signals. Let  $\mathcal{S} = S_1 \times \dots \times S_n$  be the joint signal space of all experts;  $|\mathcal{S}| = m^n$ . Let  $P$  be a distribution over  $\mathcal{S} \times \Omega$ , namely, a joint distribution of signals  $s = (s_1, \ldots, s_n)$  and event  $\omega$ . Since the space  $\mathcal{S} \times \Omega$  is discrete, we can use  $P(\cdot)$  to denote the probability:  $P(s, \omega) = \operatorname{Pr}_P[s, \omega]$ . Signals of different experts can be correlated conditioned on  $\omega$ . We assume that each expert  $i$  knows the marginal joint distribution of their own signal  $s_i$  and  $\omega$ ,  $P(s_i, \omega)$ . Neither any expert nor the principal knows the entire distribution  $P$ . Each expert  $i$  reports to the principal a forecast (or prediction)  $r_i$  for the event  $\omega$ , which is equal to the conditional probability of  $\omega = 1$  given their signal  $s_i$ :
$$
r _ {i} = P (\omega = 1 \mid s _ {i}) = \frac {P (\omega = 1) P \left(s _ {i} \mid \omega = 1\right)}{P (\omega = 1) P \left(s _ {i} \mid \omega = 1\right) + P (\omega = 0) P \left(s _ {i} \mid \omega = 0\right)}. \tag {1}
$$
We note that  $r_i$  depends on  $s_i$  and  $P$ , but not on  $\omega$  or other experts' signals  $s_{-i}$ . Let  $r = (r_1, \ldots, r_n) \in [0, 1]^n$  denote the reports (joint report) of all experts. We sometimes use  $r_{-i}$  to
One may wonder whether the experts are willing to report  $r_i = P(\omega = 1|s_i)$  truthfully. This can be guaranteed by a proper scoring rule. For example, we can reward each expert the Brier score  $C - |r_i - \omega|^2$  after seeing the realization of  $\omega$  [9]. Each expert maximizes its expected reward by reporting its belief truthfully.
denote the reports of all experts except  $i$ . The principal aggregates the experts' reports  $r$  into a single forecast  $f(r)$  using some aggregation function, or aggregator,  $f:[0,1]^n\to [0,1]$ . We measure the performance of an aggregator by the mean squared loss:
$$
L _ {P} (f) = \mathbb {E} _ {P} \left[ | f (\boldsymbol {r}) - \omega | ^ {2} \right]. \tag {2}
$$
The notation  $\mathbb{E}_P[\cdot ]$  makes it explicit that the expectation is over the random draw of  $(s,\omega)\sim P$  followed by letting  $r_i = P(\omega = 1\mid s_i)$ . We omit  $P$  and write  $\mathbb{E}[\cdot ]$  when it is clear from the context.
Let  $f^*$  be the optimal aggregator with respect to  $P$ , which minimizes  $L_P(f)$ :
$$
f ^ {*} = \underset {f: [ 0, 1 ] ^ {n} \rightarrow [ 0, 1 ]} {\operatorname {a r g m i n}} L _ {P} (f) = \underset {f: [ 0, 1 ] ^ {n} \rightarrow [ 0, 1 ]} {\operatorname {a r g m i n}} \mathbb {E} _ {P} \left[ | f (\boldsymbol {r}) - \omega | ^ {2} \right]. \tag {3}
$$
We have the following characterization of  $f^{*}$  and  $L_{P}(f)$ :  $f^{*}$  is equal to the Bayesian aggregator, which computes the posterior probability of  $\omega = 1$  given all the reports  $r = (r_1, \dots, r_n)$ . And the difference between the loss of  $f$  and the loss of  $f^{*}$  is equal to their expected squared difference.
Lemma 2.1. The optimal aggregator  $f^{*}$  and any aggregator  $f$  satisfy:
-  $f^{*}(\boldsymbol{r}) = P(\omega = 1 \mid \boldsymbol{r})$ , for almost every  $\boldsymbol{r}$ .
-  $L_{P}(f) - L_{P}(f^{*}) = \mathbb{E}_{P}\bigl [|f(\pmb {r}) - f^{*}(\pmb {r})|^{2}\bigr ]$
An aggregator  $f$  is  $\varepsilon$ -optimal (with respect to  $P$ ) if  $L_{P}(f) \leq L_{P}(f^{*}) + \varepsilon$ . By Lemma 2.1, this is equivalent to  $\mathbb{E}_P\big[|f(\pmb {r}) - f^* (\pmb {r})|^2\big] \leq \varepsilon$ . For an  $\varepsilon$ -optimal  $f$ , we also say it  $\varepsilon$ -approximates  $f^{*}$ .
Definition 2.2. An aggregator  $f$  is  $\varepsilon$ -optimal (with respect to  $P$ ) if  $\mathbb{E}_P\big[|f(\boldsymbol {r}) - f^* (\boldsymbol {r})|^2\big]\leq \varepsilon$ .
Discussion of the benchmark  $f^{*}$ . Our benchmark, the Bayesian aggregator  $f^{*}$ , is common in the forecast aggregation literature (e.g., [27, 26, 45]). It is stronger than the typical "best expert" benchmark in no-regret learning (e.g., [13, 22]), but weaker than the "omniscient" aggregator that has access to the experts' signals:  $f_{\mathrm{omni}}(s) = P(\omega = 1|s)$ . If there is a one-to-one mapping between signals  $s$  and reports  $r$ , then  $f_{\mathrm{omni}}$  and  $f^{*}$  are the same. Otherwise,  $f_{\mathrm{omni}}$  could be much stronger than  $f^{*}$  and an  $\varepsilon$ -approximation to  $f_{\mathrm{omni}}$  using experts' reports only is not always possible. In contrast, an  $\varepsilon$ -approximation to  $f^{*}$  is always achievable (in fact, achieved by  $f^{*}$  itself). The difference between  $f^{*}$  and  $f_{\mathrm{omni}}$  is known as the difference between "aggregating forecasts" and "aggregating information sets" in the literature [27, p.198-199], [26, p.168-169], [45, p.143].
2.2 Sample Complexity of Forecast Aggregation
The principal has access to  $T$  i.i.d. samples of forecasts and event realizations drawn from the underlying unknown distribution  $P$ :
$$
S _ {T} = \left\{\left(\boldsymbol {r} ^ {(1)}, \omega^ {(1)}\right), \dots , \left(\boldsymbol {r} ^ {(T)}, \omega^ {(T)}\right) \right\}, \quad \left(\boldsymbol {s} ^ {(t)}, \omega^ {(t)}\right) \sim P, \quad r _ {i} ^ {(t)} = P (\omega = 1 \mid s _ {i} ^ {(t)}). \tag {4}
$$
Here, we implicitly regard  $P$  as a distribution over  $(\boldsymbol{r},\omega)$  instead of  $(s,\omega)$ . The principal uses samples  $S_{T}$  to learn an aggregator  $\hat{f} = \hat{f}_{S_T}$ , in order to approximate  $f^{*}$ . Our main question is:
How many samples are necessary and sufficient for finding an  $\varepsilon$ -optimal aggregator  $\hat{f}$  (with probability at least  $1 - \delta$  over the random draw of samples)?
The answer to the above question depends on the family of distributions we are interested in. Let  $\mathcal{P}$  denote a family of distributions over  $S\times \Omega$ . It could be the set of all distributions over  $S\times \Omega$ , or in Section 5 we will only consider distributions where the signals are independent conditioned on  $\omega$ . We define the sample complexity of forecast aggregation, with respect to  $\mathcal{P}$ , formally:
Definition 2.3. The sample complexity of forecast aggregation (with respect to  $\mathcal{P}$ ) is the minimum function  $T_{\mathcal{P}}(\cdot, \cdot)$  of  $\varepsilon, \delta \in (0,1)$ , such that: if  $T \geq T_{\mathcal{P}}(\varepsilon, \delta)$ , then for any distribution  $P \in \mathcal{P}$ , with probability at least  $1 - \delta$  over the random draw of  $T$  samples  $S_T$  from  $P$  (and over the randomness of the learning procedure if it is randomized), we can obtain an aggregator  $\hat{f} = \hat{f}_{S_T}$  satisfying  $\mathbb{E}_P[|\hat{f}(\boldsymbol{r}) - f^*(\boldsymbol{r})|^2] \leq \varepsilon$ .
This has been noted by [2, 4, 39]. They give an XOR example where  $\omega = s_1 \oplus s_n$ ,  $s_1$  and  $s_2$  are i.i.d. Uniform  $\{0,1\}$  distributed. The experts always report  $r_i = 0.5$ ,  $f_{\mathrm{omni}}(s_1,s_2) = \omega$ ,  $f^{*}(r_{1},r_{2}) = 0.5$ , so  $L_{P}(f_{\mathrm{omni}}) = 0$  but  $L_{P}(f^{*}) = 0.25 > 0$ . No aggregator that uses experts' reports only can do better than  $f^{*}$ .
The principal is assumed to know the family of distributions  $\mathcal{P}$  but not the specific distribution  $P\in \mathcal{P}$ . There should be at least two different distributions in  $\mathcal{P}$ . Otherwise, the principal knows what the distribution is and there is no need for learning.
In this section, we briefly introduce some notions that will be used in our analysis of the sample complexity of forecast aggregation, including some definitions of distances between distributions, the distribution learning problem, and the distinguishing distributions problem.
Distances between distributions We recall two distance metrics for discrete distributions: the total variation distance and the (squared) Hellinger distance.
Definition 3.1. Let  $D_{1}, D_{2}$  be two distributions on a discrete space  $\mathcal{X}$ .
- The total variation distance between  $D_{1}$  and  $D_{2}$  is  $d_{\mathrm{TV}}(D_1,D_2) = \frac{1}{2}\sum_{x\in \mathcal{X}}\bigl {|}D_1(x) - D_2(x)\bigr{|}$ .
- The squared Hellinger distance between  $D_{1}$  and  $D_{2}$  is  $d_{\mathrm{H}}^{2}(D_{1},D_{2}) = \frac{1}{2}\sum_{x\in \mathcal{X}}\left(\sqrt{D_{1}(x)} - \sqrt{D_{2}(x)}\right)^{2} = 1 - \sum_{x\in \mathcal{X}}\sqrt{D_{1}(x)D_{2}(x)}.$
The total variation distance has the following well-known property that upper bounds the difference between the expected values of a function on two distributions:
Fact 3.2. For any function  $h: \mathcal{X} \to [0,1]$ ,  $|\mathbb{E}_{x \sim D_1} h(x) - \mathbb{E}_{x \sim D_2} h(x)| \leq d_{\mathrm{TV}}(D_1, D_2)$ .
In Appendix A we give some properties of the Hellinger distance that will be used in the proofs.
Distribution learning in total variation distance Our analysis of the sample complexity of the forecast aggregation problem will leverage on the sample complexity of another learning problem: learning discrete distributions in total variation distance. We review this problem below.
Let  $\mathcal{D}$  be a family of distributions over  $\mathcal{X}$ . The sample complexity of learning distributions in  $\mathcal{D}$  within total variation distance  $\varepsilon$  is the minimum function  $T_{\mathcal{D}}^{\mathrm{TV}}(\varepsilon, \delta)$ , such that: if  $T \geq T_{\mathcal{D}}^{\mathrm{TV}}(\varepsilon, \delta)$ , then for any distribution  $D \in \mathcal{D}$ , with probability at least  $1 - \delta$  over the random draw of  $T$  samples from  $D$ , we can obtain (from the  $T$  samples) a distribution  $\hat{D}$  such that  $d_{\mathrm{TV}}(\hat{D}, D) \leq \varepsilon$ .
Proposition 3.3 (e.g., [12, 33]). Let  $\mathcal{D}_{\mathrm{all}}$  be the set of all distributions over  $\mathcal{X}$ . Then,  $T_{\mathcal{D}_{\mathrm{all}}}^{\mathrm{TV}}(\varepsilon, \delta) = \Theta \left( \frac{|\mathcal{X}| + \log(1 / \delta)}{\varepsilon^2} \right)$ . In particular, the upper bound can be achieved by using the empirical estimate  $\hat{D}_{\mathrm{emp}}$  (which is the uniform distribution over the  $T$  samples). The lower bound holds regardless of what learning algorithm is used.
Distinguishing distributions Another learning problem that we will leverage on is the problem of distinguishing (two) distributions: given samples from a distribution randomly chosen from  $\{D_1, D_2\}$ , we are to guess whether the samples are from  $D_1$  or  $D_2$ . The sample complexity of distinguishing distributions is characterized by the squared Hellinger distance. It is known that at least  $T = \Omega \left( \frac{1}{d_{\mathrm{H}}^2(D_1, D_2)} \log \frac{1}{\delta} \right)$  samples are needed to distinguish two distributions with probability at least  $1 - \delta$ . See Appendix A for a formal statement of this result.
4 Sample Complexity for General Distributions
In this section we characterize the sample complexity of forecast aggregation for general distributions  $P$ . We give an exponential (in the number of experts,  $n$ ) upper bound and an exponential lower bound on the sample complexity, as follows:
Theorem 4.1. Let  $\mathcal{P}_{\mathrm{all}}$  be the set of all distributions over  $\mathbf{S} \times \Omega$ , with  $|\mathbf{S}| = m^n$ . Suppose  $n \geq 2$ . The sample complexity of forecast aggregation with respect to  $\mathcal{P}_{\mathrm{all}}$  is
$$
O \left(\frac {m ^ {n} + \log (1 / \delta)}{\varepsilon^ {2}}\right) \geq T _ {\mathcal {P} _ {\mathrm {a l l}}} (\varepsilon , \delta) \geq \Omega \left(\frac {m ^ {n - 2} + \log (1 / \delta)}{\varepsilon}\right). \tag {5}
$$
This theorem is for  $n \geq 2$ . When there is only one expert ( $n = 1$ ), there is no need to learn to aggregate because the optimal "aggregator"  $f^*$  simply outputs the forecast given by the only expert:  $f^*(r_1) = P(\omega = 1 | r_1) = P(\omega = 1 | s_1) = r_1$ . The sample complexity is 0 in this case.
There is a gap in the dependency on  $\varepsilon$  in the upper bound and the lower bound in Theorem 4.1. We conjecture that the tight dependency on  $\varepsilon$  should be  $\frac{1}{\varepsilon}$  (so the lower bound is tight). See Section 7 for a detailed discussion of this conjecture, where we show that the dependency on  $\varepsilon$  in the upper bound can be improved to  $\frac{1}{\varepsilon}$  for a large family of distributions.
4.1 Proof of the Upper Bound