Title: Sample Complexity of Forecast Aggregation
Venue: NeurIPS 2023
Accepted: yes
Keywords: information aggregation

sample complexity

distribution learning

Bayesian forecast aggregation
Abstract: We consider a Bayesian forecast aggregation model where $n$ experts, after observing private signals about an unknown binary event, report their posterior beliefs about the event to a principal, who then aggregates the reports into a single prediction for the event. The signals of the experts and the outcome of the event follow a joint distribution that is unknown to the principal, but the principal has access to i.i.d. "samples" from the distribution, where each sample is a tuple of the experts' reports (not signals) and the realization of the event. Using these samples, the principal aims to find an $\varepsilon$-approximately optimal aggregator, where optimality is measured in terms of the expected squared distance between the aggregated prediction and the realization of the event. We show that the sample complexity of this problem is at least $\tilde \Omega(m^{n-2} / \varepsilon)$ for arbitrary discrete distributions, where $m$ is the size of each expert's signal space.  This sample complexity grows exponentially in the number of experts $n$. But, if the experts' signals are independent conditioned on the realization of the event, then the sample complexity is significantly reduced, to $\tilde O(1 / \varepsilon^2)$, which does not depend on $n$. Our results can be generalized to non-binary events.  The proof of our results uses a reduction from the distribution learning problem and reveals the fact that forecast aggregation is almost as difficult as distribution learning.

Sample Complexity of Forecast Aggregation

Abstract

We consider a Bayesian forecast aggregation model where n experts, after observing private signals about an unknown binary event, report their posterior beliefs about the event to a principal, who then aggregates the reports into a single prediction for the event. The signals of the experts and the outcome of the event follow a joint distribution that is unknown to the principal, but the principal has access to i.i.d. 'samples' from the distribution, where each sample is a tuple of the experts' reports (not signals) and the realization of the event. Using these samples, the principal aims to find an ε -approximately optimal aggregator, where optimality is measured in terms of the expected squared distance between the aggregated prediction and the realization of the event. We show that the sample complexity of this problem is at least ˜ Ω( m n -2 /ε ) for arbitrary discrete distributions, where m is the size of each expert's signal space. This sample complexity grows exponentially in the number of experts n . But, if the experts' signals are independent conditioned on the realization of the event, then the sample complexity is significantly reduced, to ˜ O (1 /ε 2 ) , which does not depend on n . Our results can be generalized to non-binary events. The proof of our results uses a reduction from the distribution learning problem and reveals the fact that forecast aggregation is almost as difficult as distribution learning.

1 Introduction

Suppose you want to know whether it will rain tomorrow. A Google search on 'weather' returns 40% probability of raining. The weather forecasting app on your phone shows 85%. And one of your friends, who is an expert in meteorology, predicts 65%. How do you aggregate these different predictions into a single, accurate prediction? This problem is called forecast aggregation or forecast combination [7, 16, 45]. It has innumerable applications in fields ranging from economics, statistics, operations research, to machine learning, decision theory, and of course, climate science.

A straightforward solution to forecast aggregation is to take the (unweighted) average of all experts' forecasts. Simple as it is, unweighted average performs surprisingly well in practice, as observed by many forecast aggregation works from 1960s to 2020s (e.g., [7, 38, 35, 30, 16, 45, 34]). Naturally, when past data of expert forecasts and event outcomes (e.g., historical weather forecasts and outcomes) are available, one may hope to leverage on such data to learn more accurate aggregators. While adjusting the weights of experts in averaging according to their individual historical accuracy has led to improved accuracy for the aggregated prediction [49], interestingly, more sophisticated data-driven methods [23, 35, 49] were often outperformed by the unweighted average. The dominance of simple aggregators over more sophisticated data-driven methods is observed so often in empirical applications that it is termed 'the forecast combination puzzle' [44, p.428].

There are many potential explanations for the forecast combination puzzle [43, 23, 15]. In some scenarios, the past events are different from the future events in fundamental ways (e.g., geopolitical

forecasting) and hence past data may not be informative in predicting the future. Another widely accepted conjecture is that the amount of past data is not large enough for a data-intensive method to perform well. Indeed, the sample sizes in many empirical forecast aggregation works are small under today's 'big-data' standard (24 in [27], 69 in [42], 87 in [6], 100 in [30]). However, there are aggregation settings where future events are arguably similar to past events and we do have abundant data - for instance, the forecasting of weather, stock prices [21], and some forecasting competitions on Kaggle 1 . Such settings are well-suited for data-driven methods. It is hence tempting to ask how many data are needed for data-driven aggregators to achieve high accuracy in these settings.

In this paper, we initiate the study of sample complexity of forecast aggregation , building upon a standard Bayesian forecasting model [37, 48, 24]. In this model, there are n experts who observe private signals about an unknown binary event and make posterior predictions about the event. The experts' signals and the event are jointly distributed according to some underlying unknown distribution (information structure) P , which determines the optimal way to aggregate the experts' predictions. With sample access to the unknown distribution P , we ask the following question:

How many samples do we need to approximately learn the optimal aggregator?

Our model favors the use of data-driven aggregation methods because historical tasks are i.i.d. as future tasks. We show that however, even in this benign model, optimal aggregation in general needs exponentially many samples. One may thus expect that data-driven methods can hardly perform well in more realistic and non-i.i.d. scenarios. Nevertheless, for some special, yet interesting, families of information structures, the sample complexity of forecast aggregation is significantly lower.

Main results (1) If P can be an arbitrary discrete distribution, then at least ˜ Ω( m n -2 /ε ) samples are needed, and ˜ O ( m n /ε 2 ) samples are sufficient, to learn an ε -optimal aggregator with high probability, where m is the number of signals an expert can possibly observe. 2 (2) If the experts' signals are conditionally independent, then the sample complexity is exponentially reduced and surprisingly does not depend on the number of experts or the number of signals: ˜ O (1 /ε 2 ) samples are sufficient, and ˜ Ω(1 /ε ) samples are necessary, to learn an ε -optimal aggregator with high probability.

Main techniques The main technical part of our paper is to prove the ˜ Ω( m n -2 /ε ) lower bound on the sample complexity of forecast aggregation for the general case, via a reduction from the distribution learning problem. It is known that learning a discrete distribution with support size | S | in total variation distance ε requires ˜ Ω( | S | /ε 2 ) samples. By reducing distribution learning to forecast aggregation, we obtain the lower bound on the sample complexity of the latter problem. This reduction is highly nontrivial. To do this reduction we define a new distribution learning problem that is different from the one in the literature, which is of independent interest. This reduction also reveals an interesting fact: learning to aggregate optimally on some distribution is almost as difficult as learning the distribution itself. This is a little surprising because one might initially think that aggregation should be easier than distribution learning - we show that this is not the case.

Structure of the paper We discuss related works in Section 1.1. Section 2 introduces our model. Section 3 includes some preliminaries, including the distribution learning problem, which we will use in the proof. Section 4 studies the sample complexity for general distributions. Section 5 focuses on the conditional independence case. Section 6 summarizes how our results can be generalized to non-binary (multi-outcome) events. Section 7 concludes and discusses future directions.

1.1 Related Works

Data-driven aggregation Data-driven approaches to forecast aggregation date back to perhaps the first paper on forecast aggregation [7] and have been a standard practice in the literature (see, e.g., surveys [16, 45] and more recent works [51, 42, 4, 47, 22, 40]). Many of these works focus on specific weighted average aggregators like linear pooling (weighted arithmetic mean) [51] and logarithmic pooling (normalized weighted geometric mean) [42, 40], and the goal is to estimate the optimal weights from data. However, those weighted average aggregators are not necessarily the

optimal ( Bayesian ) aggregator unless the underlying information structure satisfies some strict conditions (e.g., experts' forecasts are equal to the true probability of the event plus normally distributed noises [48]). Our work aims to understand the sample complexity of Bayesian aggregation, namely how many samples are needed to approximate the Bayesian aggregator.

Our work is closely related to a recent work [4] on Bayesian aggregation via online learning . For the case of conditionally independent experts, [4] shows that the Bayesian aggregator can be approximated with ε = ˜ O ( n √ T ) regret. By an online-to-batch conversion, this implies a T = ˜ O ( n 2 ε 2 ) sample complexity upper bound for the batch learning setting. Our paper studies the batch learning setting. For the conditional independence case, we obtain an improved bound of ˜ O ( 1 ε 2 ) .

Robust forecast aggregation Recent works on 'robust forecast aggregation' [2, 39, 18, 3] also study information aggregation problems where the principal does not know the underlying information structure. They take a worst-case approach, assuming that the information structure is chosen adversarially. This often leads to negative results: e.g., a bad approximation ratio [2, 39] or a degenerate maximin aggregator that solely relies on a random expert's opinion [18, 3]. In contrast, we assume sample access to the unknown information structure. Our sample complexity approach is orthogonal and complementary to the robust forecast aggregation approach.

Sample complexity of mechanism design Our work may remind the reader of a long line of research on the sample complexity of revenue maximization in mechanism design (e.g., [17, 20, 36, 19, 5, 11, 29, 10, 25, 50, 28]). In particular, [28] gives a general framework to bound the sample complexity for mechanism design problems that satisfy a 'strong monotonicity' property, but this property is not satisfied by our forecast aggregation problem. A key observation from this line of works is that the number of samples needed to learn an ε -optimal auction for n ≥ 1 independent bidders is increasing in n , because when there are more bidders, although we can obtain higher revenue, the optimal auction benchmark is also improved. We see a similar phenomenon that the sample complexity of forecast aggregation increases with the number of experts in the general case, but not in the case where experts are conditionally independent.

2 Model

2.1 Forecast Aggregation

There are n ≥ 2 experts and one principal. The principal wants to predict the probability that a binary event ω ∈ Ω = { 0 , 1 } happens ( ω = 1 ), based on information provided by the experts. For example, ω may represent whether it will rain tomorrow. We present binary events to simplify notations. All our results can be generalized to multi-outcome events with | Ω | > 2 (see Section 6). We also refer to ω as 'the state of the world'. Each expert i = 1 , . . . , n observes some private signal s i ∈ S i that is correlated with ω , where S i denotes the space of all possible signals of expert i . We assume for now that S i is finite, with size |S i | = m . We relax this assumption in Section 5 where we consider conditionally independent signals. Let S = S 1 ×··· × S n be the joint signal space of all experts; | S | = m n . Let P be a distribution over S × Ω , namely, a joint distribution of signals s = ( s 1 , . . . , s n ) and event ω . Since the space S × Ω is discrete, we can use P ( · ) to denote the probability: P ( s , ω ) = Pr P [ s , ω ] . Signals of different experts can be correlated conditioned on ω . We assume that each expert i knows the marginal joint distribution of their own signal s i and ω , P ( s i , ω ) . Neither any expert nor the principal knows the entire distribution P . Each expert i reports to the principal a forecast (or prediction) r i for the event ω , which is equal to the conditional probability of ω = 1 given their signal s i : 3

We note that r i depends on s i and P , but not on ω or other experts' signals s -i . Let r = ( r 1 , . . . , r n ) ∈ [0 , 1] n denote the reports (joint report) of all experts. We sometimes use r -i to

denote the reports of all experts except i . The principal aggregates the experts' reports r into a single forecast f ( r ) using some aggregation function , or aggregator , f : [0 , 1] n → [0 , 1] . We measure the performance of an aggregator by the mean squared loss:

L P ( f ) = E P [ | f ( r ) -ω | 2 ] . (2) The notation E P [ · ] makes it explicit that the expectation is over the random draw of ( s , ω ) ∼ P followed by letting r i = P ( ω = 1 | s i ) . We omit P and write E [ · ] when it is clear from the context.

Let f ∗ be the optimal aggregator with respect to P , which minimizes L P ( f ) :

We have the following characterization of f ∗ and L P ( f ) : f ∗ is equal to the Bayesian aggregator, which computes the posterior probability of ω = 1 given all the reports r = ( r 1 , . . . , r n ) . And the difference between the loss of f and the loss of f ∗ is equal to their expected squared difference.

Lemma 2.1. The optimal aggregator f ∗ and any aggregator f satisfy:

f ∗ ( r ) = P ( ω = 1 | r ) , for almost every r .

An aggregator f is ε -optimal (with respect to P ) if L P ( f ) ≤ L P ( f ∗ ) + ε . By Lemma 2.1, this is equivalent to E P [ | f ( r ) -f ∗ ( r ) | 2 ] ≤ ε . For an ε -optimal f , we also say it ε -approximates f ∗ .

L P ( f ) -L P ( f ∗ ) = E P [ | f ( r ) -f ∗ ( r ) | 2 ] .

Definition 2.2. An aggregator f is ε -optimal (with respect to P ) if E P [ | f ( r ) -f ∗ ( r ) | 2 ] ≤ ε .

Discussion of the benchmark f ∗ Our benchmark, the Bayesian aggregator f ∗ , is common in the forecast aggregation literature (e.g., [27, 26, 45]). It is stronger than the typical 'best expert' benchmark in no-regret learning (e.g., [13, 22]), but weaker than the 'omniscient' aggregator that has access to the experts' signals : f omni ( s ) = P ( ω = 1 | s ) . If there is a one-to-one mapping between signals s and reports r , then f omni and f ∗ are the same. Otherwise, f omni could be much stronger than f ∗ and an ε -approximation to f omni using experts' reports only is not always possible. 4 In contrast, an ε -approximation to f ∗ is always achievable (in fact, achieved by f ∗ itself). The difference between f ∗ and f omni is known as the difference between 'aggregating forecasts' and 'aggregating information sets' in the literature [27, p.198-199], [26, p.168-169], [45, p.143].

2.2 Sample Complexity of Forecast Aggregation

The principal has access to T i.i.d. samples of forecasts and event realizations drawn from the underlying unknown distribution P :

Here, we implicitly regard P as a distribution over ( r , ω ) instead of ( s , ω ) . The principal uses samples S T to learn an aggregator ˆ f = ˆ f S T , in order to approximate f ∗ . Our main question is:

How many samples are necessary and sufficient for finding an ε -optimal aggregator ˆ f (with probability at least 1 -δ over the random draw of samples)?

The answer to the above question depends on the family of distributions we are interested in. Let P denote a family of distributions over S × Ω . It could be the set of all distributions over S × Ω , or in Section 5 we will only consider distributions where the signals are independent conditioned on ω . We define the sample complexity of forecast aggregation, with respect to P , formally:

Definition 2.3. The sample complexity of forecast aggregation (with respect to P ) is the minimum function T P ( · , · ) of ε, δ ∈ (0 , 1) , such that: if T ≥ T P ( ε, δ ) , then for any distribution P ∈ P , with probability at least 1 -δ over the random draw of T samples S T from P (and over the randomness of the learning procedure if it is randomized), we can obtain an aggregator ˆ f = ˆ f S T satisfying E P [ | ˆ f ( r ) -f ∗ ( r ) | 2 ] ≤ ε .

The principal is assumed to know the family of distributions P but not the specific distribution P ∈ P . There should be at least two different distributions in P . Otherwise, the principal knows what the distribution is and there is no need for learning.

3 Preliminaries

In this section, we briefly introduce some notions that will be used in our analysis of the sample complexity of forecast aggregation, including some definitions of distances between distributions, the distribution learning problem, and the distinguishing distributions problem.

Distances between distributions We recall two distance metrics for discrete distributions: the total variation distance and the (squared) Hellinger distance .

Definition 3.1. Let D 1 , D 2 be two distributions on a discrete space X .

The total variation distance has the following well-known property that upper bounds the difference between the expected values of a function on two distributions:

The total variation distance between D 1 and D 2 is d TV ( D 1 , D 2 ) = 1 2 ∑ x ∈X ∣ ∣ D 1 ( x ) -D 2 ( x ) ∣ ∣ . · The squared Hellinger distance between D 1 and D 2 is d 2 H ( D 1 , D 2 ) = 1 2 ∑ x ∈X (√ D 1 ( x ) -√ D 2 ( x ) ) 2 = 1 -∑ x ∈X √ D 1 ( x ) D 2 ( x ) .

In Appendix A we give some properties of the Hellinger distance that will be used in the proofs.

Distribution learning in total variation distance Our analysis of the sample complexity of the forecast aggregation problem will leverage on the sample complexity of another learning problem: learning discrete distributions in total variation distance . We review this problem below.

Let D be a family of distributions over X . The sample complexity of learning distributions in D within total variation distance ε is the minimum function T TV D ( ε, δ ) , such that: if T ≥ T TV D ( ε, δ ) , then for any distribution D ∈ D , with probability at least 1 -δ over the random draw of T samples from D , we can obtain (from the T samples) a distribution ˆ D such that d TV ( ˆ D,D ) ≤ ε .

Proposition 3.3 (e.g., [12, 33]) . Let D all be the set of all distributions over X . Then, T TV D all ( ε, δ ) = Θ ( |X| +log(1 /δ ) ε 2 ) . In particular, the upper bound can be achieved by using the empirical estimate ˆ D emp (which is the uniform distribution over the T samples). The lower bound holds regardless of what learning algorithm is used.

Distinguishing distributions Another learning problem that we will leverage on is the problem of distinguishing (two) distributions : given samples from a distribution randomly chosen from { D 1 , D 2 } , we are to guess whether the samples are from D 1 or D 2 . The sample complexity of distinguishing distributions is characterized by the squared Hellinger distance. It is known that at least T = Ω ( 1 d 2 H ( D 1 ,D 2 ) log 1 δ ) samples are needed to distinguish two distributions with probability at least 1 -δ . See Appendix A for a formal statement of this result.

4 Sample Complexity for General Distributions

In this section we characterize the sample complexity of forecast aggregation for general distributions P . We give an exponential (in the number of experts, n ) upper bound and an exponential lower bound on the sample complexity, as follows:

Theorem 4.1. Let P all be the set of all distributions over S × Ω , with | S | = m n . Suppose n ≥ 2 . The sample complexity of forecast aggregation with respect to P all is

This theorem is for n ≥ 2 . When there is only one expert ( n = 1 ), there is no need to learn to aggregate because the optimal 'aggregator' f ∗ simply outputs the forecast given by the only expert: f ∗ ( r 1 ) = P ( ω = 1 | r 1 ) = P ( ω = 1 | s 1 ) = r 1 . The sample complexity is 0 in this case.

There is a gap in the dependency on ε in the upper bound and the lower bound in Theorem 4.1. We conjecture that the tight dependency on ε should be 1 ε (so the lower bound is tight). See Section 7 for a detailed discussion of this conjecture, where we show that the dependency on ε in the upper bound can be improved to 1 ε for a large family of distributions.

4.1 Proof of the Upper Bound

In this subsection we prove the O ( m n +log(1 /δ ) ε 2 ) upper bound in Theorem 4.1. This is a direct corollary of the distribution learning result introduced in Section 3.

We regard P as a distribution over r and ω instead of over s and ω . Then P is a discrete distribution with support size at most 2 m n because each possible report r i ∈ [0 , 1] corresponds to some discrete signal s i in S i . Let ˆ P emp be the empirical distribution of reports and event realizations: ˆ P emp = Uniform { ( r (1) , ω (1) ) , . . . , ( r ( T ) , ω ( T ) ) } . By Proposition 3.3, with probability at least 1 -δ over the random draw of T = O ( 2 m n +log(1 /δ ) ε 2 ) samples, we have d TV ( ˆ P emp , P ) ≤ ε . According to Fact 3.2, and by the definition of L P ( f ) , we have: for any aggregator f : [0 , 1] n → [0 , 1] ,

which means that ˆ f emp is a 2 ε -optimal aggregator for P .

4.2 Proof of the Lower Bound

In this subsection we prove the Ω ( m n -2 +log(1 /δ ) ε ) lower bound in Theorem 4.1. The main idea is a reduction from the distribution learning problem (defined in Section 3) for a specific family D of distributions over the joint signal space S = S 1 ×··· × S n . We construct a corresponding family of distributions P = { P D : D ∈ D} for the forecast aggregation problem, such that, if we can obtain an ε -optimal aggregator ˆ f for P D , then we can convert ˆ f into a distribution ˆ D such that d TV ( ˆ D,D ) ≤ O ( √ ε ) . We then prove that learning D within total variation distance ε TV = O ( √ ε ) requires Ω ( m n -2 +log(1 /δ ) ε 2 TV ) = Ω ( m n -2 +log(1 /δ ) ε ) samples. This gives the sample complexity lower bound for the forecast aggregation problem for P (and hence P all ).

Definition 4.2. We say a family of distributions D satisfies

We will need a family of distributions D that satisfies the following three properties:

B -uniformly bounded , if: D ( s ) ≤ B | S | = B m n , ∀ s ∈ S , ∀ D ∈ D , where B ≥ 1 is a constant.

same marginal across distributions , if: for any D,D ′ ∈ D , any i , any s i ∈ S i , D ( s i ) = D ′ ( s i ) .

distinct marginals across signals , if: for any D ∈ D , any i , any s i = s ′ i ∈ S i , D ( s i ) = D ( s ′ i ) .

/negationslash

/negationslash

How do we construct the family P ? For each distribution D ∈ D , we construct distribution P D as follows: the marginal distribution of ω is Uniform { 0 , 1 } , i.e., P D ( ω = 0) = P D ( ω = 1) = 1 2 ; conditioning on ω = 0 , the joint signal s is uniformly distributed: P D ( s | ω = 0) = 1 | S | = 1 m n , ∀ s ∈ S ; conditioning on ω = 1 , the joint signal is distributed according to D : P D ( s | ω = 1) = D ( s ) , ∀ s ∈ S . The family P is { P D : D ∈ D} .

We show that if we can obtain ε -optimal aggregators for distributions in P , then we can learn the distributions in D within total variation distance (1+ B ) 2 √ ε , and thus the sample complexity of the former is lower bounded by the sample complexity of the latter:

Lemma 4.3. Let D be a family of distributions that satisfies the three properties in Definition 4.2. Let P = { P D : D ∈ D} be defined above. Then, T P ( ε, δ ) ≥ T TV D ( (1 + B ) 2 √ ε, δ ) .

Proof sketch of Lemma 4.3. The full proof is in Appendix E.1. We give a sketch here. According to the definition of P D , by Bayes' rule, we have

The 'distinct marginals across signals' property in Definition 4.2 ensures that there is a one-to-one mapping between signal s i and report r i = P D ( ω = 1 | s i ) , and hence a one-to-one mapping between joint signal s and joint report r . So, the Bayesian aggregator f ∗ satisfies f ∗ ( r ) = P D ( ω = 1 r ) = P D ( ω = 1 s ) = D ( s ) 1 /m n + D ( s ) . This gives

Suppose we have obtained an ε -optimal aggregator ˆ f for P D , E P D [ | ˆ f ( r ) -f ∗ ( r ) | 2 ] ≤ ε , then we convert ˆ f into ˆ D by letting ˆ D ( s ) = 1 m n ˆ f ( r ) 1 -ˆ f ( r ) . The total variation distance between ˆ D and D is:

The ' B -uniformly bounded' property in Definition 4.2 ensures D ( s ) = O ( 1 m n ) , which has two consequences: (1) P D ( s ) = O ( 1 m n ) ; (2) f ∗ ( r ) = O (1) , which implies ∣ ∣ ˆ f ( r ) 1 -ˆ f ( r ) -f ∗ ( r ) 1 -f ∗ ( r ) ∣ ∣ = O ( | ˆ f ( r ) -f ∗ ( r ) | ) due to Lipschitz property of the function x 1 -x for x = O (1) . These two consequences imply

So, we obtain d TV ( ˆ D,D ) ≤ O ( √ ε ) = (1 + B ) 2 √ ε .

There is a subtlety, however: In the distribution learning problem we are given samples from D , which are signals { s ( t ) } T t =1 . We need to convert them into the corresponding reports { r ( t ) } T t =1 as the samples for the forecast aggregation problem. To do this we need to know P D or D , which we do not know. So, we make use of the 'same marginal across distributions' property in Definition 4.2 here: because all the distributions D ∈ D have the same marginal probability D ( s i ) = D ′ ( s i ) (but possibly different joint probabilities D ( s ) = D ′ ( s ) ), we are able to compute the report

/negationslash separately for each expert i without knowing D , since we know what the family D is. This allows us to reduce the distribution learning problem for D to the forecast aggregation problem for P .

Then, we find a family of distributions D that satisfies the three properties in Definition 4.2 and requires many samples to learn.

Proposition 4.4. There exists a family of distributions D that satisfies the three properties in Definition 4.2 (with B = e + 1 2 ) and requires T TV D ( ε TV , δ ) = Ω ( m n -2 +log(1 /δ ) ε 2 TV ) samples to learn.

The above sample complexity is smaller than the Ω ( | S | +log(1 /δ ) ε 2 TV ) lower bound in Proposition 3.3 because we are restricting to a smaller set of distributions than the set of all distributions over S . The proof of Proposition 4.4 is analogous to a textbook proof of Proposition 3.3, which uses reductions from the distinguishing distributions problem. See details in Appendix E.2.

Finishing the proof of Theorem 4.1: By Lemma 4.3 and Proposition 4.4, plugging in ε TV = (1 + B ) 2 √ ε with B = e + 1 2 , we obtain the lower bound on the sample complexity of forecast aggregation for P (and hence for P all ):

5 Sample Complexity for Conditionally Independent Distributions

Section 4 proved that learning ε -optimal aggregators for all discrete distributions needs exponentially many samples. As shown in the proof, this large sample complexity is because the experts' signals can be arbitrarily correlated conditioned on the event ω . Accurate estimation of such correlation requires many samples. So, in this section we restrict attentions to the case where the experts' signals are conditionally independent. It turns out that an ε -optimal aggregator can be learned using only O ( 1 ε 2 log 1 εδ ) samples in this case, which does not depend on n . The assumption of discrete signal space can be relaxed here. We also investigate two special and interesting families of conditionally independent distributions that admit an even smaller sample complexity of O ( 1 ε log 1 δ ) .

5.1 General Conditionally Independent Distributions

Let P be a conditionally independent distributions over S × Ω , namely, P ( s | ω ) = ∏ n i =1 P ( s i | ω ) for all s ∈ S , for ω ∈ { 0 , 1 } . Here, S i can be a continuous space, in which case, P ( · | ω ) represents a density function. We introduce some additional notations. Let p = P ( ω = 1) be the prior probability of ω = 1 . For technical convenience we assume p ∈ (0 , 1) . Define

We will be working with ratios like ' r i 1 -r i ' and ' p 1 -p ' a lot in this section. We will use the following characterization of the optimal aggregator f ∗ for conditionally independent distributions:

Lemma 5.1 ([8]) . For conditionally independent distribution P , given signals s = ( s i ) n i =1 , with corresponding reports r = ( r i ) n i =1 where r i = P ( ω = 1 | s i ) , the posterior probability of ω = 1 is:

The main result of this section is that the sample complexity of forecast aggregation with respect to all conditionally independent distributions is between Ω( 1 ε log 1 δ ) and O ( 1 ε 2 log 1 εδ ) :

Lemma 5.1 implies that one way to learn f ∗ is to simply learn the value of ρ . If we can learn ρ with accuracy √ ε n , then we can learn ρ n -1 with accuracy √ ε and obtain an ˆ f that is ε -close to f ∗ for every possible input r ∈ [0 , 1] n . However, by standard concentration inequalities, learning ρ with accuracy √ ε n requires ˜ O ( n 2 ρε ) samples, which is larger than the ˜ O ( 1 ε 2 ) bound we will prove. The key is that we do not actually need ˆ f ( r ) to be close to f ∗ ( r ) for every r ∈ [0 , 1] n + ; we only need the expectation E [ | ˆ f ( r ) -f ∗ ( r ) | 2 ] ≤ ε . This allows us to prove a smaller sample complexity bound, using a pseudo-dimension argument.

Theorem 5.2. Let P ind be the set of all conditionally independent distributions over S × Ω . Suppose n ≥ 2 . The sample complexity of forecast aggregation with respect to P ind is

We provide the main ideas of the proof of Theorem 5.2 here. The upper bound O ( 1 ε 2 log 1 εδ ) is a corollary of our theorem for multi-outcome events (Theorem C.1), so we only give a sketch here. We note that, according to Lemma 5.1, the optimal aggregator has the form f ∗ ( r ) = 1 1+ ρ n -1 ∏ n 1 -r i .

i =1 r i We consider the class of aggregators F = { f θ : f θ ( r ) = 1 1+ θ n -1 ∏ n i =1 1 -r i r i } parameterized by

We prove the Ω( 1 ε log 1 δ ) lower bound by a reduction from the distinguishing distributions problem (introduced in Section 3). We construct two conditionally independent distributions P 1 , P 2 over

θ ∈ (0 , + ∞ ) . The class of loss functions G = { g θ : g θ ( r , ω ) = | f θ ( r ) -ω | 2 } associated with F has pseudo-dimension Pdim( G ) = O (1) . By the known result (e.g., [1]) that the pseudo-dimension gives a sample complexity upper bound on the uniform convergence of a class of functions, we conclude that the empirically optimal aggregator in F must be O ( ε ) -optimal on the true distribution (with probability at least 1 -δ ), given O ( 1 ε 2 (Pdim( G ) log 1 ε +log 1 δ ) ) = O ( 1 ε 2 log 1 εδ ) samples.

the space Ω × S that differ by d 2 H ( P 1 , P 2 ) = O ( ε ) in squared Hellinger distance. Specifically, P 1 has prior P 1 ( ω = 1) = 0 . 5 -O ( 1 n ) + O ( √ ε n ) and P 2 has prior P 2 ( ω = 1) = 0 . 5 -O ( 1 n ) -O ( √ ε n ) ; the conditional distributions of each signal, P 1 ( s i | ω ) and P 2 ( s i | ω ) , differ by O ( ε n ) in squared Hellinger distance; taking the product of n signals, P 1 ( s | ω ) and P 2 ( s | ω ) differ by O ( ε ) . The distinguishing distributions problem asks: given T samples from either P 1 or P 2 , tell which distribution the samples come from. We show that, if we can solve the forecast aggregation problem, namely, ε -approximate f ∗ ( r ) = 1 1+ ρ n -1 ∏ n i =1 1 -r i r i , then we can estimate ρ with accuracy O ( √ ε n ) , and hence distinguish P 1 and P 2 . But distinguishing P 1 and P 2 requires Ω( 1 d 2 H ( P 1 ,P 2 ) log 1 δ ) = Ω( 1 ε log 1 δ ) samples. This gives the lower bound we claimed. See details in Appendix F.1.

5.2 Strongly and Weakly Informative Experts

While the sample complexity of forecast aggregation for general conditionally independent distributions is O ( 1 ε 2 log 1 εδ ) , under further assumptions this bound can be improved. In particular, we find two special yet natural families of conditionally independent distributions that admit O ( 1 ε log 1 δ ) sample complexity. In these two cases, the experts are either 'very informative' or 'very noninformative'. Roughly speaking, an expert is 'very informative' if the conditional distributions of the expert's signal under event ω = 0 and ω = 1 are significantly different, so the expert's prediction r i is away from the prior p . An expert is 'very non-informative' if the opposite is true. Intuitively, an expert being informative should help aggregation and hence reduce the sample complexity. Interestingly though, we show that even if the experts are non-informative the sample complexity of forecast aggregation can also be reduced. See details in Appendix B.

6 Extension: Multi-Outcome Events

Our main results regarding the sample complexity of forecast aggregation for binary events (Theorems 4.1 and 5.2) can be generalized to multi-outcome events with | Ω | > 2 . We prove that: for general distributions, the sample complexity is ˜ Ω ( m n -2 ε ) = Ω ( m n -2 +log(1 /δ ) ε ) ≤ T P ( ε, δ ) ≤ O ( | Ω | m n +log(1 /δ ) ε 2 ) = ˜ O ( | Ω | m n ε 2 ) ; for conditionally independent distributions, the sample complexity is ˜ Ω ( 1 ε ) = Ω ( 1 ε log 1 δ ) ≤ T P ind ( ε, δ ) ≤ O ( | Ω | log | Ω | ε 2 log 1 ε + 1 ε 2 log 1 δ ) = ˜ O ( | Ω | ε 2 ) . See Appendix C for details.

7 Conclusion and Discussion

In this work, we showed an exponential gap between the sample complexity of forecast aggregation for general distributions, ˜ Ω( m n -2 ε ) , and conditionally independent distributions, ˜ O ( 1 ε 2 ) . This gap is due to the need of estimating the conditional correlation between experts in the general case, which is not needed in the conditional independence case. Notably, the bound ˜ O ( 1 ε 2 ) for conditionally independent distributions does not depend on the number of experts.

We discuss the dependency of the sample complexity on ε and other directions for future works:

The dependency on ε An open question left by our work is the dependency of the sample complexity on the parameter ε . We conjecture that the tight dependency should be 1 ε (so our lower bounds are tight). This is supported by the following evidence:

Theorem 7.1. For the case of | Ω | = 2 and for general distributions, if the distribution P has a minimum joint probability min ( s ,ω ) ∈ S × Ω P ( s , ω ) > c m n for some c > 0 , then the sample complexity of forecast aggregation is at most O ( m n cε ( n log m +log 1 δ ) ) = ˜ O ( nm n cε ) . 5

In particular, this theorem can be applied to distributions that are close to uniform, where P ( s , ω ) ≈ 1 | S × Ω | = 1 2 m n , giving a bound of ˜ O ( nm n ε ) . Notably, the set of distributions we constructed in the proof of the ˜ Ω( m n -2 ε ) lower bound in Theorem 4.1 are also close to uniform. This means that

close-to-uniform distributions have a tight sample complexity bound of the form Θ( f ( n,m ) ε ) , not Θ( f ( n,m ) ε 2 ) . Moreover, since close-to-uniform distributions are the 'most difficult' distributions to learn in the distribution learning problem, it is likely that they are also the most difficult distributions for the forecast aggregation problem, and therefore the tight sample complexity of forecast aggregation should be determined by the sample complexity for those distributions, which is Θ( f ( n,m ) ε ) .

Other future directions

The middle ground between fully correlated experts and conditionally independent experts: An example is the partial evidence model in [4]. Applying [4]'s results, one can show that the sample complexity of forecast aggregation in the partial evidence model is at most ˜ O ( n 2 ε 2 ) . 6 Giving a lower bound for the partial evidence model and exploring other intermediate models is open.

Weaker benchmark: Since the Bayesian aggregator needs exponentially many samples to approximate, can we find a weaker yet meaningful benchmark with a small sample complexity?

Samples vs experts: In reality, obtaining samples of experts' historical forecasts can be difficult, while recruiting experts is easy. Can we achieve better aggregation by recruiting more experts instead of collecting more samples? How many experts do we need?

Eliciting more information: Previous works on information elicitation and aggregation have noticed that better aggregation can be achieved by eliciting more information than agents' own predictions, for example, also eliciting each agent's prediction about other agents' predictions (e.g., [41, 47, 31, 14]). One can ask whether and how eliciting more information can help to reduce the sample complexity of information aggregation.

Continuous distributions: In our model the random variable ω to be predicted is discrete. One can study a setting where ω is a continuous random variable and the experts report, e.g., the means of their posterior beliefs about ω . The results for continuous random variables might be very different from the results in this work.

Other loss functions: We focused on the squared loss E [ | f ( r ) -ω | 2 ] due to its popularity in machine learning problems and its useful property that the difference between the squared losses of any aggregator and the optimal aggregator is equal to their expected squared difference (Lemma 2.1). Alternatively, one can consider other loss functions like the logarithmic loss E [ ω log( f ( r ))+(1 -ω ) log(1 -f ( r ))] and the absolute loss E [ | f ( r ) -ω | ] . There might be some technical challenges in the analysis of sample complexity for those loss functions, though: e.g., the logarithmic loss can be unbounded [4, 40] and the absolute loss does not enjoy a property like Lemma 2.1.

Acknowledgments and Disclosure of Funding

We would like to thank Yannai Gonczarowski, Ariel Procaccia, Milind Tambe, David Parkes, Bo Waggoner, Rafael Frongillo, Grant Schoenebeck, Yuqing Kong, and anonymous reviewers for their helpful comments. This research is based upon work supported in part by the National Science Foundation under grants no. IIS-2007887 and no. IIS-2147187.

References

Martin Anthony and Peter L. Bartlett. Neural Network Learning: Theoretical Foundations . Cambridge University Press, 1 edition, November 1999.

Itai Arieli, Yakov Babichenko, and Rann Smorodinsky. Robust forecast aggregation. Proceedings of the National Academy of Sciences , 115(52), December 2018.

Itai Arieli, Yakov Babichenko, Inbal Talgam-Cohen, and Konstantin Zabarnyi. Universally Robust Information Aggregation for Binary Decisions. In Proceedings of the 24th ACM Conference on Economics and Computation , pages 118-118, London United Kingdom, July 2023. ACM.

Yakov Babichenko and Dan Garber. Learning Optimal Forecast Aggregation in Partial Evidence Environments. Mathematics of Operations Research , 46(2):628-641, May 2021.

Maria-Florina F Balcan, Tuomas Sandholm, and Ellen Vitercik. Sample complexity of automated mechanism design. Advances in Neural Information Processing Systems , 29, 2016.

Jonathan Baron, Barbara A. Mellers, Philip E. Tetlock, Eric Stone, and Lyle H. Ungar. Two Reasons to Make Aggregated Probability Forecasts More Extreme. Decision Analysis , 11(2):133-145, June 2014.

John M. Bates and Clive W. J. Granger. The Combination of Forecasts. Journal of the Operational Research Society , 20(4):451-468, December 1969.

Robert F. Bordley. A Multiplicative Formula for Aggregating Probability Assessments. Management Science , 28(10):1137-1148, 1982. Publisher: INFORMS.

Glenn W. Brier. Verification of Forecasts Expressed in Terms of Probability. Monthly Weather Review , 78(1):1-3, January 1950.

Johannes Brustle, Yang Cai, and Constantinos Daskalakis. Multi-Item Mechanisms without ItemIndependence: Learnability via Robustness. In Proceedings of the 21st ACM Conference on Economics and Computation , pages 715-761, Virtual Event Hungary, July 2020. ACM.

Yang Cai and Constantinos Daskalakis. Learning Multi-Item Auctions with (or without) Samples. In 2017 IEEE 58th Annual Symposium on Foundations of Computer Science (FOCS) , pages 516-527, Berkeley, CA, October 2017. IEEE.

Clément L. Canonne. A short note on learning discrete distributions, 2020.

Nicolo Cesa-Bianchi and Gabor Lugosi. Prediction, Learning, and Games . Cambridge University Press, Cambridge, 2006.

Yi-Chun Chen, Manuel Mueller-Frank, and Mallesh Pai. The Wisdom of the Crowd and Higher-Order Beliefs. In Proceedings of the 24th ACM Conference on Economics and Computation , pages 450-450, London United Kingdom, July 2023. ACM.

Gerda Claeskens, Jan R. Magnus, Andrey L. Vasnev, and Wendun Wang. The forecast combination puzzle: A simple theoretical explanation. International Journal of Forecasting , 32(3):754-762, July 2016.

Robert T. Clemen. Combining forecasts: A review and annotated bibliography. International Journal of Forecasting , 5(4):559-583, January 1989.

Richard Cole and Tim Roughgarden. The sample complexity of revenue maximization. In Proceedings of the 46th Annual ACM Symposium on Theory of Computing - STOC '14 , pages 243-252, New York, New York, 2014. ACM Press.

Henrique De Oliveira, Yuhta Ishii, and Xiao Lin. Robust Merging of Information. In Proceedings of the 22nd ACM Conference on Economics and Computation , pages 341-342, Budapest Hungary, July 2021. ACM.

Nikhil R. Devanur, Zhiyi Huang, and Christos-Alexandros Psomas. The sample complexity of auctions with side information. In Proceedings of the forty-eighth annual ACM symposium on Theory of Computing , pages 426-439, Cambridge MA USA, June 2016. ACM.

Peerapong Dhangwatnotai, Tim Roughgarden, and Qiqi Yan. Revenue maximization with a single sample. Games and Economic Behavior , 91:318-333, May 2015.

R. Glen Donaldson and Mark Kamstra. Forecast combining with neural networks. Journal of Forecasting , 15(1):49-61, January 1996.

Rafael Frongillo, Robert Gomez, Anish Thilagar, and Bo Waggoner. Efficient Competitions and Online Learning with Strategic Forecasters. In Proceedings of the 22nd ACM Conference on Economics and Computation , pages 479-496, Budapest Hungary, July 2021. ACM.

Véronique Genre, Geoff Kenny, Aidan Meyler, and Allan Timmermann. Combining expert forecasts: Can anything beat the simple average? International Journal of Forecasting , 29(1):108-121, January 2013.

John Geweke and Charles Whiteman. Chapter 1 Bayesian Forecasting. In Handbook of Economic Forecasting , volume 1, pages 3-80. Elsevier, 2006.

Yannai A. Gonczarowski and S. Matthew Weinberg. The Sample Complexity of Up-to/epsilon1 Multidimensional Revenue Maximization. Journal of the ACM , 68(3):1-28, March 2021.

Clive W. J. Granger. Invited review combining forecasts-twenty years later. Journal of Forecasting , 8(3):167-173, July 1989.

Clive W. J. Granger and Ramu Ramanathan. Improved methods of combining forecasts. Journal of Forecasting , 3(2):197-204, April 1984.

Chenghao Guo, Zhiyi Huang, Zhihao Gavin Tang, and Xinzhi Zhang. Generalizing complex hypotheses on product distributions: Auctions, prophet inequalities, and pandora's problem. In Conference on Learning Theory , pages 2248-2288. PMLR, 2021.

Chenghao Guo, Zhiyi Huang, and Xinzhi Zhang. Settling the sample complexity of single-parameter revenue maximization. In Proceedings of the 51st Annual ACM SIGACT Symposium on Theory of Computing - STOC 2019 , pages 662-673, Phoenix, AZ, USA, 2019. ACM Press.

Sunil Gupta and Peter C. Wilton. Combination of Forecasts: An Extension. Management Science , 33(3):356-372, 1987. Publisher: INFORMS.

Yuqing Kong, Yunqi Li, Yubo Zhang, Zhihuan Huang, and Jinzhao Wu. Eliciting Thinking Hierarchy without a Prior. In Advances in Neural Information Processing Systems , volume 35, pages 13329-13341. Curran Associates, Inc., 2022.

Jasper Lee. Lecture 11: Distinguishing (discrete) distributions, 2020.

Jasper Lee. Lecture 12: Learning Discrete Distributions, 2020.

Spyros Makridakis, Evangelos Spiliotis, and Vassilios Assimakopoulos. The M4 Competition: 100,000 time series and 61 forecasting methods. International Journal of Forecasting , 36(1):54-74, January 2020.

Spyros Makridakis and Robert L. Winkler. Averages of Forecasts: Some Empirical Results. Management Science , 29(9):987-996, September 1983.

Jamie H Morgenstern and Tim Roughgarden. On the pseudo-dimension of nearly optimal auctions. Advances in Neural Information Processing Systems , 28, 2015.

Peter A. Morris. Decision Analysis Expert Use. Management Science , 20(9):1233-1241, 1974. Publisher: INFORMS.

Paul Newbold and Clive W. J. Granger. Experience with Forecasting Univariate Time Series and the Combination of Forecasts. Journal of the Royal Statistical Society. Series A (General) , 137(2):131-146, 1974.

Eric Neyman and Tim Roughgarden. Are You Smarter Than a Random Expert? The Robust Aggregation of Substitutable Signals. In Proceedings of the 23rd ACM Conference on Economics and Computation , pages 990-1012, Boulder CO USA, July 2022. ACM.

Eric Neyman and Tim Roughgarden. No-regret learning with unbounded losses: The case of logarithmic pooling. arXiv preprint arXiv:2202.11219 , 2022.

Dražen Prelec, H. Sebastian Seung, and John McCoy. A solution to the single-question crowd wisdom problem. Nature , 541(7638):532-535, January 2017.

Ville A. Satopää, Jonathan Baron, Dean P. Foster, Barbara A. Mellers, Philip E. Tetlock, and Lyle H. Ungar. Combining multiple probability predictions using a simple logit model. International Journal of Forecasting , 30(2):344-356, April 2014.

Jeremy Smith and Kenneth F. Wallis. A Simple Explanation of the Forecast Combination Puzzle. Oxford Bulletin of Economics and Statistics , 71(3):331-355, June 2009.

James H. Stock and Mark W. Watson. Combination forecasts of output growth in a seven-country data set. Journal of Forecasting , 23(6):405-430, September 2004.

Allan Timmermann. Chapter 4 Forecast Combinations. In Handbook of Economic Forecasting , volume 1, pages 135-196. Elsevier, 2006.

Alexandre B. Tsybakov. Introduction to nonparametric estimation . Springer series in statistics. Springer, New York ; London, 2009. OCLC: ocn300399286.

Juntao Wang, Yang Liu, and Yiling Chen. Forecast Aggregation via Peer Prediction. Proceedings of the AAAI Conference on Human Computation and Crowdsourcing , 9(1):131-142, October 2021.

Robert L. Winkler. Combining Probability Distributions from Dependent Information Sources. Management Science , 27(4):479-488, 1981. Publisher: INFORMS.

Robert L. Winkler and Spyros Makridakis. The Combination of Forecasts. Journal of the Royal Statistical Society. Series A (General) , 146(2):150, 1983.

Chunxue Yang and Xiaohui Bei. Learning Optimal Auctions with Correlated Valuations from Samples. In Proceedings of the 38th International Conference on Machine Learning , volume 139 of Proceedings of Machine Learning Research , pages 11716-11726. PMLR, July 2021.

Yuhong Yang. Combining Forecasting Procedures: Some Theoretical Results. Econometric Theory , 20(1):176-222, 2004. Publisher: Cambridge University Press.

A Additional Preliminaries

A.1 Properties of Hellinger Distance

We review some useful properties of the Hellinger distance. First, the Hellinger distance gives upper bounds on the total variation distance:

Fact A.1 (e.g., Lemma 2.3 in [46]) .

Second, we will use the following lemma to upper bound the squared Hellinger distance between two distributions that are close to each other:

Lemma A.2. Let D 1 and D 2 be two distributions on X satisfying 1 -ε ≤ D 2 ( x ) D 1 ( x ) ≤ 1 + ε , ∀ x ∈ X . Then, d 2 H ( D 1 , D 2 ) ≤ 1 2 ε 2 .

Proof. By definition,

If 1 -ε ≤ D 2 ( x ) D 1 ( x ) < 1 , then we have ( 1 -D 2 ( x ) D 1 ( x ) ) 2 ≤ ( 1 1 -ε ) 2 ≤ ( 1 -(1 -ε ) ) 2 = ε 2 . If √

1 + ε ≥ D 2 ( x ) D 1 ( x ) > 1 , then we have 1 -D 2 ( x ) D 1 ( x ) 2 ≤ 1 + ε -1 2 ≤ (1 + ε ) -1 2 = ε 2 . These two cases together imply

Third, we will use a property of the Hellinger distance between distributions defined on a product space. Suppose D 1 and D 2 are two distributions over a product space X × Y . They can be decomposed into the marginal distribution of x ∈ X and the conditional distribution of y ∈ Y given x , namely, D 1 ( x, y ) = D 1 ,x ( x ) · D 1 ,y | x ( y | x ) and D 2 ( x, y ) = D 2 ,x ( x ) · D 2 ,y | x ( y | x ) . Then, the squared Hellinger distance between D 1 and D 2 satisfies the following:

Lemma A.3. d 2 H ( D 1 , D 2 ) ≤ d 2 H ( D 1 ,x , D 2 ,x ) + max x ∈X d 2 H ( D 1 ,y | x , D 2 ,y | x ) . In particular, if x and y are independent, then d 2 H ( D 1 , D 2 ) ≤ d 2 H ( D 1 ,x , D 2 ,x ) + d 2 H ( D 1 ,y , D 2 ,y ) .

Proof. By definition,

Finally, let D ⊗ T denote the distribution of T i.i.d. samples from D , namely, the product of T independent D 's. We have the following lemma that relates d 2 H ( D ⊗ T 1 , D ⊗ T 2 ) with d 2 H ( D 1 , D 2 ) :

A.2 Distinguishing Distributions

Let D 1 , D 2 be two distributions over a discrete space X . A distribution D i is chosen uniformly at random from { D 1 , D 2 } . Then, we are given T samples from D i and want to guess whether the distribution is D 1 or D 2 . It is known that at least T = Ω ( 1 d 2 H ( D 1 ,D 2 ) log 1 δ ) samples are needed to guess correctly with probability at least 1 -δ , no matter how we guess. Formally:

/negationslash

Lemma A.5 (e.g., [32]) . Let j ∈ { 1 , 2 } be the index of the distribution we guess based on the samples. The probability of making a mistake when distinguishing D 1 and D 2 using T samples, namely Pr[ j = i ] = 1 2 Pr[ j = i | i = 1] + 1 2 Pr[ j = i | i = 2] , is at least:

/negationslash

/negationslash

/negationslash

/negationslash

The second item implies that, in order to achieve Pr[ j = i ] ≤ δ , we must have T ≥ 1 4 d 2 H ( D 1 ,D 2 ) log 1 4 δ .

We provide a proof of this lemma for completeness:

Proof. Let D ⊗ T 1 and D ⊗ T 2 denote the distributions of T i.i.d. samples from D 1 and D 2 , respectively. The draw of T samples from D 1 or D 2 is equivalent to the draw of one sample from D ⊗ T 1 or D ⊗ T 2 . Given one sample from D ⊗ T 1 or D ⊗ T 2 , the probability of making a mistake when guessing the distribution is at least:

/negationslash

We then upper bound d TV ( D ⊗ T 1 , D ⊗ T 2 ) in two ways, which will prove the two items of the lemma, respectively. First, according to first item of Fact A.1, we have

The above two inequalities give d TV ( D ⊗ T 1 , D ⊗ T 2 ) ≤ √ 2 Td H ( D 1 , D 2 ) . This proves the first item of the lemma.

Second, according to the second item of Fact A.1 and Lemma A.4, we have

Since 1 -d 2 TV ( D ⊗ T 1 , D ⊗ T 2 ) = ( 1 + d TV ( D ⊗ T 1 , D ⊗ T 2 ) )( 1 -d TV ( D ⊗ T 1 , D ⊗ T 2 ) ) ≤ 2 ( 1 -d TV ( D ⊗ T 1 , D ⊗ T 2 ) ) , we have

/negationslash

Plugging into (12), we get

/negationslash

When d 2 H ( D 1 , D 2 ) < 1 2 , we use the inequality 1 -x ≥ e -2 x for 0 < x < 1 2 to conclude that

/negationslash

B Special Cases: Strongly and Weakly Informative Experts

In this section we investigate two special families of conditionally independent distributions that admit O ( 1 ε log 1 δ ) sample complexity, which is smaller than the O ( 1 ε 2 log 1 εδ ) bound for general conditionally independent distributions. In these two cases, the (signals of) experts are either 'very informative' or 'very non-informative'.

Definition B.1. Let γ ∈ [0 , ∞ ] be a parameter. For an expert i , we say its signal s i ∈ S i is

γ -weakly informative if 1 1+ γ ≤ P ( s i | ω =1) P ( s i | ω =0) ≤ 1 + γ .

γ -strongly informative if either P ( s i | ω =1) P ( s i | ω =0) ≥ 1 + γ or P ( s i | ω =1) P ( s i | ω =0) ≤ 1 1+ γ holds.

An expert i is γ -strongly (or γ -weakly) informative if all of its signals in S i are γ -strongly (or γ -weakly) informative. 7

A signal s i being γ -strongly (or γ -weakly) informative implies that its corresponding report r i will be ' γ -away from' (or ' γ -close to') the prior p = P ( ω = 1) , in the ' r i 1 -r i and p 1 -p ' ratio form. Specifically, if s i is γ -strongly informative, then from Equation (1) we have

As γ gets larger, a γ -strongly informative signal (expert) is more informative for predicting whether ω = 1 or 0 . This would make aggregation easier. If s i is γ -weakly informative, then:

As γ gets smaller, a γ -weakly informative signal (expert) is less informative for predicting ω , but in this case their report r i will be close to the prior p , which allows us to estimate the ρ n -1 term in the optimal aggregator f ∗ ( r ) = 1 1+ ρ n -1 ∏ n i =1 1 -r i r i better. Those are some intuitions why both strongly and weakly informative signals can reduce the sample complexity of forecast aggregation.

Formally, for γ -strongly informative experts with not-too-small γ , we have the following result:

Theorem B.2. If n ≥ 32 log 2 ε and all experts are γ -strongly informative with γ 1+ γ ≥ 8 √ 2 n log 2 ε , then the sample complexity of forecast aggregation is ≤ O ( 1 εn ( γ 1+ γ ) 2 log 1 δ + 1 ε log 1 δ ) = O ( 1 ε log 1 δ ) .

We remark that the conditions of the theorem, n ≥ 32 log 2 ε and γ 1+ γ ≥ 8 √ 2 n log 2 ε , are easier to be satisfied when the number of experts n increases, if the informativeness parameter γ of each expert is a constant or does not decrease with n (which we believe is a reasonable assumption given that experts are independent of each other). Also, if γ is fixed or increasing, then as n increases the sample complexity decreases.

The proof of Theorem B.2 is in Appendix G.1. Roughly speaking, we divide each expert's signal set into two sets, S 1 i and S 0 i : signals that are more likely to occur under ω = 1 (i.e., P ( s i | ω =1) P ( s i | ω =0) ≥ 1 + γ ) and under ω = 0 (i.e., P ( s i | ω =1) P ( s i | ω =0) ≤ 1 1+ γ ). If the realized ω is 1 , then one may expect to see Ω ( γ 1+ γ n ) more S 1 i signals than the S 0 i signals from the n experts, because the probabilities of these two types of signals differ by Ω( γ 1+ γ ) for each expert. If ω is 0 then one may expect to see more S 0 i signals than the S 1 i signals. So, by checking which type of signals are more we can tell whether ω = 0 or

1 . To tell whether a signal belongs to S 1 i or S 0 i , we compare the corresponding report r i 1 -r i with ρ (namely, r i 1 -r i ≥ (1 + γ ) ρ ⇐⇒ s i ∈ S 1 i and r i 1 -r i ≤ 1 1+ γ ρ ⇐⇒ s i ∈ S 0 i ), where ρ is estimated with accuracy γ 1+ γ using O ( 1 ρn ( γ 1+ γ ) 2 log 1 δ ) samples. Dealing with the case of ρ < ε separately, we obtain the bound O ( 1 εn ( γ 1+ γ ) 2 log 1 δ + 1 ε log 1 δ ) .

For γ -weakly informative experts with small γ , we have:

Theorem B.3. If all experts are γ -weakly informative with γ ≤ 1 , then the sample complexity of forecast aggregation is ≤ min { O ( γn ε log 1 δ ) , O ( 1 ε 2 log 1 εδ ) } , which is O ( 1 ε log 1 δ ) if γ = O ( 1 n ) .

The O ( 1 ε 2 log 1 εδ ) term in the sample complexity follows from the result for general conditionally independent distributions (Theorem 5.2). The O ( γn ε log 1 δ ) term is proved in Appendix G.2. We give the rough idea here using γ = 1 n as an example. The proof relies on the observation that E [ ∏ n i =1 r i 1 -r i | ω = 0 ] = ρ n . Since experts are weakly informative, each of their reports r i 1 -r i is around ρ in the range [ 1 1+ γ ρ, (1 + γ ) ρ ] ⊆ [ 1 1+ 1 n ρ, (1 + 1 n ) ρ ] . Taking the product, we have ∏ n i =1 r i 1 -r i ∈ [ ρ n e , eρ n ] , which is in a bounded range. This allows us to use Chernoff bound to argue that the ρ n (or ρ n -1 ) term in the optimal aggregator f ∗ ( r ) = 1 1+ ρ n -1 ∏ n i =1 1 -r i r i can be estimated, with O ( √ ε ) accuracy, using only O ( e ( √ ε ) 2 log 1 δ ) = O ( 1 ε log 1 δ ) samples of ∏ n i =1 r i 1 -r i . The aggregator using the estimate ˆ ρ n -1 , ˆ f ( r ) = 1 1+ˆ ρ n -1 ∏ n i =1 1 -r i r i , turns out to be O ( ε ) -optimal.

C Extension: Multi-Outcome Events

In this section we generalize our model from binary events to multi-outcome events. The event space now becomes Ω = { 1 , 2 , . . . , | Ω |} with | Ω | > 2 . The joint distribution of event ω ∈ Ω and experts' signals s = ( s i ) n i =1 ∈ S is still denoted by P , which belongs to some class of distributions P . The size of each expert's signal space is still assumed to be |S i | = m < + ∞ for general distributions and can be infinite for conditionally independent distributions (where s 1 , . . . , s n are independent conditioned on ω ). After observing signal s i , expert i reports its posterior belief of the event given s i , which is r i = ( r ij ) j ∈ Ω where r ij = P ( ω = j | s i ) . An aggregator now is a vector-valued function f = ( f j ) j ∈ Ω that maps the joint report r = ( r i ) n i =1 = ( r ij ) ij to a probability distribution f ( r ) over Ω , where f j ( r ) is the aggregated predicted probability for ω = j . We assume f j ( r ) ≥ 0 and ∑ j ∈ Ω f j ( r ) = 1 . The definition of the (expected) loss of an aggregator f becomes:

It is easy to see that the optimal aggregator f ∗ = argmin f L P ( f ) in the multi-outcome case is still the Bayesian aggregator (this is a generalization of Lemma 2.1):

and the difference between the losses of f and f ∗ satisfies

Using T = T P ( ε, δ ) samples { ( r ( t ) , ω ( t ) ) } T t =1 from P , the principal wants to find an ε -optimal aggregator ˆ f with probability at least 1 -δ , for any distribution P in a class P . We give lower bounds and upper bounds on the sample complexity:

So, an ε -optimal aggregator f is an aggregator that satisfies E P [ ∑ j ∈ Ω | f j ( r ) -f ∗ j ( r ) | 2 ] ≤ ε .

Theorem C.1. The sample complexity of forecast aggregation for multi-outcome events is:

For the class P of general distributions: Ω ( m n -2 +log(1 /δ ) ε ) ≤ T P ( ε, δ ) ≤ O ( | Ω | m n +log(1 /δ ) ε 2 ) .

For the class P ind of conditionally independent distributions: Ω ( 1 ε log 1 δ ) ≤ T P ind ( ε, δ ) ≤ O ( | Ω | log | Ω | ε 2 log 1 ε + 1 ε 2 log 1 δ ) .

C.1 Proof of Theorem C.1

Before proving the theorem, we note that the loss of any aggregator f satisfying f j ( r ) ≥ 0 and ∑ j ∈ Ω f j ( r ) = 1 is bounded by [0 , 2] :

Similarly,

C.1.1 Lower Bounds

The lower bounds directly follow from the lower bounds for the binary case (Theorem 4.1 and Theorem 5.2) because the binary case is a special case of the multi-outcome case. Specifically, we can regard any binary event distribution P as a multi-outcome distribution that puts probability only on outcomes { 1 , 2 } ⊆ Ω . If we can learn an ε -optimal aggregator ˆ f for the multi-outcome case: E [ ∑ j ∈ Ω | ˆ f j ( r ) -f ∗ j ( r ) | 2 ] ≤ ε , then this aggregator satisfies

So, the aggregator ˆ f 1 ( · ) is an ε 2 -optimal aggregator for the binary case.

C.1.2 Upper Bounds

General distributions: the O ( | Ω | m n +log(1 /δ ) ε 2 ) upper bound. The proof for general distributions in the multi-outcome case is the same as the proof for general distributions in the binary case (Section 4.1), except for three differences: (1) P (regarded as a distribution over reports r and event ω ) now is a discrete distribution with support size at most | Ω | m n ; (2) the loss | f ( r ) -ω | 2 ∈ [0 , 1] in the binary case is replaced by the loss ∑ j ∈ Ω | f j ( r ) -/BD [ ω = j ] | 2 ∈ [0 , 2] in the multi-outcome case; (3) the ε in the binary case is replaced by ε 2 because the loss is now upper bounded by 2 . Thus, the bound O ( m n +log(1 /δ ) ε 2 ) in the binary case becomes O ( | Ω | m n +log(1 /δ ) ( ε 2 ) 2 ) = O ( | Ω | m n +log(1 /δ ) ε 2 ) in the multi-outcome case.

Conditionally independent distributions: the O ( | Ω | log | Ω | ε 2 log 1 ε + 1 ε 2 log 1 δ ) upper bound. The sample complexity upper bound for conditionally independent distributions is proved by a pseudodimension argument. We first show in Lemma C.2 that the optimal aggregator f ∗ belongs to some parametric family of aggregators. Then, we upper bound the pseudo-dimension of the loss functions associated with this parametric family of aggregators, which will give the desired sample complexity upper bound.

Lemma C.2. For multi-outcome conditionally independent distribution P , given signals s = ( s i ) n i =1 , with corresponding reports r = ( r i ) n i =1 = ( r ij ) ij where r ij = P ( ω = j | s i ) , the posterior probability of ω = j satisfies:

Proof. By Bayes' rule and the fact that s 1 , . . . , s n are independent conditioning on ω ,

Since the above expression depends only on r i 's but not on s i 's, we have:

(For the special binary event case,

According to Lemma C.2, the optimal aggregator f ∗ = ( f j ) j ∈ Ω belongs to the following family of aggregators, parameterized by θ = ( θ j ) j ∈ Ω ∈ R | Ω | + :

∣ (The optimal aggregator f ∗ has parameter θ j = 1 P ( ω = j ) n -1 .) Let G be the family of loss functions associated with F , which is also parameterized by θ ∈ R | Ω | + ,

We recall the definition of pseudo-dimension of a family of functions:

Definition C.3 (e.g., [1]) . Let H be a family of functions from input space X to real numbers R . Let x (1) , . . . , x ( d ) ∈ X be d inputs. Let t (1) , . . . , t ( d ) ∈ R be d 'thresholds'. Let b = ( b (1) , . . . , b ( d ) ) ∈ {-1 , +1 } d be a vector of labels. We say b can be generated by H (on inputs x (1) , . . . , x ( d ) with thresholds t (1) , . . . , t ( d ) ) if there exists a function h b ∈ H such that h b ( x ( i ) ) > t ( i ) if b ( i ) = 1 and h b ( x ( i ) ) < t ( i ) if b ( i ) = -1 (namely, sign( h b ( x ( i ) ) -t ( i ) ) = b ( i ) ) for every i ∈ { 1 , . . . , d } . The pseudo-dimension of H , Pdim( H ) , is the largest integer d for which there exist d inputs and d thresholds such that all the 2 d label vectors in {-1 , +1 } d can be generated by H .

Pseudo-dimension gives a sample complexity upper bound for the uniform convergence of the empirical means of a family of functions to their true means:

Theorem C.4 (e.g., [1]) . Let H be a family of functions from X to [0 , U ] with Pdim( H ) = d . For any distribution D on X , with probability at least 1 -δ over the random draw of samples x (1) , . . . , x ( T ) from D , we have: for every h ∈ H , ∣ ∣ E x ∼D [ h ( x )] -1 T ∑ T t =1 h ( x ( t ) ) ∣ ∣ ≤ ε . We now upper bound the pseudo-dimension of G :

Lemma C.5. Pdim( G ) ≤ O ( | Ω | log | Ω | ) .

Consider the /lscript -th input ( r ( /lscript ) , ω ( /lscript ) ) and threshold t ( /lscript ) . To simplify notations we omit superscript ( /lscript ) , so we have ( r , ω ) and t . We write x = ( x j ) j ∈ Ω where x j = ∏ n i =1 r ij . For any function g θ ∈ G , we have

Proof. Suppose Pdim( G ) = d . By definition, there exist d inputs ( r (1) , ω (1) ) , . . . , ( r ( d ) , ω ( d ) ) and d thresholds t (1) , . . . , t ( d ) ∈ R such that all the 2 d label vectors b ∈ {-1 , +1 } d can be generated by some function g θ b ∈ G . We will count how many label vectors can actually be generated by G .

.

By definition, the set of parameters that give input ( r , ω ) label ' +1 ' is { θ ∈ R | Ω | + | g θ ( r , ω ) > t } . By the above equation, this set is equal to the set

We note that the above set is the solution set to the quadratic form inequality θ /latticetop A θ > 0 for some matrix A ∈ R | Ω |×| Ω | . Similarly, the set of parameters that give input ( r , ω ) label ' -1 ' is the solution set to θ /latticetop A θ < 0 . These two sets share a boundary: the solution set to θ /latticetop A θ = 0 , which is a hyper-ellipsoid in R | Ω | . In other words, the input ( r , ω ) and threshold t define a hyper-ellipsoid which divides the parameter space R | Ω | + into two regions such that all the parameters in one region generate the same label for that input.

Enumerating all inputs ( r (1) , ω (1) ) , . . . , ( r ( d ) , ω ( d ) ) . They define d hyper-ellipsoids in total, dividing the parameter space R | Ω | + into several regions. Within each region, all the parameters generate the same label for each input and hence generate the same label vector. So, the number of label vectors that can be generated by all the parameters in R | Ω | + is upper bounded by the number of regions. The number of regions divided by d hyper-ellipsoids in R | Ω | + is in the order of O ( d | Ω | ) . Hence, to generate all the 2 d label vectors we need O ( d | Ω | ) ≥ 2 d . This gives d ≤ O ( | Ω | log | Ω | ) .

By Theorem C.4 and Lemma C.5, plugging in d = Pdim( G ) ≤ O ( | Ω | log | Ω | ) and U = 2 (because g θ ( r , ω ) is bounded by [0 , 2] according to (18)), we obtain the following: with probability at least 1 -δ over the random draw of samples ( r (1) , ω (1) ) , . . . , ( r ( T ) , ω ( T ) ) from P , we have for any g θ ∈ G ,

This is equivalent to for any f θ ∈ F ,

∣ ∣ where ˆ P emp is the empirical distribution Uniform { ( r (1) , ω (1) ) , . . . , ( r ( T ) , ω ( T ) ) } . By the same logic as the proof of the upper bound in Theorem 4.1 (Section 4.1), the empirically optimal aggregator ˆ f ∗ = argmin f θ ∈F L ˆ P emp ( f θ ) is 2 ε -optimal.

D Missing Proofs from Section 2

D.1 Proof of Lemma 2.1

To prove the first item f ∗ ( r ) = P ( ω = 1 | r ) , we simply note that should minimize E [ | f ( r ) -ω | 2 | r ] for almost every r . This gives f ∗ ( r ) = E [ ω | r ] = P ( ω = 1 | r ) .

To prove the second item L P ( f ) -L P ( f ∗ ) = E P [ | f ( r ) -f ∗ ( r ) | 2 ] , we note that, by the definition of L P ( · ) and the fact that f ∗ ( r ) = E [ ω | r ] proven above,

E Missing Proofs from Section 4

E.1 Proof of Lemma 4.3

Recall that we have a family D of distributions over S = S 1 ×···×S n satisfying the three properties in Definition 4.2 ( B -uniformly bounded, same marginal across distributions, and distinct marginals across signals). We have constructed from D the family of distributions P = { P D : D ∈ D} for the forecast aggregation problem as follows:

First, we have the following observations about D and P :

 We want to show that ε -optimal aggregation with respect to P will imply (1+ B ) 2 √ ε total variation distance learning with respect to D , and hence T P ( ε, δ ) ≥ T TV D ( (1 + B ) 2 √ ε, δ ) .

Fact E.1. For a family of distributions D that satisfies the three properties in Definition 4.2:

Given signal s i , expert i 's report is r i = P D ( ω = 1 | s i ) = D ( s i ) 1 m + D ( s i ) , which is the same for all distributions .

/negationslash

For every expert i , given different signals s i = s ′ i , its reports r i = r ′ i . So, there is a one-toone mapping between s i and r i for every i ∈ { 1 , . . . , n } and also a one-to-one mapping between the joint signal s = ( s 1 , . . . , s n ) and the joint report r = ( r 1 , . . . , r n ) .

/negationslash

For any joint signal s , with corresponding joint report r , we have D ( s ) = f ∗ ( r ) m n (1 -f ∗ ( r )) .

Proof. We prove the three items one by one:

By definition, the marginal distribution of joint signal, P D ( s ) , is

D ∈ D

Fixing s i , summing over s -i = ( s 1 , . . . , s i -1 , s i +1 , . . . , s n ) ∈ S -i , we get

So, given signal s i , expert i reports

and this is the same for all D ∈ D since D ( s i ) = D ′ ( s i ) by the 'same marginal across distributions' property.

/negationslash

Given s i = s ′ i , by the 'distinct marginals across signals' property, we have D ( s i ) = D ( s ′ i ) . Since r i = D ( s i ) 1 m + D ( s i ) and x 1 m + x is a strictly increasing function of x , it follows that r i = r ′ i .

/negationslash

/negationslash

According to item 2, there is a one-to-one mapping between s = ( s 1 , . . . , s n ) and r = ( r 1 , . . . , r n ) ; in other words, observing signals s 1 , . . . , s n is equivalent to observing reports r 1 , . . . , r n . Therefore, by Bayes' rule we have

Rearranging, we obtain D ( s ) = f ∗ ( r ) m n (1 -f ∗ ( r )) .

Claim E.2. If we have an aggregator ˆ f that is ε -optimal with respect to P D , then we can find a distribution ˆ D such that d TV ( ˆ D,D ) ≤ (1 + B ) 2 √ ε .

Proof. Because D is B -uniformly bounded, from (25) we can verify that f ∗ ( r ) satisfies f ∗ ( r ) ≤ B 1+ B . So, we can assume ˆ f ( r ) ≤ B 1+ B as well (if ˆ f ( r ) > B 1+ B , we can let ˆ f ( r ) be B 1+ B ; this only reduces the approximation error E [ | ˆ f ( r ) -f ∗ ( r ) | 2 ] ). Define ˆ D by letting ˆ D ( s ) = ˆ f ( r ) m n (1 -ˆ f ( r )) , ∀ s ∈ S , where r is the reports corresponding to s (cf., Fact E.1). Then, d TV ( ˆ D,D ) is

Because ˆ f ( r ) , f ∗ ( r ) ≤ B 1+ B and the function x 1 -x has derivative 1 (1 -x ) 2 ≤ (1+ B ) 2 when x ≤ B 1+ B , we have

By Jensen's inequality ( E [ X ]) 2 ≤ E [ X 2 ] and by the assumption that ˆ f is ε -optimal, we have ( E P D [ | ˆ f ( r ) -f ∗ ( r ) | ]) 2 ≤ E P D [ | ˆ f ( r ) -f ∗ ( r ) | 2 ] ≤ ε . Thus, which proves the claim.

Now, we present the reduction from learning D in total variation distance to forecast aggregation for P . We use notations x (1) , . . . , x ( T ) ∈ S to represent the samples from D . From x (1) , . . . , x ( T ) we construct the samples ( r (1) , ω (1) ) , . . . , ( r ( T ) , ω ( T ) ) for the forecast aggregation problem. After obtaining a solution ˆ f to the latter problem, we convert it into a solution ˆ D to the former. Details are as follows:

Input: T i.i.d. samples x (1) , . . . , x ( T ) from an unknown distribution D ∈ D .

Reduction:

Draw T samples ω (1) , . . . , ω ( T ) ∼ Uniform { 0 , 1 } .

For each t = 1 , . . . , T , do the following:

If ω ( t ) = 0 , draw s ( t ) ∼ Uniform( S ) .

If ω ( t ) = 1 , let s ( t ) = x ( t ) .

For each i , compute r ( t ) i = D ( s ( t ) i ) 1 m + D ( s ( t ) i ) . Let r ( t ) = ( r ( t ) 1 , . . . , r ( t ) n ) .

Feed S T = { ( r (1) , ω (1) ) , . . . , ( r ( T ) , ω ( T ) ) } to the forecast aggregation problem. Obtain solution ˆ f .

Convert ˆ f into ˆ D according to Claim E.2.

Output: ˆ D .

We remark that, in the second step of the reduction, the report r ( t ) i can be computed even if D is unknown, because the D ( s ( t ) i ) is the same for all D ∈ D and hence known.

Using the above reduction, we show that the sample complexity of ε -optimal forecast aggregation for P cannot be smaller than the sample complexity of learning D within total variation distance (1 + B ) 2 √ ε , which will prove Lemma 4.3:

Proof of Lemma 4.3: First, we verify that the distribution of samples S T in the above reduction is exactly the distribution of T samples { ( r (1) , ω (1) ) , . . . , ( r ( T ) , ω ( T ) ) } from P D . This is because: (1) the distribution of ω ( t ) is Uniform { 0 , 1 } , as defined in P D ; (2) given ω ( t ) = 0 , the distribution of s ( t ) is Uniform( S ) , as defined in P D ; (3) given ω ( t ) = 1 , the distribution of s ( t ) is the same as the distribution of x ( t ) , which is D , because the random draws of ω ( t ) and x ( t ) are independent; (4) according to Fact E.1, the report r ( t ) i = D ( s ( t ) i ) 1 m + D ( s ( t ) i ) = P D ( ω = 1 | s ( t ) i ) , as desired.

Then, by the definition of sample complexity of forecast aggregation, if we are given T = T P ( ε, δ ) samples S T for the forecast aggregation problem, then with probability at least 1 -δ we should be able to find an ε -optimal aggregator ˆ f with respect to P D . According to Claim E.2, we can convert ˆ f into a ˆ D such that

By the definition of sample complexity T TV D ( · , δ ) of distribution learning, T must be at least which proves the lemma.

E.2 Proof of Proposition 4.4

To prove Proposition 4.4 we will construct a family of distributions D that satisfies the three properties in Definition 4.2 and requires T TV D ( ε TV , δ ) = Ω ( m n -2 +log(1 /δ ) ε 2 TV ) samples to learn. For simplicity, we write ε = ε TV . For technical convenience, we assume ε < 1 40 , δ < 0 . 01 .

E.2.1 Part 1: Constructing D

We index the distributions D z ∈ D by z ; the meaning of z will be defined later. Without loss of generality, we assume |S i | = m to be an even integer, and denote S i = { 1 , . . . , m } =: S . We will define D z to be a distribution over the joint signal space S = S 1 ×··· × S n = S n = { 1 , . . . , m } n . In the following we will call a joint signal s ∈ S n simply a signal. We write a signal s ∈ S n as s = ( b , x, y ) , where b ∈ S n -2 and x, y ∈ S . Wesort all the m n signals in S n by the lexicographical

order, from (1 , . . . , 1 , 1) , (1 , . . . , 1 , 2) , ..., to ( m,... , m, m ) . We number the signals from 1 to m n , using to denote their numbers. We divide the whole signal space S n into | S n -2 | = m n -2 'buckets', each of size m 2 and denoted by B b :

We first define a 'base' distribution D base , then construct the distributions D z 's by modifying the probabilities of the base distribution within each bucket B b . Let γ = 1 + 1 m n . The base distribution is defined as follows:

Because 1 ≤ γ /lscript ≤ γ m n ≤ (1 + 1 m n ) m n ≤ e , we have

We assign a sign z b ∈ { +1 , -1 } to each bucket B b , and let z be a vector of length m n -2 that includes the signs of all buckets:

We have 2 m n -2 different z 's in total, and hence 2 m n -2 different distributions D z 's in D in total. Let c = 20 , so that cε ≤ 1 / 2 . For each z , we define D z as follows: within each bucket B b , for each element ( b , x, y ) ∈ B b let

Claim E.3. The family of distributions D = { D z } z defined above satisfies the three properties in Definition 4.2: B -uniformly bounded with B = e +1 / 2 , same marginal across distributions, distinct marginals across signals.

Proof. B -uniformly bounded: For any s , any D z , by definition,

So, the distribution is B -uniformly bounded with B = e +1 / 2 .

Same marginal across distributions: Consider each D z ( s i ) . We want to show that D z ( s i ) does not depend on z , and in fact, D z ( s i ) = D base ( s i ) . If i ∈ { 1 , . . . , n -2 } , namely, s i is a component of the vector b , then we have

We notice that, fixing any b , the numbers of + z b cε W and -z b cε W in the summation ∑ m x =1 ∑ m y =1 D z ( b , x, y ) are the same. So, they cancel out, and we obtain

If i = n -1 , namely s i = x , then we have:

Fixing any b , the numbers of + z b cε W and -z b cε W in the summation ∑ m y =1 D z ( b , x, y ) are the same. So, they cancel out, and we obtain

Finally, if i = n , namely s i = y , then by a similar argument as above we have

/negationslash

Distinct marginals across signals: By the 'same marginal across distributions' property above we have D z ( s i ) = D base ( s i ) . So, to prove 'distinct marginals across signals' we only need to prove D base ( s i ) = D base ( s ′ i ) for s i = s ′ i . Without loss of generality assume s i < s ′ i . By the definition D base ( s ) = γ num( s ) W and the fact that num( s i , s -i ) < num( s ′ i , s -i ) for any s -i ∈ S -i , we have

/negationslash

base i base i

/negationslash

E.2.2 Part 2: Sample Complexity Lower Bound of Learning D

Overview We want to prove the proposition that the sample complexity of learning the family of distributions D = { D z } z defined above is at least T TV D ( ε, δ ) = Ω ( m n -2 +log(1 /δ ) ε 2 ) . This proof is analogous to a textbook proof of Proposition 3.3 (the sample complexity for learning all distributions), which uses reductions from the distinguishing distributions problem. Roughly speaking, if one can learn the unknown distribution D z well then one must be able to guess most of the components of the sign vector z = ( z b ) b ∈ S n -2 correctly, meaning that one can distinguish whether the distribution D z b on bucket B b is D z b =+1 or D z b = -1 . However, since D z b =+1 and D z b = -1 are ' O ( ε )-close' to each other, distinguishing them requires Ω( 1 ε 2 ) samples. In average, there are only O ( T m n -2 ) samples falling into a bucket (because there are m n -2 buckets in total and the distribution D z is close to uniform). We thus need O ( T m n -2 ) = Ω( 1 ε 2 ) , which gives T = Ω( m n -2 ε 2 ) . Ignoring logarithmic terms, this proves the proposition.

Formal argument First, we note that if we can learn D z very well, then we can guess the vector z correctly for a large fraction of its m n -2 components. Formally, suppose we obtain from T samples a distribution ˆ D such that d TV ( ˆ D,D z ) ≤ ε . We find the distribution D w in D , w = ( w b ) b ∈ S n -2 ∈ { +1 , -1 } m n -2 , that is closest to ˆ D in total variation distance. By definition, we have d TV ( D w , ˆ D ) ≤ d TV ( D z , ˆ D ) ≤ ε . Hence, by triangle inequality,

/negationslash

TV w z ≤ TV w TV z ≤ ε. Let # = ∣ ∣ { b ∈ S n -2 | w b = z b } ∣ ∣ be the number of different components of w and z . We claim that:

/negationslash

Proof. Whenever we have a different component w b = z b , this different component contributes the following to the total variation distance between D w and D z :

So, the number of different components of w and z is at most d TV ( D w ,D z ) m 2 cε W ≤ 2 W cm 2 .

We first show the Ω( log(1 /δ ) ε 2 ) part in the sample complexity lower bound, and then show the Ω( m n -2 ε 2 ) part. Together, they give the lower bound max { Ω( log(1 /δ ) ε 2 ) , Ω( m n -2 ε 2 ) } = Ω( m n -2 +log(1 /δ ) ε 2 ) . Consider the distribution D +1 ∈ D whose index is the all ' +1 ' vector and the distribution D -1 ∈ D whose index is the all ' -1 ' vector. According to (28), the total variation distance between D +1 and D -1 is d TV ( D +1 , D -1 ) = m n -2 · m 2 cε W = m n cε W because +1 and -1 have m n -2 different components. Since W ≤ em n (26), we have d TV ( D +1 , D -1 ) ≥ cε e > 2 ε . Consider the distinguishing distributions problem (defined in Section 3) where we want to distinguish D +1 and D -1 . If we can learn from samples a distribution ˆ D that is ε -close in total variation distance to the unknown distribution D +1 or D -1 , then we can perfectly tell whether the unknown distribution is D +1 or D -1 because the two distributions are more than 2 ε -away from each other in total variation distance. Lemma A.5 implies that, to distinguish D +1 and D -1 with probability 1 -δ , the number of samples must be at least Ω( log(1 /δ ) d 2 H ( D +1 ,D -1 ) ) = Ω( log(1 /δ ) ε 2 ) . This proves the Ω( log(1 /δ ) ε 2 ) part.

We then prove the Ω( m n -2 ε 2 ) part. Suppose we first draw the vector z from { +1 , -1 } m n -2 uniformly at random, then draw T samples from D z . We obtain the D w as above. Let's consider the expected number of different components of w and z in this two-step random draw procedure:

/negationslash

/negationslash

We consider each component E [ /BD { z b = w b } ] in the above summation. Suppose that, within the T samples drawn from D z , T b of them fall into the bucket B b . So, T b follows the Binomial ( T, D ( B b )) distribution with

/negationslash

(Notice that the + z b cε W and -z b cε W cancel out in the summation and hence D ( B b ) does not depend on z .) Let D z b denote the ' B b -part' of distribution D z , namely, D z conditioning on B b :

We think of the random draw of the vector z and the T samples as follows: first, we draw T b from Binomial ( T, D ( B b )) ; second, we draw z b ∈ { +1 , -1 } uniformly at random; third, we draw T b samples from the conditional distribution D z b ; forth, we draw the remaining vector z -b and the remaining T -T b samples (which are samples outside of B b ). Only writing the first two steps explicitly, we have

/negationslash

/negationslash

/negationslash

/negationslash

Claim E.5. For any T b , 1 2 Pr [ z b = w b | z b = +1 , T b ] + 1 2 Pr [ z b = w b | z b = -1 , T b ] ≥ 1 2 -2 cε √ T b .

/negationslash

/negationslash

/negationslash

/negationslash

Proof. We notice that 1 2 Pr [ z b = w b | z b = +1 , T b ] + 1 2 Pr [ z b = w b | z b = -1 , T b ] is the probability that we make a mistake when guessing the sign z b using w b , if (1) z b is chosen from {-1 , +1 } uniformly at random; (2) we are given T b samples from D z b ; (3) we then draw the remaining vector z -b and the remaining samples; (4) finally, we use the D w computed from all samples to get w b . The steps (3) and (4) define a randomized function that maps the T b samples of D z b to w b ∈ { 0 , 1 } , and therefore, according to the first item of Lemma A.5, we have

/negationslash

/negationslash

/negationslash

/negationslash

Then we consider d H ( D z b =+1 , D z b = -1 ) . We use Lemma A.2 to do so. For any s ∈ B b , we have, on the one hand,

because a -b a + b = 1 -2 b a + b ≥ 1 -2 b for a = γ num( s ) ≥ 1 . On the other hand,

because a + b a -b = 1 + 2 b a -b ≤ 1 + 4 b for a = γ num( s ) ≥ 1 and b = cε ≤ 1 2 . Therefore, by Lemma A.2 we have

Combining (31) and (32) proves our claim.

/negationslash

By (30) and Claim E.5, we have E [ /BD { z b = w b } ] ≥ E T b [ 1 2 -2 cε √ T b ] . Summing over b ∈ S n -2 , (29) becomes

/negationslash

Because E [ T b ] = T D ( B b ) = T W ( b ,x,y ) B b γ num( b ,x,y ) T W m 2 γ m n , we have

/negationslash

/negationslash

Now, let's consider the probability with which we can obtain ˆ D such that d TV ( ˆ D,D z ) ≤ ε . We will show that this probability is at most 0 . 99 < 1 -δ if T is less than 10 -5 · m n -2 ε 2 . Recall from Claim E.4 that d TV ( ˆ D,D z ) ≤ ε implies # = ∑ b ∈ S n -2 /BD { z b = w b } ≤ 2 W cm 2 . So, the probability is at most

/negationslash when c = 20 , T < 10 -5 · m n -2 ε 2 , and δ < 0 . 01 . This means that, in order to obtain such ˆ D with probability at least 1 -δ , at least 10 -5 · m n -2 ε 2 samples are needed.

F Missing Proofs from Section 5

F.1 Proof of Theorem 5.2: the Ω ( 1 ε log 1 δ ) Lower Bound

Let c = 32 . We assume ε < 2 -18 , so that c √ ε < 1 16 . For P 1 , we let

For P 2 , we let

The proof uses a reduction from the distinguishing distributions problem (defined in Section 3). We construct two conditionally independent distributions P 1 , P 2 over the space Ω × S 1 × · · · × S n with each |S i | = 2 , S i = { a, b } . Given T samples from either P 1 or P 2 , we want to tell which distribution the samples are coming from. We will show that, if we can solve the forecast aggregation problem, then we can distinguish the two distributions (with high probability), which requires T = Ω( 1 d 2 H ( P 1 ,P 2 ) log 1 δ ) = Ω( 1 ε log 1 δ ) samples according to Lemma A.5.

We require that, in the forecast aggregation problem under both distributions P 1 and P 2 , whenever expert i sees signal a , b , she reports

respectively. This gives the following conditional probabilities P 1 ( · | ω ) , P 2 ( · | ω ) : [ P 1 ( a | ω = 0) P 1 ( a | ω = 1) ] = p 1 -r b r a -r b [ 1 -r a 1 -p 1 r a p 1 ] = [ 1 -1 8 n + 2 c √ ε n 1+ 1 8 n -2 c √ ε n 1 ] , [ P 1 ( b | ω = 0) P 1 ( b | ω = 1) ] = [ 1 4 n -4 c √ ε n 1+ 1 8 n -2 c √ ε n 0 ] . (34) [ P 2 ( a | ω = 0) P 2 ( a | ω = 1) ] = p 2 -r b r a -r b [ 1 -r a 1 -p 2 r a p 2 ] = [ 1 -1 8 n -2 c √ ε n 1+ 1 8 n + 2 c √ ε n 1 ] , [ P 2 ( b | ω = 0) P 2 ( b | ω = 1) ] = [ 1 4 n + 4 c √ ε n 1+ 1 8 n + 2 c √ ε n 0 ] . (35)

Given T samples from the unknown distribution P ∈ { P 1 , P 2 }, each of which is a vector of ω ( t ) and all experts' signals s ( t ) i ∈ { a, b } , we feed the corresponding reports r ( t ) i ∈ { r a , r b } and ω ( t ) to the forecast aggregation problem and obtain a solution ˆ f , which is an ε -optimal aggregator. We want to use ˆ f to estimate the prior p = P ( ω = 1) ∈ { p 1 , p 2 } so that we can tell apart P 1 and P 2 . Recall from Lemma 5.1 that f ∗ ( r ) = 1 1+ ρ n -1 ∏ n i =1 1 -r i r i , where ρ = p 1 -p . Writing ρ in terms of

f ∗ ( r ) , we have

In particular, when r i = r a = 0 . 5 for all i ∈ { 1 , . . . , n } , we have:

So, we estimate ρ by:

Now, we want to argue that, if ˆ f is ε -optimal, then | ˆ ρ -ρ | is at most O ( √ ε n ) . Consider the function:

whose derivative is

By definition, we have

Proof. For P ∈ { P 1 , P 2 } , its ρ = p 1 -p satisfies

where in the last three transitions we used the inequalities 1 -x 1+ x > 1 -2 x and (1 -x/n ) n ≥ 1 -x for x ∈ (0 , 1) and the fact that c √ ε < 1 16 . So,

which proves the claim.

With Claim F.1, we can without loss of generality assume 1 2 ≤ ˆ f ( r 0 . 5 ) ≤ 2 3 as well (otherwise, we can truncate ˆ f ( r 0 . 5 ) to this range; this only reduces the approximation error E [ | ˆ f ( r ) -f ∗ ( r ) | 2 ] ).

Proof.

Claim F.3. If ˆ f is ε -optimal, then | ˆ f ( r 0 . 5 ) -f ∗ ( r 0 . 5 ) | < 2 √ ε .

Proof. If ˆ f is ε -optimal, i.e., E [ | ˆ f ( r ) -f ∗ ( r ) | 2 ] ≤ ε , then, by Jensen's inequality E [ X 2 ] ≥ E [ X ] 2 , we have

For both P ∈ { P 1 , P 2 } , we have

Plugging P ( r 0 . 5 ) > 1 into (38), we get f ( r 0 . 5 ) f ∗ ( r 0 . 5 ) < 2 ε .

2 | ˆ -| √

From (36), Claim F.1, Claim F.2, and Claim F.3, we get

Since p = ρ 1+ ρ as a function of ρ has a bounded derivative ∂p ∂ρ = 1 (1+ ρ ) 2 ≤ 1 , Equation (39) implies

if we use ˆ p = ˆ ρ 1+ˆ ρ as an estimate of p . This allows us to tell part P 1 and P 2 because the difference between p 1 and p 2 is greater than twice of our estimation error | ˆ p -p | :

Therefore, we can tell part P 1 and P 2 by checking whether p 1 or p 2 is closer to ˆ p .

Finally, we upper bound the squared Hellinger distance between P 1 and P 2 . This will give the sample complexity lower bound we want.

Claim F.4. d 2 H ( P 1 , P 2 ) ≤ O ( c 2 ε ) .

Proof. For the marginal distributions of ω , P 1 ω and P 2 ω , according to Lemma A.2 and the fact that 1 ≥ P 2 ω ( ω ) P 1 ω ( ω ) = 1 -1 8 n -2 c √ ε n 1 -1 8 n + 2 c √ ε n = 1 -4 c √ ε n 1 -1 8 n + 2 c √ ε n = 1 -O ( c √ ε n ) , we have

Given ω = 0 or 1 , we consider the conditional distributions of each s i , P 1 s i | ω and P 2 s i | ω . For s i = a , we have

For s i = b , we have

So, d 2 H ( P 1 s i | ω , P 2 s i | ω ) can be upper bounded as follows:

Since P 1 = P 1 ω · ∏ n i =1 P 1 s i | ω and P 2 = P 2 ω · ∏ n i =1 P 2 s i | ω , we have, by Lemma A.3 and Lemma A.4,

Therefore, according to Lemma A.5, to tell apart P 1 and P 2 with probability at least 1 -δ we need at least samples. This concludes the proof.

G Missing Proofs from Section B

G.1 Proof of Theorem B.2

G.1.1 Additional Notations and Lemmas

We introduce some additional notations and lemmas for the proof. Let µ 0 be the expected average report of all experts conditioning on ω = 0 :

which is equal to the expected prediction of ω given expert i 's signal s i where s i is distributed conditioning on ω = 0 , averaged over all experts. Symmetrically, let

Recall that p = P ( ω = 1) .

Fact G.1. (1 -p ) µ 0 = pµ 1 .

Proof. For each expert i , by the law of total expectation and the fact that r i = P ( ω = 1 | s i ) , we have the following equations:

Averaging over all experts i ∈ { 1 , . . . , n } , we conclude that

Lemma G.2. If (1 -p ) µ 0 = pµ 1 < ε 2 , then the averaging aggregator f avg ( r ) = 1 n ∑ n i =1 r i is ε -optimal.

Proof. If (1 -p ) µ 0 = pµ 1 < ε 2 , then the expected loss of f avg is at most

which implies that f avg is ε -optimal.

The following lemma says that O ( 1 ε log 1 δ ) samples are sufficient to tell whether the mean of a random variable is below ε or above ε 2 :

Lemma G.3. Given T = 40 ε log 2 δ i.i.d. samples X (1) , . . . , X ( T ) of a random variable X ∈ [0 , 1] with unknown mean E [ X ] = µ , with probability at least 1 -δ we can tell whether µ < ε or µ ≥ ε 2 . This can be done by checking whether the empirical mean ˆ µ = 1 T ∑ T t =1 X ( t ) is < 3 4 ε or ≥ 3 4 ε .

Proof. If µ ≥ ε , using the multiplicative version of Chernoff bound we have

Namely, with probability at least 1 -δ , it holds that

If µ < ε , then using the additive version of Chernoff-Hoeffding theorem, we have

where D ( x || y ) = x ln x y +(1 -x ) ln 1 -x 1 -y . Using the inequality D ( x || y ) ≥ ( x -y ) 2 2 y for x ≤ y and D ( x || y ) ≥ ( x -y ) 2 2 x for x ≥ y , we obtain:

By a union bound, with probability at least 1 -δ , we have

Combining the case of µ ≥ ε and µ < ε , we conclude that: with probability at least 1 -δ ,

If ˆ µ < 3 4 ε , then we must have µ < ε .

If ˆ µ ≥ 3 4 ε , then we have µ ≥ ε or ε > µ ≥ ˆ µ -ε 4 ≥ ε 2 . In either case, we have µ ≥ ε 2 .

The last lemma we will use shows how to estimate the unknown value of ρ = p 1 -p = P ( ω =1) P ( ω =0) with accuracy ∆ using T = O ( 1 n ∆ 2 log 1 δ ) samples. Notice that, if one simply uses the empirical value ˆ ρ = ∑ T t =1 /BD { ω ( t ) =1 } ∑ T t =1 /BD { ω ( t ) =0 } to estimate ρ , then by Chernoff bound this needs T = O ( 1 ∆ 2 log 1 δ ) samples, which is larger than what we claim by a factor of n . This sub-optimality is because one only uses the ω ( t ) 's in the samples to estimate ρ , wasting the reports r ( t ) i 's. By using r ( t ) i 's to estimate ρ , we can reduce the number of samples by a factor of n . The basic idea is the following: According to Fact G.1, we have ρ = p 1 -p = µ 0 µ 1 = E [ ∑ n i =1 r i | ω =0] E [ ∑ n i =1 (1 -r i ) | ω =1] . The numerator E [ ∑ n i =1 r i | ω = 0] and the denominator E [ ∑ n i =1 (1 -r i ) | ω = 1] can be estimated from samples of r ( t ) i 's where ω ( t ) = 0 and 1 respectively. The total number of r ( t ) i 's is Tn , because we have n experts per sample. This reduces the needed number of samples by a factor of n . Formally:

Lemma G.4. For conditionally independent distribution P , we can estimate ρ = P ( ω =1) P ( ω =0) with accuracy | ˆ ρ -ρ | ρ ≤ ∆ < 1 (equivalently, ˆ ρ ρ ∈ 1 ± ∆ ) with probability at least 1 -δ using

samples of ( ω ( t ) , r ( t ) ) 's, by letting ˆ ρ = 1 #0 ∑ t : ω ( t ) =0 ∑ n i =1 r ( t ) i 1 #1 ∑ t : ω ( t ) =1 ∑ n i =1 (1 -r ( t ) i ) , where # 0 and # 1 are the numbers of samples with ω ( t ) = 0 and 1 respectively.

Proof. Recall that p = P ( ω = 1) , 1 -p = P ( ω = 0) , and ρ = p 1 -p . According to Fact G.1 (which says (1 -p ) µ 0 = pµ 1 ), we have

Consider the following way of estimating ρ from T samples ( ω ( t ) , r ( t ) ) T t =1 : Let # 0 , # 1 be the numbers of samples where ω ( t ) = 0 , 1 , respectively:

We let

Now, we compare the ˆ ρ in (45) and the ρ in (44): we see that 1 # 0 ∑ t : ω ( t ) =0 ∑ n i =1 r ( t ) i is an (unbiased) estimate of the numerator ∑ n i =1 E [ r i | ω = 0] = nµ 0 and that 1 # 1 ∑ t : ω ( t ) =1 ∑ n i =1 (1 -r ( t ) i ) is an (unbiased) estimate of the denominator ∑ n i =1 E [1 -r i | ω = 1] = nµ 1 . We use Chernoff bounds to argue that the accuracy of the two estimates is within ∆ with high probability if # 0 and # 1 are big enough. Suppose that, when drawing the T samples, we draw all the ω ( t ) 's first (and hence # 0 , # 1 are determined), and then draw all the r ( t ) i 's. After all the ω ( t ) 's are drawn, the r ( t ) i 's become independent, because the signals s ( t ) i 's are conditionally independent given ω ( t ) . Therefore, we can use Chernoff bounds:

Requiring δ ≥ 2 e -#0 nµ 0 ∆ 2 3 and δ ≥ 2 e -#1 nµ 1 ∆ 2 3 , namely,

we have, with probability at least 1 -2 δ , both of the following hold:

Then, we argue that (46) can be satisfied with high probability if T is large enough. This is again done by a Chernoff bound: since E [# j ] = E [ ∑ T t =1 /BD { ω ( t ) = j } ] = T · P ( ω = j ) , for j = 0 , 1 , we have

samples, then we can ensure that with probability at least 1 -2 δ , it holds # 0 ≥ 1 2 T (1 -p ) and # 1 ≥ 1 2 Tp . Then, in order for (46) to be satisfied, we can let

This gives

Both (49) and (50) are satisfied when

To conclude, if we are given T = 6 (1 -p ) µ 0 · n ∆ 2 log 2 δ + 12 min { p, 1 -p } log 2 δ samples, then with probability at least 1 -4 δ , (47) holds, which implies

G.1.2 The Proof

Wewant to show the O ( 1 εn ( γ 1+ γ ) 2 log 1 δ + 1 ε log 1 δ ) sample complexity upper bound for the case where experts have γ -strongly informative signals.

We first use O ( 1 ε log 1 δ ) samples tell whether (1 -p ) µ 0 = pµ 1 < ε 2 or (1 -p ) µ 0 = pµ 1 ≥ ε 4 . We note that

For each expert i , let S 1 i = { s i ∈ S i : P ( s i | ω =1) P ( s i | ω =0) ≥ 1 + γ } be its set of γ -strongly informative signals that are more likely to be realized under ω = 1 than under ω = 0 . Let S 0 i = S i \S 1 i = { s i ∈ S i : P ( s i | ω =1) P ( s i | ω =0) ≤ 1 1+ γ } be the set of signals that are more likely to be realized under ω = 0 . Since r i 1 -r i = P ( s i | ω =1) P ( s i | ω =0) ρ by Equation (1), whenever an expert receives a signal in S 1 i , its report satisfies which is the expectation of the random variable X = /BD { ω = 0 } · 1 n ∑ n i =1 r i . So, according to Lemma G.3, we can tell whether (1 -p ) µ 0 < ε 2 or ≥ ε 4 with probability at least 1 -δ using O ( 1 ε log 1 δ ) samples of X . If (1 -p ) µ 0 = pµ 1 < ε 2 , then according to Lemma G.2, the averaging aggregator f avg ( r ) = 1 n ∑ n i =1 r i is ε -optimal. We hence obtained an ε -optimal aggregator in this case. So, in the following proof, we assume (1 -p ) µ 0 = pµ 1 ≥ ε 4 .

and whenever it receives a signal in S 0 i , its report satisfies

We will use the notation P ( S u i | ω ) = P ( s i ∈ S u i | ω ) = ∑ s i ∈S u i P ( s i | ω ) , for u ∈ { 0 , 1 } . Given a set of n signals s 1 , . . . , s n , one per expert, we let X 1 = ∑ n i =1 /BD { s i ∈ S 1 i } be the total number of signals that belong to the S 1 i sets, and similarly let X 0 = ∑ n i =1 /BD { s i ∈ S 0 i } . We have X 0 + X 1 = n , and by definition,

Claim G.5. At least one of E [ X 1 | ω = 1] and E [ X 0 | ω = 0] is ≥ n 2 .

Proof. Suppose on the contrary both E [ X 1 | ω = 1] and E [ X 0 | ω = 0] are < n 2 . Then, from (54) we have

This implies n = E [ X 0 + X 1 | ω = 1] < n 2 + n 2 = n , a contradiction.

Let u ∈ { 0 , 1 } be an index such that

Claim G.5 guarantees that such a u exists. We construct a 'hypothetical' aggregator f hypo that, having access to ρ and E [ X u | ω ] , predicts whether ω = 0 or 1 by counting the number X u of signals that belong to the S u i sets and comparing it with its expectations under ω = 0 and 1 , E [ X u | ω = 0] and E [ X u | ω = 1] , respectively. Specifically, given reports r = ( r 1 , . . . , r n ) as input, with corresponding unobserved signals s = ( s 1 , . . . , s n ) , f hypo does the following:

If u = 1 , count how many reports r i 's satisfy r i 1 -r i ≥ (1 + γ ) ρ ; If u = 0 , count how many reports r i 's satisfy r i 1 -r i ≤ 1 1+ γ ρ . According to (51) and (52), this number is exactly equal to the number of signals that belong to the S u i sets, X u .

Then, check whether X u is closer (in terms of absolute difference) to E [ X u | ω = u ] or E [ X u | ω = 1 -u ] . If X u is closer to E [ X u | ω = u ] , output f hypo ( r ) = u ; otherwise, output f hypo ( r ) = 1 -u .

We claim that f hypo is ε -optimal.

Claim G.6. Given γ 1+ γ ≥ 8 √ 2 n log 2 ε and E [ X u | ω = u ] ≥ n 4 , f hypo is ε -optimal.

Proof. Given either ω = 0 or 1 , consider the conditional random draw of signals s 1 , . . . , s n . Because X u = ∑ n i =1 /BD { s i ∈ S u i } and the random variables /BD { s i ∈ S u i } , i = 1 , . . . , n , are [0 , 1] -bounded and independent conditioning ω , by Hoeffding's inequality we have

Let

Then with probability at least 1 -2 e -2 a 2 n = 1 -ε , it holds

∣ ∣ X u -E [ X u | ω ] ∣ ∣ < a. (57) Consider the difference between E [ X u | ω = u ] and E [ X u | ω = 1 -u ] . By (53) and (54), we have

By the assumption E [ X u | ω = u ] ≥ n 4 ,

By the assumption γ 1+ γ ≥ 8 √ 2 n log 2 ε , we have γ 1+ γ · n 4 ≥ 8 √ 2 n log 2 ε · n 4 = 4 √ n 2 log 2 ε = 4 a. Therefore

Because we already had ∣ ∣ X u -E [ X u | ω ] ∣ ∣ < a (which happened with probability at least 1 -ε ), if X u turns out to be closer to E [ X u | ω = u ] it must be that ω = u ; if X u turns out to be closer to E [ X u | ω = 1 -u ] it must be that ω = 1 -u . In either case, our output f hypo ( r ) is equal to

ω , having a loss 0 . If ∣ ∣ X u -E [ X u | ω ] ∣ ∣ < a did not happen, our loss is at most 1 . Therefore, the expected loss of our aggregator f hypo is at most

In the remaining proof, we show how to use samples to learn a 'real' aggregator ˆ f that implements the same functionality as the hypothetical aggregator f hypo and hence is ε -optimal. We have two learning tasks: First, we need to estimate ρ , so that we can implement the step (1) of f hypo which tells apart r i 1 -r i ≥ (1 + γ ) ρ and r i 1 -r i ≤ 1 1+ γ ρ . Second, we need to find an index u ∈ { 0 , 1 } such that E [ X u | ω = u ] ≥ n 4 and estimate E [ X u | ω = u ] , so that we can implement the step (2) of f hypo which tells whether X u is closer to E [ X u | ω = u ] or E [ X u | ω = 1 -u ] . We show that these two tasks can be achieved using O ( 1 εn ( γ 1+ γ ) 2 log 1 δ + 1 ε log 1 δ ) samples, with probability at least 1 -O ( δ ) .

Task 1: estimate ρ , using T 1 = O ( 1 εn ( γ 1+ γ ) 2 log 1 δ + 1 ε log 1 δ ) samples. We want to use samples to obtain an estimate ˆ ρ of ρ such that 1 1+ γ ρ < ˆ ρ < (1 + γ ) ρ . So, by checking whether r i 1 -r i > ˆ ρ or r i 1 -r i < ˆ ρ we can tell apart r i 1 -r i ≥ (1 + γ ) ρ and r i 1 -r i ≤ 1 1+ γ ρ . Using Lemma G.4 with ∆ = γ 1+ γ , we obtain a ˆ ρ such that with probability at least 1 -δ using

samples (recall that we have min { p, 1 -p } ≥ (1 -p ) µ 0 = pµ 1 ≥ ε 4 ). The ˆ ρ then satisfies as desired.

Task 2: find u such that E [ X u | ω = u ] ≥ n 4 and estimate E [ X u | ω = u ] , using T 2 = O ( 1 ε log 1 δ ) samples. First, we show how to use T 2 = O ( 1 ε log 1 δ ) samples to estimate both E [ X 0 | ω = 0] and E [ X 1 | ω = 1] with accuracy a = √ n 2 log 2 ε . By the same argument as in the proof of Lemma G.4 (Equations 48 and 49), we know that with probability at least 1 -2 δ over the random draws of samples, the numbers of samples ( ω ( t ) , r ( t ) 1 , . . . , r ( t ) n ) 's where ω ( t ) = 0 and ω ( t ) = 1 , denoted by # 0 and # 1 , must satisfy

We consider the samples where ω ( t ) = 0 . There are # 0 n total number of r ( t ) i 's. Suppose we have accomplished Task 1. Then, for each r ( t ) i , we can tell whether the corresponding signal s ( t ) i belongs to S 0 i by checking whether r ( t ) i 1 -r ( t ) i < ˆ ρ . So, we can exactly compute the total number of such signals in the t -th sample, X 0( t ) = ∑ n i =1 /BD { s ( t ) i ∈ S 0 i } , whose expected value is E [ X 0 | ω = 0] . Because signals are independent given ω ( t ) = 0 , by Hoeffding's inequality we have

Plugging in a = √ n 2 log 2 ε and # 0 ≥ 1 2 (1 -p ) T 2 , we get

Therefore, if we require

then with probability at least 1 -2 δ , both

hold. Namely, 1 # 0 ∑ t : ω ( t ) =0 X 0( t ) and 1 # 1 ∑ t : ω ( t ) =1 X 1( t ) are a -accurate estimates of E [ X 0 | ω = 0] and E [ X 1 | ω = 1] . Equations (59) and (60) together imply that T 2 = O ( 1 min { p, 1 -p } log 2 δ ) ≤ O ( 1 ε log 1 δ ) samples suffice.

Then, we identify an index u ∈ { 0 , 1 } such that E [ X u | ω = u ] ≥ n 4 . By Claim G.5, there exists a v ∈ { 0 , 1 } with E [ X v | ω = v ] ≥ n 2 . Since 1 # v ∑ t : ω ( t ) = v X v ( t ) is an a -accurate estimate of E [ X v | ω = v ] , we must have

So, at least one of u ∈ { 0 , 1 } must satisfy 1 # u ∑ t : ω ( t ) = u X u ( t ) ≥ n 2 -a . By picking any such a u , we are guaranteed that E [ X u | ω = u ] ≥ 1 # u ∑ t : ω ( t ) = u X u ( t ) -a ≥ n 2 -2 a . Given the assumption n ≥ 32 log 2 ε in the statement of the theorem, we have

Finally, as argued above, an a -accurate estimate of E [ X u | ω = u ] is given by 1 # u ∑ t : ω ( t ) = u X u ( t ) .

Hence, E [ X u | ω = u ] ≥ n 2 -2 a ≥ n 2 -2( n 8 ) = n 4 .

Constructing ˆ f . After accomplishing Tasks 1 and 2 using T 1 + T 2 = O ( 1 εn ( γ 1+ γ ) 2 log 1 δ + 1 ε log 1 δ ) samples, we construct a ˆ f that implements the same functionality as f hypo . Let

Claim G.7. E [ X u | ω = u ] -a > M > E [ X u | ω = 1 -u ] + a .

where 1 # u ∑ t : ω ( t ) = u X u ( t ) is our estimate of E [ X u | ω = u ] in Task 2 and a = √ n 2 log 2 ε .

Proof. Because 1 # u ∑ t : ω ( t ) = u X u ( t ) is an a -accurate estimate of E [ X u | ω = u ] , we have

Recall from Equation (58) that E [ X u | ω = 1 -u ] ≤ E [ X u | ω = u ] -4 a . So,

The above two inequalities prove the claim.

Given reports r = ( r 1 , . . . , r n ) as input, we let ˆ f do the following:

If u = 1 , count how many reports r i 's satisfy r i 1 -r i > ˆ ρ ; If u = 0 , count how many reports r i 's satisfy r i 1 -r i < ˆ ρ . Let this number be X ;

Then, check whether X > M or X ≤ M . If X > M , output ˆ f ( r ) = u ; otherwise, output ˆ f ( r ) = 1 -u .

We argue that ˆ f implements the same functionality as f hypo : (1) In Task 1 we got 1 1+ γ ρ < ˆ ρ < (1 + γ ) ρ . So, by checking whether r i 1 -r i > ˆ ρ or < ˆ ρ we can exactly tell whether r i 1 -r i ≥ (1 + γ ) ρ or ≤ 1 1+ γ ρ . Hence, we have X = X u , the number of signals that belong to the S u i sets. (2) Recall from Equation (57) that with probability at least 1 -ε , X is a -close to its expectation E [ X u | ω ] . Then, according to Claim G.7, X > M implies that X is closer to E [ X u | ω = u ] ; X < M implies that X is closer to E [ X u | ω = 1 -u ] . So, ˆ f implements both of the two steps in f hypo . Hence, according to Claim G.6, ˆ f is ε -optimal.

G.2 Proof of Theorem B.3

According to Lemma 5.1, the optimal aggregator is

where ρ = p 1 -p . We claim that an approximately optimal aggregator can be obtained by first estimating ρ from samples and then use the aggregator with the estimate ˆ ρ :

Claim G.8. If | ˆ ρ -ρ | ρ ≤ 2 √ ε n -1 < 1 2 , then the aggregator ˆ f defined above is ε -optimal.

Proof. Consider the function g (ˆ ρ ) = 1 1+ˆ ρ n -1 ∏ n i =1 1 -r i r i (where ˆ ρ is the variable and r i 's are constants). We claim that

To see this, we note that if r i = 0 for some i then g (ˆ ρ ) = 0 and g ′ (ˆ ρ ) = 0 . Otherwise, we let y = ∏ n i =1 1 -r i r i < + ∞ and take the derivative with respect to ˆ ρ ,

By the AM-GM inequality a + b ≥ 2 √ ab , we get

as claimed.

Using (63), we have, for ˆ ρ ≥ ρ 2 ,

So, to obtain ε -approximation E [ | ˆ f ( r ) -f ∗ ( r ) | 2 ] ≤ ε , we can require | ˆ f ( r ) -f ∗ ( r ) | ≤ n -1 2 · | ˆ ρ -ρ | ρ ≤ √ ε . This can be satisfied if the error in estimating ρ is at most | ˆ ρ -ρ | ρ ≤ 2 √ ε n -1 .

We then show how to use O ( γn ε log 1 δ ) samples to estimate the value of ρ with O ( √ ε n -1 ) accuracy, which will give us an ε -optimal aggregator according to Claim G.8. Recall from (14) that when signals are γ -weakly informative, the reports always satisfy

The following observation is the key:

Lemma G.9. For each expert i , we have

E [ r i 1 -r i | ω = 0 ] = ρ ;

As corollaries, for k conditionally independent reports r 1 , . . . , r k , we have E [ ∏ k i =1 r i 1 -r i | ω = 0 ] = ρ k and E [ ∏ k i =1 1 -r i r i | ω = 1 ] = 1 ρ k .

Proof. Because r i 1 -r i = P ( s i | ω =1) P ( s i | ω =0) ρ (from (13)), we have

For conditionally independent r 1 , . . . , r k , we have

Let ∆ = √ ε 3 γn and suppose we are given T = 6 e γn ∆ 2 log 2 δ = 54 eγn ε log 2 δ = O ( γn ε log 1 δ ) samples. Suppose when drawing the samples we first draw the events ω ( t ) 's, and then draw the reports r ( t ) i 's conditioning on ω ( t ) being 0 or 1 . After the first step, the numbers of samples with ω ( t ) = 0 and ω ( t ) = 1 are determined, which we denote by # 0 and # 1 . Since # 0 +# 1 = T , one of them must be at least T/ 2 . We argue that whether # 0 ≥ T 2 or # 1 ≥ T 2 we can estimate ρ 1 /γ with accuracy 3∆ . (For simplicity, we assume that 1 /γ is an integer.)

Similarly, we can prove E [ 1 -r i r i | ω = 1 ] = 1 ρ and E [ ∏ k i =1 1 -r i r i | ω = 1 ] = 1 ρ k .

If # 0 ≥ T/ 2 , then we consider the # 0 n reports r ( t ) i 's in the samples with ω ( t ) = 0 . We divide these # 0 n reports evenly into # 0 nγ groups, each of size 1 /γ , denoted by G 1 , . . . , G # 0 nγ . Consider the product of r ( t ) i 1 -r ( t ) i 's in a group G j : because r ( t ) i 's are independent given ω = 0 , by Lemma G.9 we have

Using (65) and the inequality (1 + γ ) 1 /γ ≤ e , we have

Let X j be the random variable 1 eρ 1 /γ ∏ r ( t ) i ∈ G j r ( t ) i 1 -r ( t ) i . From the above equation and inequality we have E [ X j ] = 1 e and X j ∈ [ 1 e 2 , 1] ⊆ [0 , 1] . So, by Chernoff bound,

if given our choice of T . Multiplying 1 # 0 nγ ∑ # 0 nγ j =1 X j by eρ 1 /γ , we obtain the following estimate of ρ 1 /γ :

Dividing by ρ 1 /γ , we get ( ˆ ρ 0 ρ ) 1 /γ ∈ 1 ± ∆ .

If # 1 ≥ T/ 2 , then by considering the # 1 n reports in the samples with ω ( t ) = 1 , dividing them into # 1 nγ groups of size 1 /γ , H 1 , . . . , H # 1 nγ , and similarly defining random variable Y j = ρ 1 /γ e ∏ r ( t ) i ∈ H j 1 -r ( t ) i r ( t ) i , we obtain the following estimate of ( 1 ρ ) 1 /γ :

Multiplying by ρ 1 /γ , we get ( ρ ˆ ρ 1 ) 1 /γ ∈ 1 ± ∆ . Taking the reciprocal and noticing that 1 1 ± ∆ ⊆ 1 ± 3∆ when ∆ < 1 3 , we obtain ( ˆ ρ 1 ρ ) 1 /γ ∈ 1 ± 3∆ .

From the discussion above we obtained an estimate ˆ ρ ∈ { ˆ ρ 0 , ˆ ρ 1 } of ρ such that ( ˆ ρ ρ ) 1 /γ ∈ 1 ± 3∆ . Raising to the power of γ , and using the inequality (1 -x ) γ ≥ 1 -xγ and (1+ x ) γ ≤ e xγ ≤ 1+2 xγ for xγ ≤ 1 , we get

In particular, this implies | ˆ ρ -ρ | ρ ≤ 6∆ γ = 2 √ ε n . Then, according to Claim G.8, the aggregator ˆ f defined by ˆ f ( r ) = 1 1+ˆ ρ n -1 ∏ n i =1 1 -r i r i is ε -optimal. We hence obtained an ε -optimal aggregator.

H Missing Proofs from Section 7

H.1 Proof of Theorem 7.1

Regard P as a joint distribution over reports r = ( r 1 , . . . , r n ) and the state ω , where r i is sampled by first sampling s i ∈ S i and then letting r i = P ( ω = 1 | s i ) . Since |S i | = m , there are at most m different values of r i that can be sampled, so there are at most 2 m n different tuples of ( r , ω ) in the support of P . For each such tuple ( r , ω ) , consider the empirical probability of this tuple:

By the Chernoff bound, we have

∣ ∣ Using a union bound for all the 2 m n tuples and the fact that P ( r , ω ) ≥ P ( s , ω ) > c m n (where s ∈ S are some signals that generate r ), we have

∣ ∣ holds for all tuples ( r , ω ) except with probability at most

Assuming (66) holds, we consider the 'empirical' Bayesian aggregator:

Since (66) implies ˆ P ( r , ω ) ∈ (1 ± ∆) P ( r , ω ) and ˆ P ( r ) ∈ (1 ± ∆) P ( r ) , we have and

if ∆ < 1 2 . Putting these two inequalities together:

We note that this holds for all possible r in the support of P . So, the expected approximation error of ˆ f is at most:

Letting 16∆ 2 = ε , namely ∆ 2 = ε 16 , we have E [ | ˆ f ( r ) -f ∗ ( r ) | 2 ] ≤ ε , so ˆ f is an ε -optimal aggregator. Plugging ∆ 2 = ε 16 into (67) we obtain the sample complexity

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors study the problem of forecast aggregation in a Bayesian setting. Here n experts observe an individual signal that is correlated with the truth, and each expert reports their posterior belief to the principal. The principal aggregates these reports and outputs a prediction, whose quality is assessed using square loss. The object of interest is the minimax sample complexity necessary to obtain an additive excess error of $\\epsilon$ compared to the best aggregator that knows the conditional distribution of truth given reports. \n\nThe authors derive results for arbitrary distributions $P(\\omega, s)$ on truth and signals, as well as distributions that factorize as $P(\\omega,s) = P(\\omega) \\prod_i P(s_i|\\omega)$. They show that there is an exponential gap in sample complexity between these two cases, and in the latter case the complexity does not depend on the number of experts $n$.\n\nSTRENGTHS:\n- Well written and appears technically sound\n- Introduces the study of sample complexity to the forecast aggregation literature\n- Interesting and general results that cover multiple natural settings\n- A novel lower bound construction for distribution estimation that allows reduction from estimation to aggregation\n\nWEAKNESSES:\n- Non-matching upper and lower bounds in most cases\n- Main technical difficulty lies in construction of lower bound, the upper bounds don't require significant new ideas\n- Techniques seem limited to the specific loss function + discrete truth/signal setting. Commenting on possible extensions in these directions would be interesting\n\nQUESTIONS:\nHave the authors thought about results for alternative loss functions (e.g. different weight for type2 vs type1 errors) or continuous truth/signal distributions where for example the experts report the condition mean of the truth given their signal?\n\nLIMITATIONS:\nThe authors address limitations.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 2, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-01"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper initiates the study of sample complexity of forecast aggregation under a Bayesian forecasting model. In this problem, n experts, each observe a private signal about an unknown binary event, and then report their posterior beliefs about the event to a principal, who then aggregates the reports into a single prediction. The underlying joint distribution is unknown to the principal, but he has access to i.i.d. samples from the distribution. Using these samples, the principal aims to find an epsilon-approximate optimal aggregator, where optimality is measured in terms of the mean squared error between the aggregated prediction and the real event. The authors show that the sample complexity grows exponentially in the number of experts n, but that if the experts’ signals are conditionally independent, then the sample complexity does not depend on the number of experts at all. They further consider the case of non-binary events and weakly/strongly informative experts.\n\nSTRENGTHS:\nThe paper is very elegantly written. It presents the setup of the problem clearly, motivates it thoroughly, and initiates an interesting discussion on the fundamental limits of the problem. The proof sketches are quite intuitive and convincing and their implications are reflected and discussed.\n\nWEAKNESSES:\nOne weakness for me is in the particular choice of mean squared error as an optimality measure. It seems counter-intuitive, given that the experts report their posterior beliefs, which is in essence a minimum error probability optimality measure. It would seem more natural for the principal to look for an aggregation that minimizes the probability of error given the exerts reports. Another weakness is the gap between the upper and lower bounds with respect to epsilon, which might follow from the relatively simple upper bound proposed in the paper. This also bleeds over to the conditionally independent variant. Finally, the choice of averaged squared error as an optimality measure in the multi-outcome events section brings forth quite bizarre looking results. It follows, under this measure, that is the number of events is omega(1/epsilon^2) our task succeeds with zero samples! The authors do clarify this in the appendix, but I would remove it altogether from the paper (or change the measure to additive MSE).\n\nQUESTIONS:\nDid you consider using the maximum a posterior optimality measure for the principal? An added explanation on why MSE is a good choice can benefit your work.\n\nLIMITATIONS:\nThe authors adequately addressed the limitations of their work", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-07-04"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper studies the aggregation of expert opinions in a Bayesian setting for discrete (binary) distributions. The setting further limits each expert to have at most m different opinions, making the problem fully discrete. This reduction enables the analysis of the sampling complexity of the problem, i.e., the minimum number of observed aggregation events (expert opinions and true outcome) required to get an epsilon tight estimate on the true distriution in the total variation norm.\n\nSTRENGTHS:\nThe paper approaches the more and more important problem of how to combine different \"expert\" opinions in a thorough and interesting way. It is well written, clear and, as far as I managed to go into detail, correct.\nTheir approach sparks many new ideas on how to approach aggregation problems and opens up interesting questions on the relevance of (historical) data in aggregation.\n\nWEAKNESSES:\nThe proof of the lower bound in Section 4.2 doesn't seem intuitive. The mere construction of an example is of course sufficient for a formal proof, but an explanation of why the authors had the idea for this specific construction might be more illuminating.\nThe authors sometimes speak of the \"number of signals an expert can possibly observe.\" Maybe this is better explained as the \"cardinality of the signal the experts observe\"\n\nQUESTIONS:\nIs it essential in the proof that the experts decide on Bayesian arguments? Or does it suffice to assume that each expert always just makes one of m possible reports, i.e., can we focus only on the joint distribution of r^t and omega and ignore s^t?\nIn Section 6, does bigger Omega really reduce the complexity? This is counterintuitive.\nWhat is the source of Theorem 7.1?\n\nLIMITATIONS:\nThe authors clearly state the limitations of their work, are honest and clear about the current mismatch in their bounds in epsilon, and don't oversell their results.\nI do not see potential societal impact.", "TITLE": "Official Review", "RECOMMENDATION": 8, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-05"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper studies the problem of forecast aggregation: There are n experts who are each given signals about some unknown binary event, and each output a posterior probability based on these signals. We have access to k such reports (corresponding to k events). The task is to aggregate these reports to produce a prediction that is close to the unknown event in expected squared distance. The paper shows that there is an exponential gap in sample complexity (ie k) between the cases when the experts' signals are independent conditioned on the event, versus the general case. In addition, a (slightly loose) upper bound is provided for the arbitrary case, and a lower bound is provided for the conditionally independent case.\n\nSTRENGTHS:\nThis is a very clearly written paper, and it studies an interesting problem. The story of the exponential sample complexity difference in the arbitrary signals vs conditionally independent signal setting is a compelling one. Overall, I think this is an interesting paper that deserves to be accepted to NeurIPS.\n\nMore comments:\n- It studies a natural extension where the aggregator sees multiple samples from each expert. This is a more realistic setting than the one-shot problem.\n- There are natural connections to distribution estimation here that are interesting. For this reason, I think this paper will appeal to the CS theory community.\n- The future directions mentioned are compelling, and will likely lead to several interesting results.\n\nWEAKNESSES:\nI don't see any real weaknesses.\n\nQUESTIONS:\n- Is it possible to say something about when you expect the signals to be conditionally independent in the real world? For instance, your paper points to a Kaggle dataset - is there any evidence that there is conditional independence there? Do you expect your algorithms to work well on that dataset?\n- Can you draw any connections between (variants of) this problem and distribution testing?\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-06"}