SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
0 \leq \sum_ {j \in \Omega} \left| f _ {j} (\boldsymbol {r}) - \mathbb {1} [ \omega = j ] \right| ^ {2} \leq \sum_ {j \in \Omega} \left| f _ {j} (\boldsymbol {r}) - \mathbb {1} [ \omega = j ] \right| \leq 1 + \sum_ {j \in \Omega} f _ {j} (\boldsymbol {r}) = 2. \tag {18}
$$
$$
0 \leq \sum_ {j \in \Omega} \left| f _ {j} (\boldsymbol {r}) - f _ {j} ^ {*} (\boldsymbol {r}) \right| ^ {2} \leq 2. \tag {19}
$$
C.1.1 Lower Bounds
The lower bounds directly follow from the lower bounds for the binary case (Theorem 4.1 and Theorem 5.2) because the binary case is a special case of the multi-outcome case. Specifically, we can regard any binary event distribution  $P$  as a multi-outcome distribution that puts probability only on outcomes  $\{1,2\} \subseteq \Omega$ . If we can learn an  $\varepsilon$ -optimal aggregator  $\hat{\pmb{f}}$  for the multi-outcome case:  $\mathbb{E}\big[\sum_{j\in \Omega}|\hat{f}_j(\pmb {r}) - f_j^* (\pmb {r})|^2\big]\leq \varepsilon$ , then this aggregator satisfies
$$
\begin{array}{l} \mathbb {E} \left[ \sum_ {j \in \{1, 2 \}} \left| \hat {f} _ {j} (\boldsymbol {r}) - f _ {j} ^ {*} (\boldsymbol {r}) \right| ^ {2} \right] \leq \varepsilon \\ \Longleftrightarrow \mathbb {E} \Big [ \big | \hat {f} _ {1} (\boldsymbol {r}) - f _ {1} ^ {*} (\boldsymbol {r}) \big | ^ {2} + \big | (1 - \hat {f} _ {1} (\boldsymbol {r})) - (1 - f _ {1} ^ {*} (\boldsymbol {r})) \big | ^ {2} \Big ] = \mathbb {E} \Big [ 2 \big | \hat {f} _ {1} (\boldsymbol {r}) - f _ {1} ^ {*} (\boldsymbol {r}) \big | ^ {2} \Big ] \leq \varepsilon \\ \Longleftrightarrow \mathbb {E} \left[ \left| \hat {f} _ {1} (\boldsymbol {r}) - f _ {1} ^ {*} (\boldsymbol {r}) \right| ^ {2} \right] \leq \frac {\varepsilon}{2}. \\ \end{array}
$$
So, the aggregator  $\hat{f}_1(\cdot)$  is an  $\frac{\varepsilon}{2}$ -optimal aggregator for the binary case.
C.1.2 Upper Bounds
General distributions: the  $O\left(\frac{|\Omega|m^n + \log(1 / \delta)}{\varepsilon^2}\right)$  upper bound. The proof for general distributions in the multi-outcome case is the same as the proof for general distributions in the binary case (Section 4.1), except for three differences: (1)  $P$  (regarded as a distribution over reports  $\pmb{r}$  and event  $\omega$ ) now is a discrete distribution with support size at most  $|\Omega |m^{n}$ ; (2) the loss  $|f(r) - \omega |^2\in [0,1]$  in the binary case is replaced by the loss  $\sum_{j\in \Omega}|f_j(\boldsymbol {r}) - \mathbb{1}[\omega = j]|^2\in [0,2]$  in the multi-outcome case; (3) the  $\varepsilon$  in the binary case is replaced by  $\frac{\varepsilon}{2}$  because the loss is now upper bounded by 2. Thus, the bound  $O\left(\frac{m^n + \log(1 / \delta)}{\varepsilon^2}\right)$  in the binary case becomes  $O\left(\frac{|\Omega|m^n + \log(1 / \delta)}{(\frac{\varepsilon}{2})^2}\right) = O\left(\frac{|\Omega|m^n + \log(1 / \delta)}{\varepsilon^2}\right)$  in the multi-outcome case.
Conditionally independent distributions: the  $O\left(\frac{|\Omega| \log |\Omega|}{\varepsilon^2} \log \frac{1}{\varepsilon} + \frac{1}{\varepsilon^2} \log \frac{1}{\delta}\right)$  upper bound. The sample complexity upper bound for conditionally independent distributions is proved by a pseudodimension argument. We first show in Lemma C.2 that the optimal aggregator  $f^*$  belongs to some parametric family of aggregators. Then, we upper bound the pseudo-dimension of the loss functions associated with this parametric family of aggregators, which will give the desired sample complexity upper bound.
Lemma C.2. For multi-outcome conditionally independent distribution  $P$ , given signals  $s = (s_i)_{i=1}^n$ , with corresponding reports  $r = (r_i)_{i=1}^n = (r_{ij})_{ij}$  where  $r_{ij} = P(\omega = j \mid s_i)$ , the posterior probability of  $\omega = j$  satisfies:
$$
f _ {j} ^ {*} (\boldsymbol {r}) = P (\omega = j \mid \boldsymbol {r}) = P (\omega = j \mid \boldsymbol {s}) = \frac {\frac {1}{P (\omega = j) ^ {n - 1}} \prod_ {i = 1} ^ {n} r _ {i j}}{\sum_ {k \in \Omega} \frac {1}{P (\omega = k) ^ {n - 1}} \prod_ {i = 1} ^ {n} r _ {i k}}.
$$
Proof. By Bayes' rule and the fact that  $s_1, \ldots, s_n$  are independent conditioning on  $\omega$ ,
$$
\begin{array}{l} P (\omega = j \mid s) = \frac {P (\omega = j) \prod_ {i = 1} ^ {n} P (s _ {i} \mid \omega = j)}{\sum_ {k \in \Omega} P (\omega = k) \prod_ {i = 1} ^ {n} P (s _ {i} \mid \omega = k)} \\ = \frac {\frac {1}{P (\omega = j) ^ {n - 1}} \prod_ {i = 1} ^ {n} P (\omega = j) P (s _ {i} \mid \omega = j)}{\sum_ {k \in \Omega} \frac {1}{P (\omega = k) ^ {n - 1}} \prod_ {i = 1} ^ {n} P (\omega = k) P (s _ {i} \mid \omega = k)} \\ = \frac {\frac {1}{P (\omega = j) ^ {n - 1}} \prod_ {i = 1} ^ {n} \frac {P (\omega = j) P (s _ {i} | \omega = j)}{P (s _ {i})}}{\sum_ {k \in \Omega} \frac {1}{P (\omega = k) ^ {n - 1}} \prod_ {i = 1} ^ {n} \frac {P (\omega = k) P (s _ {i} | \omega = k)}{P (s _ {i})}} \\ = \frac {\frac {1}{P (\omega = j) ^ {n - 1}} \prod_ {i = 1} ^ {n} r _ {i j}}{\sum_ {k \in \Omega} \frac {1}{P (\omega = k) ^ {n - 1}} \prod_ {i = 1} ^ {n} r _ {i k}}. \\ \end{array}
$$
Since the above expression depends only on  $r_i$ 's but not on  $s_i$ 's, we have:
(For the special binary event case,
$$
f ^ {*} (\boldsymbol {r}) = P (\omega = 1 \mid \boldsymbol {r}) = \frac {\frac {1}{P (\omega = 1) ^ {n - 1}} \prod_ {i = 1} ^ {n} r _ {i}}{\frac {1}{P (\omega = 1) ^ {n - 1}} \prod_ {i = 1} ^ {n} r _ {i} + \frac {1}{P (\omega = 0) ^ {n - 1}} \prod_ {i = 1} ^ {n} (1 - r _ {i})} = \frac {1}{1 + (\frac {P (\omega = 1)}{P (\omega = 0)}) ^ {n - 1} \prod_ {i = 1} ^ {n} \frac {1 - r _ {i}}{r _ {i}}}.
$$
This proves Lemma 5.1.)
According to Lemma C.2, the optimal aggregator  $\pmb{f}^{*} = (f_{j})_{j\in \Omega}$  belongs to the following family of aggregators, parameterized by  $\theta = (\theta_j)_{j\in \Omega}\in \mathbb{R}_+^{|\Omega|}$ :
$$
\mathcal {F} = \left\{\boldsymbol {f} ^ {\boldsymbol {\theta}} = \left(f _ {j} ^ {\boldsymbol {\theta}}\right) _ {j \in \Omega} \mid f _ {j} ^ {\boldsymbol {\theta}} (\boldsymbol {r}) = \frac {\theta_ {j} \prod_ {i = 1} ^ {n} r _ {i j}}{\sum_ {k \in \Omega} \theta_ {k} \prod_ {i = 1} ^ {n} r _ {i k}} \right\} \tag {20}
$$
(The optimal aggregator  $\pmb{f}^*$  has parameter  $\theta_j = \frac{1}{P(\omega = j)^{n-1}}$ .) Let  $\mathcal{G}$  be the family of loss functions associated with  $\mathcal{F}$ , which is also parameterized by  $\pmb{\theta} \in \mathbb{R}_+^{|\Omega|}$ ,
$$
\mathcal {G} = \left\{g ^ {\boldsymbol {\theta}} \mid g ^ {\boldsymbol {\theta}} (\boldsymbol {r}, \omega) = \sum_ {j \in \Omega} \left| f _ {j} ^ {\boldsymbol {\theta}} (\boldsymbol {r}) - \mathbb {1} [ \omega = j ] \right| ^ {2} \right\}. \tag {21}
$$
We recall the definition of pseudo-dimension of a family of functions:
Definition C.3 (e.g., [1]). Let  $\mathcal{H}$  be a family of functions from input space  $\mathcal{X}$  to real numbers  $\mathbb{R}$ . Let  $x^{(1)},\ldots ,x^{(d)}\in \mathcal{X}$  be  $d$  inputs. Let  $t^{(1)},\ldots ,t^{(d)}\in \mathbb{R}$  be  $d$  "thresholds". Let  $\pmb {b} = (b^{(1)},\dots,b^{(d)})\in$ $\{-1, + 1\} ^d$  be a vector of labels. We say  $\pmb{b}$  can be generated by  $\mathcal{H}$  (on inputs  $x^{(1)},\ldots ,x^{(d)}$  with thresholds  $t^{(1)},\ldots ,t^{(d)}$  ) if there exists a function  $h_b\in \mathcal{H}$  such that  $h_b(x^{(i)}) > t^{(i)}$  if  $b^{(i)} = 1$  and  $h_b(x^{(i)}) < t^{(i)}$  if  $b^{(i)} = -1$  (namely,  $\mathrm{sign}(h_b(x^{(i)}) - t^{(i)}) = b^{(i)})$  for every  $i\in \{1,\dots,d\}$ . The pseudo-dimension of  $\mathcal{H}$ ,  $\mathrm{Pdim}(\mathcal{H})$ , is the largest integer  $d$  for which there exist  $d$  inputs and  $d$  thresholds such that all the  $2^d$  label vectors in  $\{-1, + 1\} ^d$  can be generated by  $\mathcal{H}$ .
Pseudo-dimension gives a sample complexity upper bound for the uniform convergence of the empirical means of a family of functions to their true means:
Theorem C.4 (e.g., [1]). Let  $\mathcal{H}$  be a family of functions from  $\mathcal{X}$  to  $[0, U]$  with  $\operatorname{Pdim}(\mathcal{H}) = d$ . For any distribution  $\mathcal{D}$  on  $\mathcal{X}$ , with probability at least  $1 - \delta$  over the random draw of
$$
T = O \left(\frac {U ^ {2}}{\varepsilon^ {2}} \left(d \cdot \log \frac {U}{\varepsilon} + \log \frac {1}{\delta}\right)\right)
$$
samples  $x^{(1)},\ldots ,x^{(T)}$  from  $\mathcal{D}$ , we have: for every  $h\in \mathcal{H}$ $\left|\mathbb{E}_{x\sim \mathcal{D}}[h(x)] - \frac{1}{T}\sum_{t = 1}^{T}h(x^{(t)})\right|\leq \varepsilon$
We now upper bound the pseudo-dimension of  $\mathcal{G}$ :
Lemma C.5.  $\operatorname{Pdim}(\mathcal{G}) \leq O\big(|\Omega| \log |\Omega|\big)$ .
Proof. Suppose  $\mathrm{Pdim}(\mathcal{G}) = d$ . By definition, there exist  $d$  inputs  $(\pmb{r}^{(1)},\omega^{(1)}),\dots ,(\pmb{r}^{(d)},\omega^{(d)})$  and  $d$  thresholds  $t^{(1)},\ldots ,t^{(d)}\in \mathbb{R}$  such that all the  $2^d$  label vectors  $\pmb {b}\in \{-1, + 1\} ^d$  can be generated by some function  $g^{\theta_b}\in \mathcal{G}$ . We will count how many label vectors can actually be generated by  $\mathcal{G}$ .
Consider the  $\ell$ -th input  $(\pmb{r}^{(\ell)}, \omega^{(\ell)})$  and threshold  $t^{(\ell)}$ . To simplify notations we omit superscript  $(\ell)$  so we have  $(\pmb{r}, \omega)$  and  $t$ . We write  $\pmb{x} = (x_{j})_{j \in \Omega}$  where  $x_{j} = \prod_{i=1}^{n} r_{ij}$ . For any function  $g^{\pmb{\theta}} \in \mathcal{G}$ , we have
$$
\begin{array}{l} g ^ {\boldsymbol {\theta}} (\boldsymbol {r}, \omega) = \sum_ {j \in \Omega} \left| \frac {\theta_ {j} x _ {j}}{\sum_ {k \in \Omega} \theta_ {k} x _ {k}} - \mathbb {1} [ \omega = j ] \right| ^ {2} \\ = \sum_ {j \in \Omega} \left(\frac {\theta_ {j} x _ {j}}{\sum_ {k \in \Omega} \theta_ {k} x _ {k}}\right) ^ {2} - 2 \frac {\theta_ {\omega} x _ {\omega}}{\sum_ {k \in \Omega} \theta_ {k} x _ {k}} + 1 \\ = \frac {1}{\left(\sum_ {k \in \Omega} \theta_ {k} x _ {k}\right) ^ {2}} \bigg [ \sum_ {j \in \Omega} (\theta_ {j} x _ {j}) ^ {2} - 2 \theta_ {\omega} x _ {\omega} \sum_ {k \in \Omega} \theta_ {k} x _ {k} + \left(\sum_ {k \in \Omega} \theta_ {k} x _ {k}\right) ^ {2} \bigg ]. \\ \end{array}
$$
By definition, the set of parameters that give input  $(\pmb{r},\omega)$  label “+1” is  $\{\pmb {\theta}\in \mathbb{R}_+^{|\Omega |}\mid g^\pmb {\theta}(\pmb {r},\omega) > t\}$ . By the above equation, this set is equal to the set
$$
\left\{\boldsymbol {\theta} \in \mathbb {R} _ {+} ^ {| \Omega |} \Bigg | \sum_ {j \in \Omega} (\theta_ {j} x _ {j}) ^ {2} - 2 \theta_ {\omega} x _ {\omega} \sum_ {k \in \Omega} \theta_ {k} x _ {k} + \Big (\sum_ {k \in \Omega} \theta_ {k} x _ {k} \Big) ^ {2} > t \Big (\sum_ {k \in \Omega} \theta_ {k} x _ {k} \Big) ^ {2} \right\}.
$$
We note that the above set is the solution set to the quadratic form inequality  $\pmb{\theta}^{\top}A\pmb{\theta} > 0$  for some matrix  $A \in \mathbb{R}^{|\Omega| \times |\Omega|}$ . Similarly, the set of parameters that give input  $(r,\omega)$  label “ $-1$ ” is the solution set to  $\pmb{\theta}^{\top}A\pmb{\theta} < 0$ . These two sets share a boundary: the solution set to  $\pmb{\theta}^{\top}A\pmb{\theta} = 0$ , which is a hyper-ellipsoid in  $\mathbb{R}^{|\Omega|}$ . In other words, the input  $(r,\omega)$  and threshold  $t$  define a hyper-ellipsoid which divides the parameter space  $\mathbb{R}_+^{|\Omega|}$  into two regions such that all the parameters in one region generate the same label for that input.
Enumerating all inputs  $(r^{(1)},\omega^{(1)}),\ldots ,(r^{(d)},\omega^{(d)})$  . They define  $^d$  hyper-ellipsoids in total, dividing the parameter space  $\mathbb{R}_+^{|\Omega |}$  into several regions. Within each region, all the parameters generate the same label for each input and hence generate the same label vector. So, the number of label vectors that can be generated by all the parameters in  $\mathbb{R}_+^{|\Omega |}$  is upper bounded by the number of regions. The number of regions divided by  $^d$  hyper-ellipsoids in  $\mathbb{R}_+^{|\Omega |}$  is in the order of  $O(d^{|\Omega |})$  . Hence, to generate all the  $2^{d}$  label vectors we need  $O(d^{|\Omega |})\geq 2^{d}$  . This gives  $d\leq O(|\Omega |\log |\Omega |)$
By Theorem C.4 and Lemma C.5, plugging in  $d = \mathrm{Pdim}(\mathcal{G}) \leq O(|\Omega| \log |\Omega|)$  and  $U = 2$  (because  $g^{\theta}(r, \omega)$  is bounded by [0, 2] according to (18)), we obtain the following: with probability at least  $1 - \delta$  over the random draw of
$$
T = O \bigg (\frac {U ^ {2}}{\varepsilon^ {2}} \Big (d \cdot \log \frac {U}{\varepsilon} + \log \frac {1}{\delta} \Big) \bigg) = O \bigg (\frac {| \Omega | \log | \Omega |}{\varepsilon^ {2}} \log \frac {1}{\varepsilon} + \frac {1}{\varepsilon^ {2}} \log \frac {1}{\delta} \bigg)
$$
samples  $(\pmb{r}^{(1)},\omega^{(1)}),\dots ,(\pmb{r}^{(T)},\omega^{(T)})$  from  $P$  , we have for any  $g^{\theta}\in \mathcal{G}$
$$
\left| \mathbb {E} _ {P} \left[ g ^ {\boldsymbol {\theta}} (\boldsymbol {r}, \omega) \right] - \frac {1}{T} \sum_ {t = 1} ^ {T} g ^ {\boldsymbol {\theta}} \left(\boldsymbol {r} ^ {(t)}, \omega^ {(t)}\right) \right| \leq \varepsilon .
$$
This is equivalent to for any  $f^{\theta}\in \mathcal{F}$
$$
\begin{array}{l} \left| \mathbb {E} _ {P} \left[ \sum_ {j \in \Omega} \left| f _ {j} ^ {\boldsymbol {\theta}} (\boldsymbol {r}) - \mathbb {1} [ \omega = j ] \right| ^ {2} \right] - \frac {1}{T} \sum_ {t = 1} ^ {T} \sum_ {j \in \Omega} \left| f _ {j} ^ {\boldsymbol {\theta}} (\boldsymbol {r} ^ {(t)}) - \mathbb {1} [ \omega^ {(t)} = 1 ] \right| ^ {2} \right| \leq \varepsilon \\ \Longleftrightarrow \left| L _ {P} (\boldsymbol {f} ^ {\boldsymbol {\theta}}) - L _ {\hat {P} _ {\mathrm {e m p}}} (\boldsymbol {f} ^ {\boldsymbol {\theta}}) \right| \leq \varepsilon , \\ \end{array}
$$
where  $\hat{P}_{\mathrm{emp}}$  is the empirical distribution  $\mathrm{Uniform}\bigl \{(r^{(1)},\omega^{(1)}),\ldots ,(r^{(T)},\omega^{(T)})\bigr \}$ . By the same logic as the proof of the upper bound in Theorem 4.1 (Section 4.1), the empirically optimal aggregator  $\hat{\pmb{f}}^{*} = \operatorname *{argmin}_{\pmb {f}^{\theta}\in \mathcal{F}}L_{\hat{P}_{\mathrm{emp}}}(\pmb{f}^{\theta})$  is  $2\varepsilon$ -optimal.
D Missing Proofs from Section 2
D.1 Proof of Lemma 2.1
To prove the first item  $f^{*}(\boldsymbol{r}) = P(\omega = 1 \mid \boldsymbol{r})$ , we simply note that
$$
f ^ {*} = \underset {f} {\mathrm {a r g m i n}} \mathbb {E} _ {\pmb {r}} \left[ \mathbb {E} \left[ | f (\pmb {r}) - \omega | ^ {2} \mid \pmb {r} \right] \right]
$$
should minimize  $\mathbb{E}\big[|f(\boldsymbol {r}) - \omega |^2\mid \boldsymbol {r}\big]$  for almost every  $\pmb{r}$ . This gives  $f^{*}(\boldsymbol {r}) = \mathbb{E}[\omega |\boldsymbol {r}] = P(\omega = 1\mid \boldsymbol {r})$
To prove the second item  $L_{P}(f) - L_{P}(f^{*}) = \mathbb{E}_{P}\bigl [|f(\boldsymbol {r}) - f^{*}(\boldsymbol {r})|^{2}\bigr ]$ , we note that, by the definition of  $L_{P}(\cdot)$  and the fact that  $f^{*}(\boldsymbol {r}) = \mathbb{E}[\omega \mid \boldsymbol {r}]$  proven above,
$$
\begin{array}{l} L _ {P} (f) - L _ {P} \left(f ^ {*}\right) = \mathbb {E} \left[ | f (\boldsymbol {r}) - \omega | ^ {2} \right] - \mathbb {E} \left[ | f ^ {*} (\boldsymbol {r}) - \omega | ^ {2} \right] \\ = \mathbb {E} [ f (\boldsymbol {r}) ^ {2} ] - 2 \mathbb {E} [ f (\boldsymbol {r}) \omega ] - \mathbb {E} [ f ^ {*} (\boldsymbol {r}) ^ {2} ] + 2 \mathbb {E} [ f ^ {*} (\boldsymbol {r}) \omega ] \\ = \mathbb {E} \left[ f (\boldsymbol {r}) ^ {2} \right] - 2 \mathbb {E} _ {\boldsymbol {r}} \left[ f (\boldsymbol {r}) \mathbb {E} [ \omega | \boldsymbol {r} ] \right] - \mathbb {E} \left[ f ^ {*} (\boldsymbol {r}) ^ {2} \right] + 2 \mathbb {E} _ {\boldsymbol {r}} \left[ f ^ {*} (\boldsymbol {r}) \mathbb {E} [ \omega | \boldsymbol {r} ] \right] \\ = \mathbb {E} [ f (\boldsymbol {r}) ^ {2} ] - 2 \mathbb {E} _ {\boldsymbol {r}} [ f (\boldsymbol {r}) f ^ {*} (\boldsymbol {r}) ] - \mathbb {E} [ f ^ {*} (\boldsymbol {r}) ^ {2} ] + 2 \mathbb {E} _ {\boldsymbol {r}} [ f ^ {*} (\boldsymbol {r}) ^ {2} ] \\ = \mathbb {E} \left[ f (\boldsymbol {r}) ^ {2} - 2 f (\boldsymbol {r}) f ^ {*} (\boldsymbol {r}) + f ^ {*} (\boldsymbol {r}) ^ {2} \right] \\ = \mathbb {E} \left[ | f (\boldsymbol {r}) - f ^ {*} (\boldsymbol {r}) | ^ {2} \right]. \\ \end{array}
$$
E Missing Proofs from Section 4
Recall that we have a family  $\mathcal{D}$  of distributions over  $\mathcal{S} = S_1 \times \dots \times S_n$  satisfying the three properties in Definition 4.2 ( $B$ -uniformly bounded, same marginal across distributions, and distinct marginals across signals). We have constructed from  $\mathcal{D}$  the family of distributions  $\mathcal{P} = \{P_D : D \in \mathcal{D}\}$  for the forecast aggregation problem as follows:
$$
\left\{ \begin{array}{l l} P _ {D} (\omega = 0) = P _ {D} (\omega = 1) = \frac {1}{2}, \\ P _ {D} (\cdot | \omega = 0) = \operatorname {U n i f o r m} (\mathcal {S}), & \text {n a m e l y ，} P _ {D} (s | \omega = 0) = \frac {1}{| \mathcal {S} |} = \frac {1}{m ^ {n}} \\ P _ {D} (\cdot | \omega = 1) = D, & \text {n a m e l y ，} P _ {D} (s | \omega = 1) = D (s). \end{array} \right. \tag {22}
$$
We want to show that  $\varepsilon$ -optimal aggregation with respect to  $\mathcal{P}$  will imply  $(1 + B)^2\sqrt{\varepsilon}$  total variation distance learning with respect to  $\mathcal{D}$ , and hence  $T_{\mathcal{P}}(\varepsilon, \delta) \geq T_{\mathcal{D}}^{\mathrm{TV}}\big((1 + B)^2\sqrt{\varepsilon}, \delta\big)$ .
First, we have the following observations about  $\mathcal{D}$  and  $\mathcal{P}$ :
Fact E.1. For a family of distributions  $\mathcal{D}$  that satisfies the three properties in Definition 4.2:
1. Given signal  $s_i$ , expert  $i$ 's report is  $r_i = P_D(\omega = 1 \mid s_i) = \frac{D(s_i)}{\frac{1}{m} + D(s_i)}$ , which is the same for all distributions  $D \in \mathcal{D}$ .
2. For every expert  $i$ , given different signals  $s_i \neq s_i'$ , its reports  $r_i \neq r_i'$ . So, there is a one-to-one mapping between  $s_i$  and  $r_i$  for every  $i \in \{1, \dots, n\}$  and also a one-to-one mapping between the joint signal  $s = (s_1, \dots, s_n)$  and the joint report  $r = (r_1, \dots, r_n)$ .
3. For any joint signal  $\mathbf{s}$ , with corresponding joint report  $\mathbf{r}$ , we have  $D(\mathbf{s}) = \frac{f^{*}(\mathbf{r})}{m^{n}(1 - f^{*}(\mathbf{r}))}$ .
Proof. We prove the three items one by one:
1. By definition, the marginal distribution of joint signal,  $P_D(s)$ , is
$$
P _ {D} (\boldsymbol {s}) = P _ {D} (\omega = 0) P _ {D} (\boldsymbol {s} \mid \omega = 0) + P _ {D} (\omega = 1) P _ {D} (\boldsymbol {s} \mid \omega = 1) = \frac {1}{2} \left(\frac {1}{m ^ {n}} + D (\boldsymbol {s})\right). \tag {23}
$$
Fixing  $s_i$ , summing over  $s_{-i} = (s_1, \ldots, s_{i-1}, s_{i+1}, \ldots, s_n) \in S_{-i}$ , we get
$$
P _ {D} (s _ {i}) = \frac {1}{2} \Big (\frac {| \boldsymbol {\mathcal {S}} _ {- i} |}{m ^ {n}} + \sum_ {\boldsymbol {s} _ {- i} \in \boldsymbol {\mathcal {S}} _ {- i}} D (\boldsymbol {s}) \Big) = \frac {1}{2} \Big (\frac {1}{m} + D (s _ {i}) \Big).
$$
So, given signal  $s_i$ , expert  $i$  reports
$$
r _ {i} = P _ {D} (\omega = 1 \mid s _ {i}) = \frac {P _ {z} (\omega = 1) P _ {D} \left(s _ {i} \mid \omega = 1\right)}{P _ {D} \left(s _ {i}\right)} = \frac {D \left(s _ {i}\right)}{\frac {1}{m} + D \left(s _ {i}\right)}, \tag {24}
$$
and this is the same for all  $D \in \mathcal{D}$  since  $D(s_{i}) = D^{\prime}(s_{i})$  by the "same marginal across distributions" property.
2. Given  $s_i \neq s_i'$ , by the "distinct marginals across signals" property, we have  $D(s_i) \neq D(s_i')$ . Since  $r_i = \frac{D(s_i)}{\frac{1}{m} + D(s_i)}$  and  $\frac{x}{\frac{1}{m} + x}$  is a strictly increasing function of  $x$ , it follows that  $r_i \neq r_i'$ .
3. According to item 2, there is a one-to-one mapping between  $\mathbf{s} = (s_1, \dots, s_n)$  and  $\mathbf{r} = (r_1, \dots, r_n)$ ; in other words, observing signals  $s_1, \dots, s_n$  is equivalent to observing reports  $r_1, \dots, r_n$ . Therefore, by Bayes' rule we have
$$
f ^ {*} (\boldsymbol {r}) = P _ {D} (\omega = 1 \mid \boldsymbol {r}) = P _ {D} (\omega = 1 \mid \boldsymbol {s}) = \frac {P _ {D} (\omega = 1) P _ {D} (\boldsymbol {s} \mid \omega = 1)}{P _ {D} (\boldsymbol {s})}
$$
$$
\text {b y} (2 2) \text {a n d} (2 3) = \frac {D (\boldsymbol {s})}{\frac {1}{m ^ {n}} + D (\boldsymbol {s})}. \tag {25}
$$
Rearranging, we obtain  $D(\pmb {s}) = \frac{f^{*}(\pmb{r})}{m^{n}(1 - f^{*}(\pmb{r}))}$
Claim E.2. If we have an aggregator  $\hat{f}$  that is  $\varepsilon$ -optimal with respect to  $P_D$ , then we can find a distribution  $\hat{D}$  such that  $d_{\mathrm{TV}}(\hat{D}, D) \leq (1 + B)^2 \sqrt{\varepsilon}$ .
Proof. Because  $D$  is  $B$ -uniformly bounded, from (25) we can verify that  $f^{*}(\boldsymbol{r})$  satisfies  $f^{*}(\boldsymbol{r}) \leq \frac{B}{1 + B}$ . So, we can assume  $\hat{f}(\boldsymbol{r}) \leq \frac{B}{1 + B}$  as well (if  $\hat{f}(\boldsymbol{r}) > \frac{B}{1 + B}$ , we can let  $\hat{f}(\boldsymbol{r})$  be  $\frac{B}{1 + B}$ ; this only reduces the approximation error  $\mathbb{E}\big[|\hat{f}(\boldsymbol{r}) - f^{*}(\boldsymbol{r})|^{2}\big]$ ). Define  $\hat{D}$  by letting  $\hat{D}(s) = \frac{\hat{f}(s)}{m^{n}(1 - \hat{f}(s))}$ ,  $\forall s \in S$ , where  $\boldsymbol{r}$  is the reports corresponding to  $s$  (cf., Fact E.1). Then,  $d_{\mathrm{TV}}(\hat{D}, D)$  is
$$
\begin{array}{l} d _ {\mathrm {T V}} (\hat {D}, D) = \frac {1}{2} \sum_ {\boldsymbol {s} \in \boldsymbol {S}} | \hat {D} (\boldsymbol {s}) - D (\boldsymbol {s}) | = \frac {1}{2} \sum_ {\boldsymbol {s} \in \boldsymbol {S}} \left| \frac {\hat {f} (\boldsymbol {r})}{m ^ {n} (1 - \hat {f} (\boldsymbol {r}))} - \frac {f ^ {*} (\boldsymbol {r})}{m ^ {n} (1 - f ^ {*} (\boldsymbol {r}))} \right| \\ = \frac {1}{2 m ^ {n}} \sum_ {\boldsymbol {s} \in \boldsymbol {\mathcal {S}}} \left| \frac {\hat {f} (\boldsymbol {r})}{1 - \hat {f} (\boldsymbol {r})} - \frac {f ^ {*} (\boldsymbol {r})}{1 - f ^ {*} (\boldsymbol {r})} \right|. \\ \end{array}
$$
Because  $\hat{f}(\boldsymbol{r}), f^*(\boldsymbol{r}) \leq \frac{B}{1 + B}$  and the function  $\frac{x}{1 - x}$  has derivative  $\frac{1}{(1 - x)^2} \leq (1 + B)^2$  when  $x \leq \frac{B}{1 + B}$ , we have
$$
d _ {\mathrm {T V}} (\hat {D}, D) \leq \frac {1}{2 m ^ {n}} (1 + B) ^ {2} \sum_ {\boldsymbol {s} \in \boldsymbol {\mathcal {S}}} | \hat {f} (\boldsymbol {r}) - f ^ {*} (\boldsymbol {r}) |
$$
$$
\text {b y} (2 3) \leq (1 + B) ^ {2} \sum_ {\boldsymbol {s} \in \boldsymbol {S}} P _ {D} (\boldsymbol {s}) \left| \hat {f} (\boldsymbol {r}) - f ^ {*} (\boldsymbol {r}) \right| = (1 + B) ^ {2} \mathbb {E} _ {P _ {D}} \left[ \left| \hat {f} (\boldsymbol {r}) - f ^ {*} (\boldsymbol {r}) \right| \right].
$$
By Jensen's inequality  $(\mathbb{E}[X])^2 \leq \mathbb{E}[X^2]$  and by the assumption that  $\hat{f}$  is  $\varepsilon$ -optimal, we have  $\left(\mathbb{E}_{P_D}\left[|\hat{f}(\boldsymbol{r}) - f^*(\boldsymbol{r})|\right]\right)^2 \leq \mathbb{E}_{P_D}\left[|\hat{f}(\boldsymbol{r}) - f^*(\boldsymbol{r})|^2\right] \leq \varepsilon$ . Thus,
$$
d _ {\mathrm {T V}} (\hat {D}, D) \leq (1 + B) ^ {2} \sqrt {\varepsilon},
$$
Now, we present the reduction from learning  $\mathcal{D}$  in total variation distance to forecast aggregation for  $\mathcal{P}$ . We use notations  $\pmb{x}^{(1)},\dots,\pmb{x}^{(T)}\in\pmb{S}$  to represent the samples from  $D$ . From  $\pmb{x}^{(1)},\dots,\pmb{x}^{(T)}$  we construct the samples  $(\pmb{r}^{(1)},\omega^{(1)}),\dots,(\pmb{r}^{(T)},\omega^{(T)})$  for the forecast aggregation problem. After obtaining a solution  $\hat{f}$  to the latter problem, we convert it into a solution  $\hat{D}$  to the former. Details are as follows:
Input:  $T$  i.i.d. samples  $\pmb{x}^{(1)},\dots ,\pmb{x}^{(T)}$  from an unknown distribution  $D\in \mathcal{D}$
1. Draw  $T$  samples  $\omega^{(1)},\ldots ,\omega^{(T)}\sim$  Uniform  $\{0,1\}$
2. For each  $t = 1,\dots ,T$  , do the following:
- If  $\omega^{(t)} = 0$ , draw  $s^{(t)} \sim \mathrm{Uniform}(\mathcal{S})$ .
- If  $\omega^{(t)} = 1$ , let  $\pmb{s}^{(t)} = \pmb{x}^{(t)}$ .
- For each  $i$ , compute  $r_i^{(t)} = \frac{D(s_i^{(t)})}{\frac{1}{m} + D(s_i^{(t)})}$ . Let  $\pmb{r}^{(t)} = (r_1^{(t)}, \dots, r_n^{(t)})$ .