SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Let  $c = 32$ . We assume  $\varepsilon < 2^{-18}$ , so that  $c\sqrt{\varepsilon} < \frac{1}{16}$ . For  $P^1$ , we let
$$
P ^ {1} (\omega = 1) = 0. 5 - \frac {1}{1 6 n} + \frac {c \sqrt {\varepsilon}}{n} =: p ^ {1}.
$$
For  $P^2$ , we let
$$
P ^ {2} (\omega = 1) = 0. 5 - \frac {1}{1 6 n} - \frac {c \sqrt {\varepsilon}}{n} =: p ^ {2}.
$$
We require that, in the forecast aggregation problem under both distributions  $P^1$  and  $P^2$ , whenever expert  $i$  sees signal  $a, b$ , she reports
$$
r _ {a} = 0. 5, \quad r _ {b} = 0,
$$
respectively. This gives the following conditional probabilities  $P^{1}(\cdot \mid \omega), P^{2}(\cdot \mid \omega)$ :
$$
\left[ \begin{array}{c} P ^ {1} (a \mid \omega = 0) \\ P ^ {1} (a \mid \omega = 1) \end{array} \right] = \frac {p ^ {1} - r _ {b}}{r _ {a} - r _ {b}} \left[ \begin{array}{c} \frac {1 - r _ {a} ^ {1}}{1 - p ^ {1}} \\ \frac {r _ {a}}{p ^ {1}} \end{array} \right] = \left[ \begin{array}{c} \frac {1 - \frac {1}{8 n} + \frac {2 c \sqrt {\varepsilon}}{n}}{1 + \frac {1}{8 n} - \frac {2 c \sqrt {\varepsilon}}{n}} \\ 1 \end{array} \right], \quad \left[ \begin{array}{c} P ^ {1} (b \mid \omega = 0) \\ P ^ {1} (b \mid \omega = 1) \end{array} \right] = \left[ \begin{array}{c} \frac {\frac {1}{4 n} - \frac {4 c \sqrt {\varepsilon}}{n}}{1 + \frac {1}{8 n} - \frac {2 c \sqrt {\varepsilon}}{n}} \\ 0 \end{array} \right]. \tag {34}
$$
$$
\left[ \begin{array}{c} P ^ {2} (a \mid \omega = 0) \\ P ^ {2} (a \mid \omega = 1) \end{array} \right] = \frac {p ^ {2} - r _ {b}}{r _ {a} - r _ {b}} \left[ \begin{array}{c} \frac {1 - r _ {a}}{1 - p ^ {2}} \\ \frac {r _ {a}}{p ^ {2}} \end{array} \right] = \left[ \begin{array}{c} \frac {1 - \frac {1}{8 n} - \frac {2 c \sqrt {\varepsilon}}{n}}{1 + \frac {1}{8 n} + \frac {2 c \sqrt {\varepsilon}}{n}} \\ 1 \end{array} \right], \quad \left[ \begin{array}{c} P ^ {2} (b \mid \omega = 0) \\ P ^ {2} (b \mid \omega = 1) \end{array} \right] = \left[ \begin{array}{c} \frac {1}{4 n} + \frac {4 c \sqrt {\varepsilon}}{n} \\ 1 + \frac {1}{8 n} + \frac {2 c \sqrt {\varepsilon}}{n} \\ 0 \end{array} \right]. \tag {35}
$$
Given  $T$  samples from the unknown distribution  $P \in \{P^1, P^2\}$ , each of which is a vector of  $\omega^{(t)}$  and all experts' signals  $s_i^{(t)} \in \{a, b\}$ , we feed the corresponding reports  $r_i^{(t)} \in \{r_a, r_b\}$  and  $\omega^{(t)}$  to the forecast aggregation problem and obtain a solution  $\hat{f}$ , which is an  $\varepsilon$ -optimal aggregator. We want to use  $\hat{f}$  to estimate the prior  $p = P(\omega = 1) \in \{p^1, p^2\}$  so that we can tell apart  $P^1$  and  $P^2$ . Recall from Lemma 5.1 that  $f^*(\boldsymbol{r}) = \frac{1}{1 + \rho^{n-1} \prod_{i=1}^{n} \frac{1 - r_i}{r_i}}$ , where  $\rho = \frac{p}{1 - p}$ . Writing  $\rho$  in terms of  $f^*(\boldsymbol{r})$ , we have
$$
\rho = \sqrt [ n - 1 ]{\left(\frac {1}{f ^ {*} (\boldsymbol {r})} - 1\right) \prod_ {i = 1} ^ {n} \frac {r _ {i}}{1 - r _ {i}}}.
$$
In particular, when  $r_i = r_a = 0.5$  for all  $i \in \{1, \dots, n\}$ , we have:
$$
\rho = \sqrt [ n - 1 ]{\frac {1}{f ^ {*} (\boldsymbol {r} _ {0 . 5})} - 1}, \quad \boldsymbol {r} _ {0. 5} = (0. 5, \dots , 0. 5).
$$
So, we estimate  $\rho$  by:
$$
\hat {\rho} = \sqrt [ n - 1 ]{\frac {1}{\hat {f} (\boldsymbol {r} _ {0 . 5})} - 1}.
$$
Now, we want to argue that, if  $\hat{f}$  is  $\varepsilon$ -optimal, then  $|\hat{\rho} - \rho|$  is at most  $O\left(\frac{\sqrt{\varepsilon}}{n}\right)$ . Consider the function:
$$
h (x) = \sqrt [ n - 1 ]{\frac {1}{x} - 1},
$$
whose derivative is
$$
h ^ {\prime} (x) = - \frac {1}{n - 1} \left(\frac {x}{1 - x}\right) ^ {1 - \frac {1}{n - 1}} \frac {1}{x ^ {2}}.
$$
$$
\left| \hat {\rho} - \rho \right| = \left| h \left(\hat {f} \left(\boldsymbol {r} _ {0. 5}\right)\right) - h \left(f ^ {*} \left(\boldsymbol {r} _ {0. 5}\right)\right) \right|. \tag {36}
$$
Claim F.1.  $\frac{1}{2} \leq f^{*}(\boldsymbol{r}_{0.5}) \leq \frac{2}{3}$ .
Proof. For  $P \in \{P^1, P^2\}$ , its  $\rho = \frac{p}{1 - p}$  satisfies
$$
\begin{array}{l} 1 \geq \rho \geq \rho^ {n - 1} \geq \rho^ {n} \geq \left(\frac {0 . 5 - \frac {1}{1 6 n} - \frac {\sqrt {\varepsilon}}{c n}}{0 . 5 + \frac {1}{1 6 n} + \frac {c \sqrt {\varepsilon}}{n}}\right) ^ {n} \\ = \left(\frac {1 - \frac {1}{8 n} - \frac {2 c \sqrt {\varepsilon}}{n}}{1 + \frac {1}{8 n} + \frac {2 c \sqrt {\varepsilon}}{n}}\right) ^ {n} > \left(1 - 2 \left(\frac {1}{8 n} + \frac {2 c \sqrt {\varepsilon}}{n}\right)\right) ^ {n} \geq 1 - 2 \left(\frac {1}{8} + 2 c \sqrt {\varepsilon}\right) > \frac {1}{2}, \tag {37} \\ \end{array}
$$
where in the last three transitions we used the inequalities  $\frac{1 - x}{1 + x} > 1 - 2x$  and  $(1 - x / n)^n \geq 1 - x$  for  $x \in (0,1)$  and the fact that  $c\sqrt{\varepsilon} < \frac{1}{16}$ . So,
$$
f ^ {*} (\boldsymbol {r} _ {0. 5}) = \frac {1}{1 + \rho^ {n - 1}} \in \left[ \frac {1}{1 + 1}, \frac {1}{1 + \frac {1}{2}} \right] = \left[ \frac {1}{2}, \frac {2}{3} \right],
$$
With Claim F.1, we can without loss of generality assume  $\frac{1}{2} \leq \hat{f}(\boldsymbol{r}_{0.5}) \leq \frac{2}{3}$  as well (otherwise, we can truncate  $\hat{f}(\boldsymbol{r}_{0.5})$  to this range; this only reduces the approximation error  $\mathbb{E}\left[|\hat{f}(\boldsymbol{r}) - f^{*}(\boldsymbol{r})|^{2}\right]$ .
Claim F.2. For  $\frac{1}{2} \leq x \leq \frac{2}{3}$ ,  $|h'(x)| \leq \frac{8}{n - 1}$ .
$$
\left| h ^ {\prime} (x) \right| = \frac {1}{n - 1} \left(\frac {x}{1 - x}\right) ^ {1 - \frac {1}{n - 1}} \frac {1}{x ^ {2}} \leq \frac {1}{n - 1} \left(\frac {\frac {2}{3}}{1 - \frac {2}{3}}\right) ^ {1 - \frac {1}{n - 1}} \frac {1}{\left(\frac {1}{2}\right) ^ {2}} = \frac {4}{n - 1} \cdot 2 ^ {1 - \frac {1}{n - 1}} \leq \frac {8}{n - 1}.
$$
Claim F.3. If  $\hat{f}$  is  $\varepsilon$ -optimal, then  $|\hat{f}(\boldsymbol{r}_{0.5}) - f^{*}(\boldsymbol{r}_{0.5})| < 2\sqrt{\varepsilon}$ .
Proof. If  $\hat{f}$  is  $\varepsilon$ -optimal, i.e.,  $\mathbb{E}\big[|\hat{f}(\boldsymbol{r}) - f^{*}(\boldsymbol{r})|^{2}\big] \leq \varepsilon$ , then, by Jensen's inequality  $\mathbb{E}[X^2] \geq \mathbb{E}[X]^2$ , we have
$$
\sqrt {\varepsilon} \geq \mathbb {E} \left[ | \hat {f} (\boldsymbol {r}) - f ^ {*} (\boldsymbol {r}) | \right] = \sum_ {\boldsymbol {r}} P (\boldsymbol {r}) | \hat {f} (\boldsymbol {r}) - f ^ {*} (\boldsymbol {r}) | \geq P \left(\boldsymbol {r} _ {0. 5}\right) | \hat {f} \left(\boldsymbol {r} _ {0. 5}\right) - f ^ {*} \left(\boldsymbol {r} _ {0. 5}\right) |. \tag {38}
$$
For both  $P\in \{P^1,P^2\}$  , we have
$$
\begin{array}{l} P (\boldsymbol {r} _ {0. 5}) = p \cdot P (\boldsymbol {r} _ {0. 5} \mid \omega = 1) + (1 - p) \cdot P (\boldsymbol {r} _ {0. 5} \mid \omega = 0) \\ = p \cdot P (a \mid \omega = 1) ^ {n} + (1 - p) \cdot P (a \mid \omega = 0) ^ {n} \\ \geq p \cdot 1 + (1 - p) \cdot \left(\frac {1 - \frac {1}{8 n} - \frac {2 c \sqrt {\varepsilon}}{n}}{1 + \frac {1}{8 n} + \frac {2 c \sqrt {\varepsilon}}{n}}\right) ^ {n} \stackrel {{\text {b y (3 7)}}} {{>} \frac {1}{2}}, \\ \end{array}
$$
Plugging  $P(\pmb{r}_{0.5}) > \frac{1}{2}$  into (38), we get  $|\hat{f} (\pmb{r}_{0.5}) - f^{*}(\pmb{r}_{0.5})| < 2\sqrt{\varepsilon}$ .
From (36), Claim F.1, Claim F.2, and Claim F.3, we get
$$
\left| \hat {\rho} - \rho \right| = \left| h \left(\hat {f} \left(\boldsymbol {r} _ {0. 5}\right)\right) - h \left(f ^ {*} \left(\boldsymbol {r} _ {0. 5}\right)\right) \right| \leq \frac {8}{n - 1} \left| \hat {f} \left(\boldsymbol {r} _ {0. 5}\right) - f ^ {*} \left(\boldsymbol {r} _ {0. 5}\right) \right| <   \frac {8}{n - 1} \cdot 2 \sqrt {\varepsilon} = \frac {1 6}{n - 1} \sqrt {\varepsilon}. \tag {39}
$$
Since  $p = \frac{\rho}{1 + \rho}$  as a function of  $\rho$  has a bounded derivative  $\frac{\partial p}{\partial \rho} = \frac{1}{(1 + \rho)^2} \leq 1$ , Equation (39) implies
$$
| \hat {p} - p | <   \frac {1 6}{n - 1} \sqrt {\varepsilon}
$$
if we use  $\hat{p} = \frac{\hat{\rho}}{1 + \hat{\rho}}$  as an estimate of  $p$ . This allows us to tell part  $P^1$  and  $P^2$  because the difference between  $p^1$  and  $p^2$  is greater than twice of our estimation error  $|\hat{p} - p|$ :
$$
| p ^ {1} - p ^ {2} | = \frac {2 c \sqrt {\varepsilon}}{n} = \frac {6 4 \sqrt {\varepsilon}}{n} \geq \frac {6 4 \sqrt {\varepsilon}}{2 (n - 1)} = \frac {3 2 \sqrt {\varepsilon}}{n - 1} > 2 | \hat {p} - p |.
$$
Therefore, we can tell part  $P^1$  and  $P^2$  by checking whether  $p^1$  or  $p^2$  is closer to  $\hat{p}$ .
Finally, we upper bound the squared Hellinger distance between  $P^1$  and  $P^2$ . This will give the sample complexity lower bound we want.
Claim F.4.  $d_{\mathrm{H}}^{2}(P^{1},P^{2})\leq O(c^{2}\varepsilon)$
Proof. For the marginal distributions of  $\omega$ ,  $P_{\omega}^{1}$  and  $P_{\omega}^{2}$ , according to Lemma A.2 and the fact that  $1 \geq \frac{P_{\omega}^{2}(\omega)}{P_{\omega}^{1}(\omega)} = \frac{1 - \frac{1}{8n} - \frac{2c\sqrt{\varepsilon}}{n}}{1 - \frac{1}{8n} + \frac{2c\sqrt{\varepsilon}}{n}} = 1 - \frac{\frac{4c\sqrt{\varepsilon}}{n}}{1 - \frac{1}{8n} + \frac{2c\sqrt{\varepsilon}}{n}} = 1 - O\left(\frac{c\sqrt{\varepsilon}}{n}\right)$ , we have
$$
d _ {\mathrm {H}} ^ {2} \left(P _ {\omega} ^ {1}, P _ {\omega} ^ {2}\right) \leq O \left(\left(\frac {c \sqrt {\varepsilon}}{n}\right) ^ {2}\right) = O \left(\frac {c ^ {2} \varepsilon}{n ^ {2}}\right). \tag {40}
$$
Given  $\omega = 0$  or  $1$ , we consider the conditional distributions of each  $s_i$ ,  $P_{s_i|\omega}^1$  and  $P_{s_i|\omega}^2$ . For  $s_i = a$ , we have
$$
\begin{array}{l} 1 \geq \frac {P ^ {2} (a \mid \omega)}{P ^ {1} (a \mid \omega)} \geq \frac {1 - \frac {1}{8 n} - \frac {2 c \sqrt {\varepsilon}}{n}}{1 + \frac {1}{8 n} + \frac {2 c \sqrt {\varepsilon}}{n}} \cdot \frac {1 + \frac {1}{8 n} - \frac {2 c \sqrt {\varepsilon}}{n}}{1 - \frac {1}{8 n} + \frac {2 c \sqrt {\varepsilon}}{n}} \\ = \frac {1 + \frac {1}{8 n} - \frac {2 c \sqrt {\varepsilon}}{n}}{1 + \frac {1}{8 n} + \frac {2 c \sqrt {\varepsilon}}{n}} \cdot \frac {1 - \frac {1}{8 n} - \frac {2 c \sqrt {\varepsilon}}{n}}{1 - \frac {1}{8 n} + \frac {2 c \sqrt {\varepsilon}}{n}} \\ \end{array}
$$
$$
(\frac {a - x}{a + x} \geq 1 - \frac {2 x}{a}) \geq \left(1 - \frac {\frac {4 c \sqrt {\varepsilon}}{n}}{1 + \frac {1}{8 n}}\right) \cdot \left(1 - \frac {\frac {4 c \sqrt {\varepsilon}}{n}}{1 - \frac {1}{8 n}}\right) = 1 - O \big (\frac {c \sqrt {\varepsilon}}{n} \big).
$$
For  $s_i = b$ , we have
$$
1 \geq \frac {P ^ {1} (b \mid \omega)}{P ^ {2} (b \mid \omega)} \geq \frac {\frac {1}{4 n} - \frac {4 c \sqrt {\varepsilon}}{n}}{1 + \frac {1}{8 n} - \frac {2 c \sqrt {\varepsilon}}{n}} \cdot \frac {1 + \frac {1}{8 n} + \frac {2 c \sqrt {\varepsilon}}{n}}{\frac {1}{4 n} + \frac {4 c \sqrt {\varepsilon}}{n}} \geq \frac {\frac {1}{4 n} - \frac {4 c \sqrt {\varepsilon}}{n}}{\frac {1}{4 n} + \frac {4 c \sqrt {\varepsilon}}{n}} = \frac {1 - 1 6 c \sqrt {\varepsilon}}{1 + 1 6 c \sqrt {\varepsilon}} = 1 - O (c \sqrt {\varepsilon}).
$$
So,  $d_{\mathrm{H}}^{2}(P_{s_{i}|\omega}^{1}, P_{s_{i}|\omega}^{2})$  can be upper bounded as follows:
$$
\begin{array}{l} d _ {\mathrm {H}} ^ {2} (P _ {s _ {i} | \omega} ^ {1}, P _ {s _ {i} | \omega} ^ {2}) = \frac {1}{2} \left[ \left(\sqrt {P ^ {1} (a | \omega)} - \sqrt {P ^ {2} (a | \omega)}\right) ^ {2} + \left(\sqrt {P ^ {1} (b | \omega)} - \sqrt {P ^ {2} (b | \omega)}\right) ^ {2} \right] \\ = \frac {1}{2} \left[ P ^ {1} (a \mid \omega) \Big (1 - \sqrt {\frac {P ^ {2} (a \mid \omega)}{P ^ {1} (a \mid \omega)}} \Big) ^ {2} + P ^ {2} (b \mid \omega) \Big (1 - \sqrt {\frac {P ^ {1} (b \mid \omega)}{P ^ {2} (b \mid \omega)}} \Big) ^ {2} \right] \\ \leq \frac {1}{2} \left[ 1 \cdot \left(1 - \sqrt {1 - O \left(\frac {c \sqrt {\varepsilon}}{n}\right)}\right) ^ {2} + \left(\frac {1}{4 n} + \frac {4 c \sqrt {\varepsilon}}{n}\right) \cdot \left(1 - \sqrt {1 - O (c \sqrt {\varepsilon})}\right) ^ {2} \right] \\ \end{array}
$$
$$
\begin{array}{l} \left(\text {s i n c e} 1 - \sqrt {1 - x} \leq x\right) \leq \frac {1}{2} \left[ O \left(\frac {c \sqrt {\varepsilon}}{n}\right) ^ {2} + O \left(\frac {1}{n}\right) \cdot O (c \sqrt {\varepsilon}) ^ {2} \right] \\ = O \left(\frac {c ^ {2} \varepsilon}{n}\right). \tag {41} \\ \end{array}
$$
Since  $P^1 = P_{\omega}^1 \cdot \prod_{i=1}^{n} P_{s_i|\omega}^1$  and  $P^2 = P_{\omega}^2 \cdot \prod_{i=1}^{n} P_{s_i|\omega}^2$ , we have, by Lemma A.3 and Lemma A.4,
$$
\begin{array}{l} d _ {\mathrm {H}} ^ {2} (P ^ {1}, P ^ {2}) \leq d _ {\mathrm {H}} ^ {2} (P _ {\omega} ^ {1}, P _ {\omega} ^ {2}) + \max _ {\omega \in \{0, 1 \}} \left\{d _ {\mathrm {H}} ^ {2} \big (\prod_ {i = 1} ^ {n} P _ {s _ {i} | \omega} ^ {1}, \prod_ {i = 1} ^ {n} P _ {s _ {i} | \omega} ^ {2} \big) \right\} \\ \leq d _ {\mathrm {H}} ^ {2} (P _ {\omega} ^ {1}, P _ {\omega} ^ {2}) + \max _ {\omega \in \{0, 1 \}} \left\{n \cdot d _ {\mathrm {H}} ^ {2} (P _ {s _ {i} | \omega} ^ {1}, P _ {s _ {i} | \omega} ^ {2}) \right\} \\ \end{array}
$$
$$
\begin{array}{l} (4 0) \text {a n d} (4 1) \leq O \left(\frac {c ^ {2} \varepsilon}{n ^ {2}}\right) + \max  _ {\omega \in \{0, 1 \}} \left\{n \cdot O \left(\frac {c ^ {2} \varepsilon}{n}\right) \right\} \\ = O \left(c ^ {2} \varepsilon\right). \\ \end{array}
$$
Therefore, according to Lemma A.5, to tell apart  $P^1$  and  $P^2$  with probability at least  $1 - \delta$  we need at least
$$
T = \Omega \Big (\frac {1}{d _ {\mathrm {H}} ^ {2} (P ^ {1} , P ^ {2})} \log \frac {1}{\delta} \Big) = \Omega \Big (\frac {1}{c ^ {2} \varepsilon} \log \frac {1}{\delta} \Big)
$$
G Missing Proofs from Section B
G.1 Proof of Theorem B.2
G.1.1 Additional Notations and Lemmas
We introduce some additional notations and lemmas for the proof. Let  $\mu_0$  be the expected average report of all experts conditioning on  $\omega = 0$ :
$$
\mu_ {0} = \frac {1}{n} \sum_ {i = 1} ^ {n} \mathbb {E} \left[ r _ {i} \mid \omega = 0 \right] = \frac {1}{n} \sum_ {i = 1} ^ {n} \mathbb {E} _ {s _ {i} | \omega = 0} \left[ P (\omega = 1 \mid s _ {i}) \mid \omega = 0 \right], \tag {42}
$$
which is equal to the expected prediction of  $\omega$  given expert  $i$ 's signal  $s_i$  where  $s_i$  is distributed conditioning on  $\omega = 0$ , averaged over all experts. Symmetrically, let
$$
\mu_ {1} = \frac {1}{n} \sum_ {i = 1} ^ {n} \mathbb {E} [ 1 - r _ {i} \mid \omega = 1 ] = \frac {1}{n} \sum_ {i = 1} ^ {n} \mathbb {E} _ {s _ {i} \mid \omega = 1} \left[ P (\omega = 0 \mid s _ {i}) \mid \omega = 1 \right]. \tag {43}
$$
Recall that  $p = P(\omega = 1)$
Fact G.1.  $(1 - p)\mu_0 = p\mu_1$
Proof. For each expert  $i$ , by the law of total expectation and the fact that  $r_i = P(\omega = 1 \mid s_i)$ , we have the following equations:
$$
\begin{array}{l} (1 - p) \cdot \mathbb {E} [ r _ {i} \mid \omega = 0 ] + p \cdot \mathbb {E} [ r _ {i} \mid \omega = 1 ] = P (\omega = 0) \cdot \mathbb {E} [ r _ {i} \mid \omega = 0 ] + P (\omega = 1) \cdot \mathbb {E} [ r _ {i} \mid \omega = 1 ] \\ = \mathbb {E} [ r _ {i} ] \\ = \mathbb {E} _ {s _ {i}} \left[ P (\omega = 1 \mid s _ {i}) \right] = P (\omega = 1) = p. \\ \end{array}
$$
Subtracting  $p$  from both sides, we get
$$
(1 - p) \cdot \mathbb {E} [ r _ {i} \mid \omega = 0 ] - p \cdot \mathbb {E} [ 1 - r _ {i} \mid \omega = 1 ] = 0.
$$
Averaging over all experts  $i\in \{1,\dots ,n\}$  , we conclude that
$$
(1 - p) \cdot \frac {1}{n} \sum_ {i = 1} ^ {n} \mathbb {E} [ r _ {i} \mid \omega = 0 ] - p \cdot \frac {1}{n} \sum_ {i = 1} ^ {n} \mathbb {E} [ 1 - r _ {i} \mid \omega = 1 ] = 0.
$$
Lemma G.2. If  $(1 - p)\mu_0 = p\mu_1 < \frac{\varepsilon}{2}$ , then the averaging aggregator  $f_{avg}(\pmb{r}) = \frac{1}{n}\sum_{i=1}^{n} r_i$  is  $\varepsilon$ -optimal.
Proof. If  $(1 - p)\mu_0 = p\mu_1 < \frac{\varepsilon}{2}$ , then the expected loss of  $f_{avg}$  is at most
$$
\begin{array}{l} L _ {P} (f _ {a v g}) = \mathbb {E} \left[ | f _ {a v g} - \omega | ^ {2} \right] \\ = p \mathbb {E} \left[ \left(f _ {\text {a v g}} (\boldsymbol {r}) - 1\right) ^ {2} \mid \omega = 1 \right] + (1 - p) \mathbb {E} \left[ \left(f _ {\text {a v g}} (\boldsymbol {r}) - 0\right) ^ {2} \mid \omega = 0 \right] \\ \leq p \mathbb {E} \left[ 1 - f _ {\text {a v g}} (\boldsymbol {r}) \mid \omega = 1 \right] + (1 - p) \mathbb {E} \left[ f _ {\text {a v g}} (\boldsymbol {r}) \mid \omega = 0 \right] \\ = p \mathbb {E} \left[ 1 - \frac {1}{n} \sum_ {i = 1} ^ {n} r _ {i} \mid \omega = 1 \right] + (1 - p) \mathbb {E} \left[ \frac {1}{n} \sum_ {i = 1} ^ {n} r _ {i} \mid \omega = 0 \right] \\ = p \mu_ {1} + (1 - p) \mu_ {0} \\ <   \frac {\varepsilon}{2} + \frac {\varepsilon}{2} = \varepsilon , \\ \end{array}
$$
which implies that  $f_{avg}$  is  $\varepsilon$ -optimal.
The following lemma says that  $O\left(\frac{1}{\varepsilon} \log \frac{1}{\delta}\right)$  samples are sufficient to tell whether the mean of a random variable is below  $\varepsilon$  or above  $\frac{\varepsilon}{2}$ :
Lemma G.3. Given  $T = \frac{40}{\varepsilon} \log \frac{2}{\delta}$  i.i.d. samples  $X^{(1)}, \ldots, X^{(T)}$  of a random variable  $X \in [0,1]$  with unknown mean  $\mathbb{E}[X] = \mu$ , with probability at least  $1 - \delta$  we can tell whether  $\mu < \varepsilon$  or  $\mu \geq \frac{\varepsilon}{2}$ . This can be done by checking whether the empirical mean  $\hat{\mu} = \frac{1}{T} \sum_{t=1}^{T} X^{(t)}$  is  $< \frac{3}{4}\varepsilon$  or  $\geq \frac{3}{4}\varepsilon$ .
Proof. If  $\mu \geq \varepsilon$ , using the multiplicative version of Chernoff bound we have
$$
\operatorname * {P r} \left[ \hat {\mu} <   \frac {3}{4} \varepsilon \right] \leq \operatorname * {P r} \left[ \hat {\mu} <   \frac {3}{4} \mu \right] \leq e ^ {- \frac {(\frac {1}{4}) ^ {2} \mu T}{2}} \leq e ^ {- \frac {\varepsilon T}{3 2}} \leq \delta .
$$
Namely, with probability at least  $1 - \delta$ , it holds that
$$
\hat {\mu} \geq \frac {3}{4} \varepsilon .
$$
If  $\mu < \varepsilon$ , then using the additive version of Chernoff-Hoeffding theorem, we have
$$
\Pr \left[ \hat {\mu} <   \mu - \frac {\varepsilon}{4} \right] \leq e ^ {- D (\mu - \frac {\varepsilon}{4} | | \mu) T},
$$
$$
\Pr \left[ \hat {\mu} > \mu + \frac {\varepsilon}{4} \right] \leq e ^ {- D (\mu + \frac {\varepsilon}{4} | | \mu) T},
$$
where  $D(x\| y) = x\ln \frac{x}{y} + (1 - x)\ln \frac{1 - x}{1 - y}$ . Using the inequality  $D(x\| y) \geq \frac{(x - y)^2}{2y}$  for  $x \leq y$  and  $D(x\| y) \geq \frac{(x - y)^2}{2x}$  for  $x \geq y$ , we obtain:
$$
\Pr \left[ \hat {\mu} <   \mu - \frac {\varepsilon}{4} \right] \leq e ^ {- \frac {(\frac {\varepsilon}{4}) ^ {2}}{2 \mu} T} \leq e ^ {- \frac {(\frac {\varepsilon}{4}) ^ {2}}{2 \varepsilon} T} = e ^ {- \frac {\varepsilon}{3 2} T} \leq \frac {\delta}{2},
$$
$$
\operatorname * {P r} \left[ \hat {\mu} > \mu + \frac {\varepsilon}{4} \right] \leq e ^ {- \frac {(\frac {\varepsilon}{4}) ^ {2}}{2 (\mu + \frac {\varepsilon}{4})} T} \leq e ^ {- \frac {(\frac {\varepsilon}{4}) ^ {2}}{2 (\frac {5 \varepsilon}{4})} T} = e ^ {- \frac {\varepsilon}{4 0} T} \leq \frac {\delta}{2}.
$$
By a union bound, with probability at least  $1 - \delta$ , we have
$$
| \hat {\mu} - \mu | \leq \frac {\varepsilon}{4}.
$$
Combining the case of  $\mu \geq \varepsilon$  and  $\mu < \varepsilon$ , we conclude that: with probability at least  $1 - \delta$ ,
- If  $\hat{\mu} < \frac{3}{4}\varepsilon$ , then we must have  $\mu < \varepsilon$ .
- If  $\hat{\mu} \geq \frac{3}{4}\varepsilon$ , then we have  $\mu \geq \varepsilon$  or  $\varepsilon > \mu \geq \hat{\mu} - \frac{\varepsilon}{4} \geq \frac{\varepsilon}{2}$ . In either case, we have  $\mu \geq \frac{\varepsilon}{2}$ .
The last lemma we will use shows how to estimate the unknown value of  $\rho = \frac{p}{1 - p} = \frac{P(\omega = 1)}{P(\omega = 0)}$  with accuracy  $\Delta$  using  $T = O\left(\frac{1}{n\Delta^2}\log \frac{1}{\delta}\right)$  samples. Notice that, if one simply uses the empirical value  $\hat{\rho} = \frac{\sum_{t = 1}^{T}\mathbb{1}\{\omega^{(t)} = 1\}}{\sum_{t = 1}^{T}\mathbb{1}\{\omega^{(t)} = 0\}}$  to estimate  $\rho$ , then by Chernoff bound this needs  $T = O\left(\frac{1}{\Delta^2}\log \frac{1}{\delta}\right)$  samples, which is larger than what we claim by a factor of  $n$ . This sub-optimality is because one only uses the  $\omega^{(t)}$ 's in the samples to estimate  $\rho$ , wasting the reports  $r_i^{(t)}$ 's. By using  $r_i^{(t)}$ 's to estimate  $\rho$ , we can reduce the number of samples by a factor of  $n$ . The basic idea is the following: According to Fact G.1, we have  $\rho = \frac{p}{1 - p} = \frac{\mu_0}{\mu_1} = \frac{\mathbb{E}[\sum_{i = 1}^{n}r_i|\omega = 0]}{\mathbb{E}[\sum_{i = 1}^{n}(1 - r_i)|\omega = 1]}$ . The numerator  $\mathbb{E}[\sum_{i = 1}^{n}r_i|\omega = 0]$  and the denominator  $\mathbb{E}[\sum_{i = 1}^{n}(1 - r_i)|\omega = 1]$  can be estimated from samples of  $r_i^{(t)}$ 's where  $\omega^{(t)} = 0$  and  $1$  respectively. The total number of  $r_i^{(t)}$ 's is  $Tn$ , because we have  $n$  experts per sample. This reduces the needed number of samples by a factor of  $n$ . Formally:
Lemma G.4. For conditionally independent distribution  $P$ , we can estimate  $\rho = \frac{P(\omega = 1)}{P(\omega = 0)}$  with accuracy  $\frac{|\hat{\rho} - \rho|}{\rho} \leq \Delta < 1$  (equivalently,  $\frac{\hat{\rho}}{\rho} \in 1 \pm \Delta$ ) with probability at least  $1 - \delta$  using
$$
T = O \left(\frac {1}{(1 - p) \mu_ {0} \cdot n \cdot \Delta^ {2}} \log \frac {1}{\delta} + \frac {1}{\min  \{p , 1 - p \}} \log \frac {1}{\delta}\right)
$$
samples of  $(\omega^{(t)},\pmb{r}^{(t)})'$ , by letting  $\hat{\rho} = \frac{\frac{1}{\#_0}\sum_{t:\omega^{(t)} = 0}\sum_{i = 1}^n r_i^{(t)}}{\frac{1}{\#_1}\sum_{t:\omega^{(t)} = 1}\sum_{i = 1}^n (1 - r_i^{(t)})}$ , where  $\#_0$  and  $\#_1$  are the numbers of samples with  $\omega^{(t)} = 0$  and  $1$  respectively.
Proof. Recall that  $p = P(\omega = 1)$ ,  $1 - p = P(\omega = 0)$ , and  $\rho = \frac{p}{1 - p}$ . According to Fact G.1 (which says  $(1 - p)\mu_0 = p\mu_1$ ), we have
$$
\rho = \frac {p}{1 - p} = \frac {\mu_ {0}}{\mu_ {1}} = \frac {\sum_ {i = 1} ^ {n} \mathbb {E} \left[ r _ {i} \mid \omega = 0 \right]}{\sum_ {i = 1} ^ {n} \mathbb {E} \left[ 1 - r _ {i} \mid \omega = 1 \right]}. \tag {44}
$$
Consider the following way of estimating  $\rho$  from  $T$  samples  $(\omega^{(t)},\boldsymbol{r}^{(t)})_{t = 1}^{T}$ : Let  $\#_0,\#_1$  be the numbers of samples where  $\omega^{(t)} = 0,1$ , respectively:
$$
\# _ {0} = \sum_ {t = 1} ^ {T} \mathbb {1} \{\omega^ {(t)} = 0 \}, \quad \# _ {1} = \sum_ {t = 1} ^ {T} \mathbb {1} \{\omega^ {(t)} = 1 \}.
$$
We let
$$
\hat {\rho} = \frac {\frac {1}{\# _ {0}} \sum_ {t : \omega^ {(t)} = 0} \sum_ {i = 1} ^ {n} r _ {i} ^ {(t)}}{\frac {1}{\# _ {1}} \sum_ {t : \omega^ {(t)} = 1} \sum_ {i = 1} ^ {n} \left(1 - r _ {i} ^ {(t)}\right)}. \tag {45}
$$
Now, we compare the  $\hat{\rho}$  in (45) and the  $\rho$  in (44): we see that  $\frac{1}{\#_0} \sum_{t: \omega^{(t)} = 0} \sum_{i = 1}^n r_i^{(t)}$  is an (unbiased) estimate of the numerator  $\sum_{i = 1}^n \mathbb{E}[r_i \mid \omega = 0] = n\mu_0$  and that  $\frac{1}{\#_1} \sum_{t: \omega^{(t)} = 1} \sum_{i = 1}^n (1 - r_i^{(t)})$  is an (unbiased) estimate of the denominator  $\sum_{i = 1}^n \mathbb{E}[1 - r_i \mid \omega = 1] = n\mu_1$ . We use Chernoff bounds to argue that the accuracy of the two estimates is within  $\Delta$  with high probability if  $\#_0$  and  $\#_1$  are big enough. Suppose that, when drawing the  $T$  samples, we draw all the  $\omega^{(t)}$ 's first (and hence  $\#_0$ ,  $\#_1$  are determined), and then draw all the  $r_i^{(t)}$ 's. After all the  $\omega^{(t)}$ 's are drawn, the  $r_i^{(t)}$ 's become independent, because the signals  $s_i^{(t)}$ 's are conditionally independent given  $\omega^{(t)}$ . Therefore, we can use Chernoff bounds:
$$
\operatorname * {P r} \left[ \left| \frac {1}{\# _ {0}} \sum_ {t: \omega^ {(t)} = 0} \sum_ {i = 1} ^ {n} r _ {i} ^ {(t)} - n \mu_ {0} \right| > \Delta n \mu_ {0} \right] \leq 2 e ^ {- \frac {\# _ {0} n \mu_ {0} \Delta^ {2}}{3}},
$$
$$
\operatorname * {P r} \left[ \left| \frac {1}{\# _ {1}} \sum_ {t: \omega^ {(t)} = 1} \sum_ {i = 1} ^ {n} (1 - r _ {i} ^ {(t)}) - n \mu_ {1} \right| > \Delta n \mu_ {1} \right] \leq 2 e ^ {- \frac {\# _ {1} n \mu_ {1} \Delta^ {2}}{3}}.
$$
Requiring  $\delta \geq 2e^{-\frac{\#_0n\mu_0\Delta^2}{3}}$  and  $\delta \geq 2e^{-\frac{\#_1n\mu_1\Delta^2}{3}}$ , namely,