SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
\# _ {0} \geq \frac {3}{n \mu_ {0} \Delta^ {2}} \log \frac {2}{\delta}, \quad \# _ {1} \geq \frac {3}{n \mu_ {1} \Delta^ {2}} \log \frac {2}{\delta}, \tag {46}
$$
we have, with probability at least  $1 - 2\delta$ , both of the following hold:
$$
\frac {1}{\# _ {0}} \sum_ {t: \omega^ {(t)} = 0} \sum_ {i = 1} ^ {n} r _ {i} ^ {(t)} \in (1 \pm \Delta) n \mu_ {0}, \quad \frac {1}{\# _ {1}} \sum_ {t: \omega^ {(t)} = 1} \sum_ {i = 1} ^ {n} \left(1 - r _ {i} ^ {(t)}\right) \in (1 \pm \Delta) n \mu_ {1}, \tag {47}
$$
Then, we argue that (46) can be satisfied with high probability if  $T$  is large enough. This is again done by a Chernoff bound: since  $\mathbb{E}[\#_j] = \mathbb{E}[\sum_{t=1}^T \mathbb{1}\{\omega^{(t)} = j\}] = T \cdot P(\omega = j)$ , for  $j = 0, 1$ , we have
$$
\Pr \left[ \left| \# _ {0} - T (1 - p) \right| \geq \frac {1}{2} T (1 - p) \right] \leq 2 e ^ {- \frac {T (1 - p) (\frac {1}{2}) ^ {2}}{3}}, \quad \Pr \left[ \left| \# _ {1} - T p \right| \geq \frac {1}{2} T p \right] \leq 2 e ^ {- \frac {T p (\frac {1}{2}) ^ {2}}{3}}. \tag {48}
$$
So, if we are given
$$
T \geq \frac {1 2}{\min  \{p , 1 - p \}} \log \frac {2}{\delta} \tag {49}
$$
samples, then we can ensure that with probability at least  $1 - 2\delta$ , it holds  $\#_0 \geq \frac{1}{2} T(1 - p)$  and  $\#_1 \geq \frac{1}{2} Tp$ . Then, in order for (46) to be satisfied, we can let
$$
\frac {1}{2} T (1 - p) \geq \frac {3}{n \mu_ {0} \Delta^ {2}} \log \frac {2}{\delta}, \quad \frac {1}{2} T p \geq \frac {3}{n \mu_ {1} \Delta^ {2}} \log \frac {2}{\delta}.
$$
$$
T \geq \max  \left\{\frac {6}{(1 - p) \mu_ {0} \cdot n \Delta^ {2}} \log \frac {2}{\delta}, \frac {6}{p \mu_ {1} \cdot n \Delta^ {2}} \log \frac {2}{\delta} \right\} \stackrel {(1 - p) \mu_ {0} = p \mu_ {1}} {=} \frac {6}{(1 - p) \mu_ {0} \cdot n \Delta^ {2}} \log \frac {2}{\delta}. \tag {50}
$$
Both (49) and (50) are satisfied when
$$
T \geq \frac {6}{(1 - p) \mu_ {0} \cdot n \Delta^ {2}} \log \frac {2}{\delta} + \frac {1 2}{\min  \{p , 1 - p \}} \log \frac {2}{\delta}.
$$
To conclude, if we are given  $T = \frac{6}{(1 - p)\mu_0\cdot n\Delta^2}\log \frac{2}{\delta} +\frac{12}{\min\{p,1 - p\}}\log \frac{2}{\delta}$  samples, then with probability at least  $1 - 4\delta$ , (47) holds, which implies
$$
\hat {\rho} \in \frac {(1 \pm \Delta) \mu_ {0}}{(1 \pm \Delta) \mu_ {1}} = \frac {(1 \pm \Delta)}{(1 \pm \Delta)} \rho \subseteq (1 \pm 4 \Delta) \rho \quad \Longrightarrow \quad \frac {| \hat {\rho} - \rho |}{\rho} \leq 4 \Delta ,
$$
for  $\Delta <  \frac{1}{4}$
G.1.2 The Proof
We want to show the  $O\left(\frac{1}{\varepsilon n\left(\frac{\gamma}{1 + \gamma}\right)^2}\log \frac{1}{\delta} +\frac{1}{\varepsilon}\log \frac{1}{\delta}\right)$  sample complexity upper bound for the case where experts have  $\gamma$ -strongly informative signals.
We first use  $O\left(\frac{1}{\varepsilon}\log \frac{1}{\delta}\right)$  samples tell whether  $(1 - p)\mu_0 = p\mu_1 < \frac{\varepsilon}{2}$  or  $(1 - p)\mu_0 = p\mu_1 \geq \frac{\varepsilon}{4}$ . We note that
$$
(1 - p) \mu_ {0} = P (\omega = 0) \cdot \mathbb {E} \left[ \frac {1}{n} \sum_ {i = 1} ^ {n} r _ {i} \mid \omega = 0 \right] = \mathbb {E} \left[ \mathbb {1} \{\omega = 0 \} \cdot \frac {1}{n} \sum_ {i = 1} ^ {n} r _ {i} \right],
$$
which is the expectation of the random variable  $X = \mathbb{1}\{\omega = 0\} \cdot \frac{1}{n}\sum_{i=1}^{n}r_i$ . So, according to Lemma G.3, we can tell whether  $(1 - p)\mu_0 < \frac{\varepsilon}{2}$  or  $\geq \frac{\varepsilon}{4}$  with probability at least  $1 - \delta$  using  $O\left(\frac{1}{\varepsilon}\log \frac{1}{\delta}\right)$  samples of  $X$ . If  $(1 - p)\mu_0 = p\mu_1 < \frac{\varepsilon}{2}$ , then according to Lemma G.2, the averaging aggregator  $f_{avg}(\boldsymbol{r}) = \frac{1}{n}\sum_{i=1}^{n}r_i$  is  $\varepsilon$ -optimal. We hence obtained an  $\varepsilon$ -optimal aggregator in this case. So, in the following proof, we assume  $(1 - p)\mu_0 = p\mu_1 \geq \frac{\varepsilon}{4}$ .
For each expert  $i$ , let  $\mathcal{S}_i^1 = \{s_i \in \mathcal{S}_i : \frac{P(s_i|\omega=1)}{P(s_i|\omega=0)} \geq 1 + \gamma\}$  be its set of  $\gamma$ -strongly informative signals that are more likely to be realized under  $\omega = 1$  than under  $\omega = 0$ . Let  $\mathcal{S}_i^0 = \mathcal{S}_i \setminus \mathcal{S}_i^1 = \{s_i \in \mathcal{S}_i : \frac{P(s_i|\omega=1)}{P(s_i|\omega=0)} \leq \frac{1}{1+\gamma}\}$  be the set of signals that are more likely to be realized under  $\omega = 0$ . Since  $\frac{r_i}{1-r_i} = \frac{P(s_i|\omega=1)}{P(s_i|\omega=0)}\rho$  by Equation (1), whenever an expert receives a signal in  $\mathcal{S}_i^1$ , its report satisfies
$$
\frac {r _ {i}}{1 - r _ {i}} \geq (1 + \gamma) \rho , \quad \forall s _ {i} \in \mathcal {S} _ {i} ^ {1}; \tag {51}
$$
and whenever it receives a signal in  $S_{i}^{0}$ , its report satisfies
$$
\frac {r _ {i}}{1 - r _ {i}} \leq \frac {1}{1 + \gamma} \rho , \quad \forall s _ {i} \in \mathcal {S} _ {i} ^ {0}. \tag {52}
$$
We will use the notation  $P(\mathcal{S}_i^u \mid \omega) = P(s_i \in \mathcal{S}_i^u \mid \omega) = \sum_{s_i \in \mathcal{S}_i^u} P(s_i \mid \omega)$ , for  $u \in \{0,1\}$ . Given a set of  $n$  signals  $s_1, \ldots, s_n$ , one per expert, we let  $X^1 = \sum_{i=1}^{n} \mathbb{1}\{s_i \in \mathcal{S}_i^1\}$  be the total number of signals that belong to the  $\mathcal{S}_i^1$  sets, and similarly let  $X^0 = \sum_{i=1}^{n} \mathbb{1}\{s_i \in \mathcal{S}_i^0\}$ . We have  $X^0 + X^1 = n$ , and by definition,
$$
\mathbb {E} \left[ X ^ {1} \mid \omega = 1 \right] = \sum_ {i = 1} ^ {n} P \left(\mathcal {S} _ {i} ^ {1} \mid \omega = 1\right) \geq (1 + \gamma) P \left(\mathcal {S} _ {i} ^ {1} \mid \omega = 0\right) = (1 + \gamma) \mathbb {E} \left[ X ^ {1} \mid \omega = 0 \right]. \tag {53}
$$
$$
\mathbb {E} \left[ X ^ {0} \mid \omega = 0 \right] = \sum_ {i = 1} ^ {n} P \left(\mathcal {S} _ {i} ^ {0} \mid \omega = 0\right) \geq (1 + \gamma) P \left(\mathcal {S} _ {i} ^ {0} \mid \omega = 1\right) = (1 + \gamma) \mathbb {E} \left[ X ^ {0} \mid \omega = 1 \right], \tag {54}
$$
Claim G.5. At least one of  $\mathbb{E}[X^1\mid \omega = 1]$  and  $\mathbb{E}[X^0\mid \omega = 0]$  is  $\geq \frac{n}{2}$ .
Proof. Suppose on the contrary both  $\mathbb{E}[X^1\mid \omega = 1]$  and  $\mathbb{E}[X^0\mid \omega = 0]$  are  $< \frac{n}{2}$ . Then, from (54) we have
$$
\mathbb {E} [ X ^ {0} \mid \omega = 1 ] \leq \frac {1}{1 + \gamma} \mathbb {E} [ X ^ {0} \mid \omega = 0 ] <   \frac {n}{2}.
$$
This implies  $n = \mathbb{E}[X^0 + X^1 \mid \omega = 1] < \frac{n}{2} + \frac{n}{2} = n$ , a contradiction.
Let  $u\in \{0,1\}$  be an index such that
$$
\mathbb {E} \left[ X ^ {u} \mid \omega = u \right] \geq \frac {n}{4}. \tag {55}
$$
Claim G.5 guarantees that such a  $u$  exists. We construct a "hypothetical" aggregator  $f_{\mathrm{hypo}}$  that, having access to  $\rho$  and  $\mathbb{E}[X^u \mid \omega]$ , predicts whether  $\omega = 0$  or 1 by counting the number  $X^u$  of signals that belong to the  $S_i^u$  sets and comparing it with its expectations under  $\omega = 0$  and 1,  $\mathbb{E}[X^u \mid \omega = 0]$  and  $\mathbb{E}[X^u \mid \omega = 1]$ , respectively. Specifically, given reports  $r = (r_1, \dots, r_n)$  as input, with corresponding unobserved signals  $s = (s_1, \dots, s_n)$ ,  $f_{\mathrm{hypo}}$  does the following:
(1) If  $u = 1$ , count how many reports  $r_i$ 's satisfy  $\frac{r_i}{1 - r_i} \geq (1 + \gamma)\rho$ ; If  $u = 0$ , count how many reports  $r_i$ 's satisfy  $\frac{r_i}{1 - r_i} \leq \frac{1}{1 + \gamma}\rho$ . According to (51) and (52), this number is exactly equal to the number of signals that belong to the  $S_i^u$  sets,  $X^u$ .
(2) Then, check whether  $X^u$  is closer (in terms of absolute difference) to  $\mathbb{E}[X^u \mid \omega = u]$  or  $\mathbb{E}[X^u \mid \omega = 1 - u]$ . If  $X^u$  is closer to  $\mathbb{E}[X^u \mid \omega = u]$ , output  $f_{\mathrm{hypo}}(\boldsymbol{r}) = u$ ; otherwise, output  $f_{\mathrm{hypo}}(\boldsymbol{r}) = 1 - u$ .
We claim that  $f_{\mathrm{hypo}}$  is  $\varepsilon$ -optimal.
Claim G.6. Given  $\frac{\gamma}{1 + \gamma} \geq 8\sqrt{\frac{2}{n}\log\frac{2}{\varepsilon}}$  and  $\mathbb{E}[X^u \mid \omega = u] \geq \frac{n}{4}$ ,  $f_{\mathrm{hypo}}$  is  $\varepsilon$ -optimal.
Proof. Given either  $\omega = 0$  or  $1$ , consider the conditional random draw of signals  $s_1, \ldots, s_n$ . Because  $X^u = \sum_{i=1}^{n} \mathbb{1}\{s_i \in S_i^u\}$  and the random variables  $\mathbb{1}\{s_i \in S_i^u\}$ ,  $i = 1, \ldots, n$ , are  $[0,1]$ -bounded and independent conditioning  $\omega$ , by Hoeffding's inequality we have
$$
\Pr \left[ \left| X ^ {u} - \mathbb {E} \left[ X ^ {u} \mid \omega \right] \right| \geq a \mid \omega \right] \leq 2 e ^ {- \frac {2 a ^ {2}}{n}}.
$$
$$
a = \sqrt {\frac {n}{2} \log \frac {2}{\varepsilon}}. \tag {56}
$$
Then with probability at least  $1 - 2e^{-\frac{2a^2}{n}} = 1 - \varepsilon$ , it holds
$$
\left| X ^ {u} - \mathbb {E} \left[ X ^ {u} \mid \omega \right] \right| <   a. \tag {57}
$$
Consider the difference between  $\mathbb{E}[X^u\mid \omega = u]$  and  $\mathbb{E}[X^u\mid \omega = 1 - u]$ . By (53) and (54), we have
$$
\mathbb {E} \left[ X ^ {u} \mid \omega = 1 - u \right] \leq \frac {1}{1 + \gamma} \mathbb {E} \left[ X ^ {u} \mid \omega = u \right] = \left(1 - \frac {\gamma}{1 + \gamma}\right) \mathbb {E} \left[ X ^ {u} \mid \omega = u \right].
$$
By the assumption  $\mathbb{E}[X^u\mid \omega = u]\geq \frac{n}{4}$
$$
\mathbb {E} [ X ^ {u} \mid \omega = 1 - u ] \leq \mathbb {E} [ X ^ {u} \mid \omega = u ] - \frac {\gamma}{1 + \gamma} \cdot \frac {n}{4}.
$$
By the assumption  $\frac{\gamma}{1 + \gamma} \geq 8\sqrt{\frac{2}{n}\log\frac{2}{\varepsilon}}$ , we have  $\frac{\gamma}{1 + \gamma} \cdot \frac{n}{4} \geq 8\sqrt{\frac{2}{n}\log\frac{2}{\varepsilon}} \cdot \frac{n}{4} = 4\sqrt{\frac{n}{2}\log\frac{2}{\varepsilon}} = 4a$ . Therefore
$$
\mathbb {E} \left[ X ^ {u} \mid \omega = u \right] - \mathbb {E} \left[ X ^ {u} \mid \omega = 1 - u \right] \geq 4 a. \tag {58}
$$
Because we already had  $\left|X^u - \mathbb{E}[X^u \mid \omega]\right| < a$  (which happened with probability at least  $1 - \varepsilon$ ), if  $X^u$  turns out to be closer to  $\mathbb{E}[X^u \mid \omega = u]$  it must be that  $\omega = u$ ; if  $X^u$  turns out to be closer to  $\mathbb{E}[X^u \mid \omega = 1 - u]$  it must be that  $\omega = 1 - u$ . In either case, our output  $f_{\mathrm{hypo}}(r)$  is equal to
$\omega$ , having a loss 0. If  $\left|X^u - \mathbb{E}[X^u \mid \omega]\right| < a$  did not happen, our loss is at most 1. Therefore, the expected loss of our aggregator  $f_{\mathrm{hypo}}$  is at most
$$
L _ {P} (f _ {\mathrm {h y p o}}) = \mathbb {E} _ {\omega} \left[ \mathbb {E} \left[ | f _ {\mathrm {h y p o}} (\boldsymbol {r}) - \omega | ^ {2} \mid \omega \right] \right] \leq \mathbb {E} _ {\omega} \left[ (1 - \varepsilon) \cdot 0 + \varepsilon \cdot 1 \right] = \varepsilon .
$$
Since the expected loss of the optimal aggregator  $f^{*}$  is non-negative,  $f_{\mathrm{hypo}}$  is  $\varepsilon$ -optimal.
In the remaining proof, we show how to use samples to learn a "real" aggregator  $\hat{f}$  that implements the same functionality as the hypothetical aggregator  $f_{\mathrm{hypo}}$  and hence is  $\varepsilon$ -optimal. We have two learning tasks: First, we need to estimate  $\rho$ , so that we can implement the step (1) of  $f_{\mathrm{hypo}}$  which tells apart  $\frac{r_i}{1 - r_i} \geq (1 + \gamma)\rho$  and  $\frac{r_i}{1 - r_i} \leq \frac{1}{1 + \gamma}\rho$ . Second, we need to find an index  $u \in \{0,1\}$  such that  $\mathbb{E}[X^u \mid \omega = u] \geq \frac{n}{4}$  and estimate  $\mathbb{E}[X^u \mid \omega = u]$ , so that we can implement the step (2) of  $f_{\mathrm{hypo}}$  which tells whether  $X^u$  is closer to  $\mathbb{E}[X^u \mid \omega = u]$  or  $\mathbb{E}[X^u \mid \omega = 1 - u]$ . We show that these two tasks can be achieved using  $O\left(\frac{1}{\varepsilon n\left(\frac{\gamma}{1 + \gamma}\right)^2} \log \frac{1}{\delta} + \frac{1}{\varepsilon} \log \frac{1}{\delta}\right)$  samples, with probability at least  $1 - O(\delta)$ .
Task 1: estimate  $\rho$ , using  $T_{1} = O\left(\frac{1}{\varepsilon n\left(\frac{\gamma}{1 + \gamma}\right)^{2}}\log \frac{1}{\delta} + \frac{1}{\varepsilon}\log \frac{1}{\delta}\right)$  samples. We want to use samples to obtain an estimate  $\hat{\rho}$  of  $\rho$  such that  $\frac{1}{1 + \gamma}\rho < \hat{\rho} < (1 + \gamma)\rho$ . So, by checking whether  $\frac{r_i}{1 - r_i} > \hat{\rho}$  or  $\frac{r_i}{1 - r_i} < \hat{\rho}$  we can tell apart  $\frac{r_i}{1 - r_i} \geq (1 + \gamma)\rho$  and  $\frac{r_i}{1 - r_i} \leq \frac{1}{1 + \gamma}\rho$ . Using Lemma G.4 with  $\Delta = \frac{\gamma}{1 + \gamma}$ , we obtain a  $\hat{\rho}$  such that
$$
\hat {\rho} \in (1 \pm \Delta) \rho ,
$$
with probability at least  $1 - \delta$  using
$$
T _ {1} = O \left(\frac {1}{(1 - p) \mu_ {0} n \Delta^ {2}} \log \frac {1}{\delta} + \frac {1}{\min  \{p , 1 - p \}} \log \frac {1}{\delta}\right) \leq O \left(\frac {1}{\varepsilon n (\frac {\gamma}{1 + \gamma}) ^ {2}} \log \frac {1}{\delta} + \frac {1}{\varepsilon} \log \frac {1}{\delta}\right)
$$
samples (recall that we have  $\min \{p, 1 - p\} \geq (1 - p)\mu_0 = p\mu_1 \geq \frac{\varepsilon}{4}$ ). The  $\hat{\rho}$  then satisfies
$$
\hat {\rho} <   \left(1 + \frac {\gamma}{1 + \gamma}\right) \rho <   (1 + \gamma) \rho \quad \text {a n d} \quad \hat {\rho} > \left(1 - \frac {\gamma}{1 + \gamma}\right) \rho = \frac {1}{1 + \gamma} \rho ,
$$
Task 2: find  $u$  such that  $\mathbb{E}[X^u \mid \omega = u] \geq \frac{n}{4}$  and estimate  $\mathbb{E}[X^u \mid \omega = u]$ , using  $T_2 = O\left(\frac{1}{\varepsilon} \log \frac{1}{\delta}\right)$  samples. First, we show how to use  $T_2 = O\left(\frac{1}{\varepsilon} \log \frac{1}{\delta}\right)$  samples to estimate both  $\mathbb{E}[X^0 \mid \omega = 0]$  and  $\mathbb{E}[X^1 \mid \omega = 1]$  with accuracy  $a = \sqrt{\frac{n}{2} \log \frac{2}{\varepsilon}}$ . By the same argument as in the proof of Lemma G.4 (Equations 48 and 49), we know that with probability at least  $1 - 2\delta$  over the random draws of
$$
T _ {2} \geq \frac {1 2}{\min  \{p , 1 - p \}} \log \frac {2}{\delta} \tag {59}
$$
samples, the numbers of samples  $(\omega^{(t)},r_1^{(t)},\ldots ,r_n^{(t)})$  's where  $\omega^{(t)} = 0$  and  $\omega^{(t)} = 1$ , denoted by  $\#_0$  and  $\#_{1}$ , must satisfy
$$
\# _ {0} \geq \frac {1}{2} (1 - p) T _ {2}, \quad \# _ {1} \geq \frac {1}{2} p T _ {2}.
$$
We consider the samples where  $\omega^{(t)} = 0$ . There are  $\#_0 n$  total numbers of  $r_i^{(t)}$ 's. Suppose we have accomplished Task 1. Then, for each  $r_i^{(t)}$ , we can tell whether the corresponding signal  $s_i^{(t)}$  belongs to  $\mathcal{S}_i^0$  by checking whether  $\frac{r_i^{(t)}}{1 - r_i^{(t)}} < \hat{\rho}$ . So, we can exactly compute the total number of such signals in the  $t$ -th sample,  $X^{0(t)} = \sum_{i=1}^{n} \mathbb{1}\{s_i^{(t)} \in \mathcal{S}_i^0\}$ , whose expected value is  $\mathbb{E}[X^0 \mid \omega = 0]$ . Because signals are independent given  $\omega^{(t)} = 0$ , by Hoeffding's inequality we have
$$
\operatorname * {P r} \left[ \Big | \sum_ {t: \omega^ {(t)} = 0} \underbrace {\sum_ {i = 1} ^ {n} \mathbb {1} \{s _ {i} ^ {(t)} \in \mathcal {S} _ {i} ^ {0} \}} _ {X ^ {0 (t)}} - \# _ {0} \mathbb {E} [ X ^ {0} \mid \omega = 0 ] \Big | \geq \# _ {0} a \right] \leq 2 e ^ {- \frac {2 (\# _ {0} a) ^ {2}}{\# _ {0} n}} = 2 e ^ {- \frac {2 \# _ {0} a ^ {2}}{n}}.
$$
Plugging in  $a = \sqrt{\frac{n}{2}\log\frac{2}{\varepsilon}}$  and  $\#_0 \geq \frac{1}{2}(1 - p)T_2$ , we get
$$
\operatorname * {P r} \left[ \left| \frac {1}{\# _ {0}} \sum_ {t: \omega^ {(t)} = 0} X ^ {0 (t)} - \mathbb {E} [ X ^ {0} \mid \omega = 0 ] \right| \geq a \right] \leq 2 e ^ {- \# _ {0} \log \frac {2}{\varepsilon}} \leq 2 e ^ {- \frac {1}{2} (1 - p) T _ {2} \log \frac {2}{\varepsilon}}.
$$
Similarly, considering the samples where  $\omega^{(t)} = 1$ , we get
$$
\operatorname * {P r} \left[ \left| \frac {1}{\# _ {1}} \sum_ {t: \omega^ {(t)} = 1} X ^ {1 (t)} - \mathbb {E} [ X ^ {1} \mid \omega = 1 ] \right| \geq a \right] \leq 2 e ^ {- \# _ {1} \log \frac {2}{\varepsilon}} \leq 2 e ^ {- \frac {1}{2} p T _ {2} \log \frac {2}{\varepsilon}}.
$$
Therefore, if we require
$$
T _ {2} \geq \frac {2 \log (2 / \delta)}{\min  \{p , 1 - p \} \log (2 / \varepsilon)}, \tag {60}
$$
then with probability at least  $1 - 2\delta$  ,both
$$
\left| \frac {1}{\# _ {0}} \sum_ {t: \omega^ {(t)} = 0} X ^ {0 (t)} - \mathbb {E} [ X ^ {0} \mid \omega = 0 ] \right| <   a, \quad \left| \frac {1}{\# _ {1}} \sum_ {t: \omega^ {(t)} = 1} X ^ {1 (t)} - \mathbb {E} [ X ^ {1} \mid \omega = 1 ] \right| <   a
$$
hold. Namely,  $\frac{1}{\#_0} \sum_{t: \omega^{(t)} = 0} X^{0(t)}$  and  $\frac{1}{\#_1} \sum_{t: \omega^{(t)} = 1} X^{1(t)}$  are  $a$ -accurate estimates of  $\mathbb{E}[X^0 \mid \omega = 0]$  and  $\mathbb{E}[X^1 \mid \omega = 1]$ . Equations (59) and (60) together imply that  $T_2 = O\left(\frac{1}{\min\{p, 1 - p\}} \log \frac{2}{\delta}\right) \leq O\left(\frac{1}{\varepsilon} \log \frac{1}{\delta}\right)$  samples suffice.
Then, we identify an index  $u \in \{0,1\}$  such that  $\mathbb{E}[X^u \mid \omega = u] \geq \frac{n}{4}$ . By Claim G.5, there exists a  $v \in \{0,1\}$  with  $\mathbb{E}[X^v \mid \omega = v] \geq \frac{n}{2}$ . Since  $\frac{1}{\#v} \sum_{t: \omega^{(t)} = v} X^{v(t)}$  is an  $a$ -accurate estimate of  $\mathbb{E}[X^v \mid \omega = v]$ , we must have
$$
\frac {1}{\# _ {v}} \sum_ {t: \omega^ {(t)} = v} X ^ {v (t)} \geq \mathbb {E} [ X ^ {v} \mid \omega = v ] - a \geq \frac {n}{2} - a.
$$
So, at least one of  $u \in \{0,1\}$  must satisfy  $\frac{1}{\#_u} \sum_{t: \omega^{(t)} = u} X^{u(t)} \geq \frac{n}{2} - a$ . By picking any such a  $u$ , we are guaranteed that  $\mathbb{E}[X^u \mid \omega = u] \geq \frac{1}{\#_u} \sum_{t: \omega^{(t)} = u} X^{u(t)} - a \geq \frac{n}{2} - 2a$ . Given the assumption  $n \geq 32 \log \frac{2}{\varepsilon}$  in the statement of the theorem, we have
$$
\frac {a}{n} = \sqrt {\frac {1}{2 n} \log \frac {2}{\varepsilon}} \leq \frac {1}{8}.
$$
Hence,  $\mathbb{E}[X^u\mid \omega = u]\geq \frac{n}{2} -2a\geq \frac{n}{2} -2(\frac{n}{8}) = \frac{n}{4}$
Finally, as argued above, an  $a$ -accurate estimate of  $\mathbb{E}[X^u \mid \omega = u]$  is given by  $\frac{1}{\#_u} \sum_{t: \omega^{(t)} = u} X^{u(t)}$ .
Constructing  $\hat{f}$ . After accomplishing Tasks 1 and 2 using  $T_{1} + T_{2} = O\left(\frac{1}{\varepsilon n\left(\frac{\gamma}{1 + \gamma}\right)^{2}}\log \frac{1}{\delta} +\frac{1}{\varepsilon}\log \frac{1}{\delta}\right)$  samples, we construct a  $\hat{f}$  that implements the same functionality as  $f_{\mathrm{hypo}}$ . Let
$$
M = \frac {1}{\# _ {u}} \sum_ {t: \omega^ {(t)} = u} X ^ {u (t)} - 2 a,
$$
where  $\frac{1}{\#_u}\sum_{t:\omega^{(t)} = u}X^{u(t)}$  is our estimate of  $\mathbb{E}[X^u\mid \omega = u]$  in Task 2 and  $a = \sqrt{\frac{n}{2}\log\frac{2}{\varepsilon}}$
Claim G.7.  $\mathbb{E}[X^u\mid \omega = u] - a > M > \mathbb{E}[X^u\mid \omega = 1 - u] + a.$
Proof. Because  $\frac{1}{\#_u}\sum_{t:\omega^{(t)} = u}X^{u(t)}$  is an  $a$ -accurate estimate of  $\mathbb{E}[X^u\mid \omega = u]$ , we have
$$
\mathbb {E} \left[ X ^ {u} \mid \omega = u \right] > \frac {1}{\# _ {u}} \sum_ {t: \omega^ {(t)} = u} X ^ {u (t)} - a = M + a.
$$
Recall from Equation (58) that  $\mathbb{E}[X^u\mid \omega = 1 - u]\leq \mathbb{E}[X^u\mid \omega = u] - 4a$  . So,
$$
\mathbb {E} \left[ X ^ {u} \mid \omega = 1 - u \right] <   \left(\frac {1}{\# _ {u}} \sum_ {t: \omega^ {(t)} = u} X ^ {u (t)} + a\right) - 4 a = M - a.
$$
The above two inequalities prove the claim.
Given reports  $\boldsymbol{r} = (r_1, \dots, r_n)$  as input, we let  $\hat{f}$  do the following:
(1) If  $u = 1$ , count how many reports  $r_i$ 's satisfy  $\frac{r_i}{1 - r_i} > \hat{\rho}$ ; If  $u = 0$ , count how many reports  $r_i$ 's satisfy  $\frac{r_i}{1 - r_i} < \hat{\rho}$ . Let this number be  $X$ ;
(2) Then, check whether  $X > M$  or  $X \leq M$ . If  $X > M$ , output  $\hat{f}(\boldsymbol{r}) = u$ ; otherwise, output  $\hat{f}(\boldsymbol{r}) = 1 - u$ .
We argue that  $\hat{f}$  implements the same functionality as  $f_{\mathrm{hypo}}$ : (1) In Task 1 we got  $\frac{1}{1 + \gamma}\rho < \hat{\rho} < (1 + \gamma)\rho$ . So, by checking whether  $\frac{r_i}{1 - r_i} > \hat{\rho}$  or  $< \hat{\rho}$  we can exactly tell whether  $\frac{r_i}{1 - r_i} \geq (1 + \gamma)\rho$  or  $\leq \frac{1}{1 + \gamma}\rho$ . Hence, we have  $X = X^u$ , the number of signals that belong to the  $S_i^u$  sets. (2) Recall from Equation (57) that with probability at least  $1 - \varepsilon$ ,  $X$  is a close to its expectation  $\mathbb{E}[X^u \mid \omega]$ . Then, according to Claim G.7,  $X > M$  implies that  $X$  is closer to  $\mathbb{E}[X^u \mid \omega = u]$ ;  $X < M$  implies that  $X$  is closer to  $\mathbb{E}[X^u \mid \omega = 1 - u]$ . So,  $\hat{f}$  implements both of the two steps in  $f_{\mathrm{hypo}}$ . Hence, according to Claim G.6,  $\hat{f}$  is  $\varepsilon$ -optimal.
G.2 Proof of Theorem B.3
According to Lemma 5.1, the optimal aggregator is
$$
f ^ {*} (\boldsymbol {r}) = \frac {1}{1 + \rho^ {n - 1} \prod_ {i = 1} ^ {n} \frac {1 - r _ {i}}{r _ {i}}}, \tag {61}
$$
where  $\rho = \frac{p}{1 - p}$ . We claim that an approximately optimal aggregator can be obtained by first estimating  $\rho$  from samples and then use the aggregator with the estimate  $\hat{\rho}$ :
$$
\hat {f} (\boldsymbol {r}) = \frac {1}{1 + \hat {\rho} ^ {n - 1} \prod_ {i = 1} ^ {n} \frac {1 - r _ {i}}{r _ {i}}}. \tag {62}
$$
Claim G.8. If  $\frac{|\hat{\rho} - \rho|}{\rho} \leq \frac{2\sqrt{\varepsilon}}{n - 1} < \frac{1}{2}$ , then the aggregator  $\hat{f}$  defined above is  $\varepsilon$ -optimal.
Proof. Consider the function  $g(\hat{\rho}) = \frac{1}{1 + \hat{\rho}^{n - 1}\prod_{i = 1}^{n}\frac{1 - r_i}{r_i}}$  (where  $\hat{\rho}$  is the variable and  $r_i$ 's are constants). We claim that
$$
\left| g ^ {\prime} (\hat {\rho}) \right| \leq \frac {n - 1}{4 \hat {\rho}}. \tag {63}
$$
To see this, we note that if  $r_i = 0$  for some  $i$  then  $g(\hat{\rho}) = 0$  and  $g'(\hat{\rho}) = 0$ . Otherwise, we let  $y = \prod_{i=1}^{n} \frac{1 - r_i}{r_i} < +\infty$  and take the derivative with respect to  $\hat{\rho}$ ,
$$
g ^ {\prime} (\hat {\rho}) = - \frac {1}{(1 + \hat {\rho} ^ {n - 1} y) ^ {2}} (n - 1) \hat {\rho} ^ {n - 2} y = - (n - 1) \frac {(\hat {\rho} ^ {\frac {n}{2} - 1} \sqrt {y}) ^ {2}}{(1 + \hat {\rho} ^ {n - 1} y) ^ {2}} = - (n - 1) \frac {1}{\left(\frac {1}{\hat {\rho} ^ {\frac {n}{2} - 1} \sqrt {y}} + \hat {\rho} ^ {\frac {n}{2}} \sqrt {y}\right) ^ {2}}.
$$
By the AM-GM inequality  $a + b \geq 2\sqrt{ab}$ , we get
$$
| g ^ {\prime} (\hat {\rho}) | \leq (n - 1) \frac {1}{\left(2 \sqrt {\frac {1}{\hat {\rho} ^ {\frac {n}{2} - 1} \sqrt {y}} \cdot \hat {\rho} ^ {\frac {n}{2}} \sqrt {y}}\right) ^ {2}} = (n - 1) \frac {1}{4 \hat {\rho}},
$$
Using (63), we have, for  $\hat{\rho} \geq \frac{\rho}{2}$
$$
\left| \hat {f} (\boldsymbol {r}) - f ^ {*} (\boldsymbol {r}) \right| = \left| g (\hat {\rho}) - g (\rho) \right| \leq \frac {n - 1}{4 \min  \left\{\hat {\rho} , \rho \right\}} \cdot \left| \hat {\rho} - \rho \right| \leq \frac {n - 1}{2} \cdot \frac {\left| \hat {\rho} - \rho \right|}{\rho}. \tag {64}
$$
So, to obtain  $\varepsilon$ -approximation  $\mathbb{E}\big[|\hat{f}(\boldsymbol{r}) - f^{*}(\boldsymbol{r})|^{2}\big] \leq \varepsilon$ , we can require  $|\hat{f}(\boldsymbol{r}) - f^{*}(\boldsymbol{r})| \leq \frac{n-1}{2} \cdot \frac{|\hat{\rho} - \rho|}{\rho} \leq \sqrt{\varepsilon}$ . This can be satisfied if the error in estimating  $\rho$  is at most  $\frac{|\hat{\rho} - \rho|}{\rho} \leq \frac{2\sqrt{\varepsilon}}{n-1}$ .
We then show how to use  $O\left(\frac{\gamma n}{\varepsilon} \log \frac{1}{\delta}\right)$  samples to estimate the value of  $\rho$  with  $O\left(\frac{\sqrt{\varepsilon}}{n - 1}\right)$  accuracy, which will give us an  $\varepsilon$ -optimal aggregator according to Claim G.8. Recall from (14) that when signals are  $\gamma$ -weakly informative, the reports always satisfy
$$
\frac {1}{1 + \gamma} \rho \leq \frac {r _ {i}}{1 - r _ {i}} \leq (1 + \gamma) \rho . \tag {65}
$$
The following observation is the key:
Lemma G.9. For each expert  $i$ , we have