SlowGuess/NeurIPS_2023_OCR huggingface dataset:
- We define a collection of distributions  $\{\mathcal{D}_i | i \in [n]\}$  in which  $\mathcal{D}_i$  is realized by  $21_{\{i\}} - 1$ . The data distribution  $\mathcal{D}_i$  put  $1 - 3(n - 1)\varepsilon$  on  $(0,1, +)$  and  $3\varepsilon$  on  $(i,1, -)$  for all  $i \neq i^*$ . Hence, note that all distributions in  $\{\mathcal{D}_i | i \in [n]\}$  share the same distribution support  $\{(0,1, +)\} \cup \{(i,1, -) | i \in [n]\}$ , but have different weights.
Randomization and improperness of the output  $f_{\mathrm{out}}$  do not help. Note that algorithms are allowed to output a randomized  $f_{\mathrm{out}}$  and to output  $f_{\mathrm{out}} \notin \mathcal{H}$ . We will show that randomization and improperness of  $f_{\mathrm{out}}$  don't make the problem easier. Supposing that the data distribution is  $\mathcal{D}_{i^*}$  for some  $i^* \in [n]$ , finding a (possibly randomized and improper)  $f_{\mathrm{out}}$  is not easier than identifying  $i^*$ . Since our feature space  $\mathcal{X}$  is finite, we can enumerate all hypotheses not equal to  $21_{\{i^*\}} - 1$  and calculate their strategic population loss as follows. The hypothesis  $21_{\emptyset} - 1$  will predict all by negative and thus  $\mathcal{L}^{\mathrm{str}}(21_{\emptyset} - 1) = 1 - 3(n - 1)\varepsilon$ . For any hypothesis predicting 0 by positive, it will predict all points in the distribution support by positive and thus incurs strategic loss  $3(n - 1)\varepsilon$ . For any hypothesis predicting 0 by negative and some  $i \neq i^*$  by positive, then it will misclassify  $(i,1, - )$  and incur strategic loss  $3\varepsilon$ . Therefore, for any hypothesis  $h \neq 21_{\{i^*\}} - 1$ , we have  $\mathcal{L}_{\mathcal{D}_{i^*}}^{\mathrm{str}}(h) \geq 3\varepsilon$ .
Similar to the proof of Theorem 7, under distribution  $\mathcal{D}_{i^*}$ , if we are able to find a (possibly randomized)  $f_{\mathrm{out}}$  with strategic loss  $\mathcal{L}^{\mathrm{str}}(f_{\mathrm{out}}) \leq \varepsilon$ . Then  $\operatorname{Pr}_{h \sim f_{\mathrm{out}}} (h = 2\mathbb{1}_{\{i^*\}} - 1) \geq \frac{2}{3}$ . We can identify  $i^*$  by checking which realization of  $f_{\mathrm{out}}$  has probability greater than  $\frac{2}{3}$ . In the following, we will focus on the sample complexity to identify the target function  $2\mathbb{1}_{\{i^*\}} - 1$  or simply  $i^*$ . Let  $i_{\mathrm{out}}$  denote the algorithm's answer to question of "what is  $i^*$ ?".
Smooth the data distribution For technical reasons (appearing later in the analysis), we don't want to analyze distribution  $\{\mathcal{D}_i|i\in [n]\}$  directly as the probability of  $(i,1, - )$  is 0 under distribution  $\mathcal{D}_i$ . Instead, for each  $i\in [n]$ , let  $\mathcal{D}_i' = (1 - p)\mathcal{D}_i + p\mathcal{D}_i''$  be the mixture of  $\mathcal{D}_i$  and  $\mathcal{D}_i''$  for some small  $p$ , where  $\mathcal{D}_i'' = (1 - 3(n - 1)\varepsilon)\mathbb{1}_{\{(0,1, + )\}} + 3(n - 1)\varepsilon \mathbb{1}_{\{(i,1, - )\}}$ . Specifically,
$$
\mathcal {D} _ {i} ^ {\prime} (z) = \left\{ \begin{array}{l l} 1 - 3 (n - 1) \varepsilon & \text {f o r} z = (0, 1, +) \\ 3 (1 - p) \varepsilon & \text {f o r} z = (j, 1, -, \forall j \neq i \\ 3 (n - 1) p \varepsilon & \text {f o r} z = (i, 1, -) \end{array} \right.
$$
For any data distribution  $\mathcal{D}$ , let  $\mathbf{P}_{\mathcal{D}}$  be the dynamics of  $(f_{1},y_{1},\widehat{y}_{1},\ldots ,f_{T},y_{T},\widehat{y}_{T})$  under  $\mathcal{D}$ . According to Lemma 4, by setting  $p = \frac{\varepsilon}{16n^2}$ , when  $T\leq \frac{n}{\varepsilon}$ , we have that for any  $i,j\in [n]$
$$
\left| \mathbf {P} _ {\mathcal {D} _ {i}} \left(i _ {\text {o u t}} = j\right) - \mathbf {P} _ {\mathcal {D} _ {i} ^ {\prime}} \left(i _ {\text {o u t}} = j\right) \right| \leq \frac {1}{8}. \tag {17}
$$
From now on, we only consider distribution  $\mathcal{D}_i^\prime$  instead of  $\mathcal{D}_i$ . The readers might have the question that why not using  $\mathcal{D}_i^\prime$  for construction directly. This is because no hypothesis has zero loss under  $\mathcal{D}_i^\prime$ , and thus  $\mathcal{D}_i^\prime$  does not satisfy realizability requirement.
Information gain from different choices of  $f_{t}$  Note that in each round, the learner picks a  $f_{t}$  and then only observes  $\widehat{y}_t$  and  $y_{t}$ . Here we enumerate choices of  $f_{t}$  as follows.
1.  $f_{t} = 2\mathbb{1}_{\emptyset} - 1$  predicts all points in  $\mathcal{X}$  by negative. No matter what  $i^{*}$  is, we observe  $\widehat{y}_t = -$  and  $y_{t} = 2\mathbb{1}(x_{t} = 0) - 1$ . Hence  $(\widehat{y}_t,y_t)$  are identically distributed for all  $i^{*}\in [n]$ , and thus, we cannot learn anything about  $i^{*}$  from this round.
2.  $f_{t}$  predicts 0 by positive. Then no matter what  $i^{*}$  is, we have  $\widehat{y}_t = +$  and  $y_{t} = \mathbb{1}(x_{t} = 0)$  Thus again, we cannot learn anything about  $i^{*}$
3.  $f_{t} = 2\mathbb{1}_{a_{t}} - 1$  for some non-empty  $a_{t}\subset [n]$ . For rounds with  $x_{t} = 0$ , we have  $\widehat{y}_t = y_t = +$  no matter what  $i^{*}$  is and thus, we cannot learn anything about  $i^{*}$ . For rounds with  $y_{t} = -$ , i.e.,  $x_{t}\neq 0$ , we will observe  $\widehat{y}_t = f_t(\Delta (x_t,f_t,1)) = \mathbb{1}(x_t\in a_t)$ .
Hence, we can only extract information with the third type of  $f_{t}$  at rounds with  $x_{t} \neq 0$ .
Reduction to stochastic linear bandits In rounds with  $f_{t} = 2\mathbb{1}_{a_{t}} - 1$  for some non-empty  $a_{t} \subset [n]$  and  $x_{t} \neq 0$ , our problem is identical to a stochastic linear bandit problem. Let us state our problem as Problem 1 and a linear bandit problem as Problem 2. Let  $A = \{0,1\}^{n} \setminus \{\mathbf{0}\}$ .
Problem 1. The environment picks an  $i^* \in [n]$ . At each round  $t$ , the environment picks  $x_t \in \{\mathbf{e}_i | i \in [n]\}$  with  $P(i) = \frac{1 - p}{n - 1}$  for  $i \neq i^*$  and  $P(i^*) = p$  and the learner picks an  $a_t \in A$  (where we use a  $n$ -bit string to represent  $a_t$  and  $a_{t,i} = 1$  means that  $a_t$  predicts  $i$  by positive). Then the learner observes  $\hat{y}_t = \mathbb{1}(a_t^\top x_t > 0)$  (where we use 0 to represent negative label).
Problem 2. The environment picks a linear parameter  $w^{*} \in \{w^{i} | i \in [n]\}$  with  $w^{i} = \frac{1 - p}{n - 1} \mathbf{1} - \left( \frac{1 - p}{n - 1} - p \right) \mathbf{e}_{i}$ . The arm set is  $A$ . For each arm  $a \in A$ , the reward is i.i.d. from the following distribution:
$$
r _ {w} (a) = \left\{ \begin{array}{l} - 1, w. p. w ^ {\intercal} a, \\ 0. \end{array} \right. \tag {18}
$$
If the linear parameter  $w^{*} = w^{i^{*}}$ , the optimal arm is  $\mathbf{e}_i$ .
Claim 1. For any  $\delta >0$ , for any algorithm  $\mathcal{A}$  that identify  $i^{*}$  correctly with probability  $1 - \delta$  within  $T$  rounds for any  $i^{*}\in [n]$  in Problem 1, we can construct another algorithm  $\mathcal{A}'$  can also identify the optimal arm in any environment with probability  $1 - \delta$  within  $T$  rounds in Problem 2.
This claim follows directly from the problem descriptions. Given any algorithm  $\mathcal{A}$  for Problem 1, we can construct another algorithm  $\mathcal{A}'$  which simulates  $\mathcal{A}$ . At round  $t$ , if  $\mathcal{A}$  selects predictor  $a_{t}$ , then  $\mathcal{A}'$  picks arm the same as  $a_{t}$ . Then  $\mathcal{A}'$  observes a reward  $r_{w^{i*}}(a_t)$ , which is  $-1$  w.p.  $w^{i^*\top}a_t$  and feed  $-r_{w^{i*}}(a_t)$  to  $\mathcal{A}$ . Since  $\widehat{y}_t$  in Problem 1 is 1 w.p.  $\sum_{i=1}^{n} a_{t,i} P(i) = w^{i^*\top}a_t$ , it is distributed identically as  $-r_{w^{i*}}(a_t)$ . Since  $\mathcal{A}$  will be able to identify  $i^*$  w.p.  $1 - \delta$  in  $T$  rounds,  $\mathcal{A}'$  just need to output  $\mathbf{e}_{i^*}$  as the optimal arm.
Then any lower bound on  $T$  for Problem 2 also lower bounds Problem 1. Hence, we adopt the information theoretical framework of proving lower bounds for linear bandits (Theorem 11 by Rajaraman et al. (2023)) to prove a lower bound for our problem. In fact, we also apply this framework to prove the lower bounds in other settings of this work, including Theorem 7 and Theorem 9.
Lower bound of the information For notation simplicity, for all  $i\in [n]$ , let  $\mathbf{P}_i$  denote the dynamics of  $(f_{1},y_{1},\widehat{y}_{1},\ldots ,f_{T},y_{T},\widehat{y}_{T})$  under  $\mathcal{D}_i^\prime$  and and  $\overline{\mathbf{P}}$  denote the dynamics under  $\overline{\mathcal{D}} = \frac{1}{n}\mathcal{D}_i^\prime$ . Let  $B_{t}$  denote the event of  $\{f_t = 2\mathbb{1}_{a_t} - 1$  for some non-empty  $a_{t}\subset [n]\}$ . As discussed before, for any  $a_{t}$ , conditional on  $\neg B_{t}$  or  $y_{t} = +1$ ,  $(x_{t},y_{t},\widehat{y}_{t})$  are identical in all  $\{\mathcal{D}_i^\prime |i\in [n]\}$ , and therefore, also identical in  $\overline{\mathcal{D}}$ . We can only obtain information at rounds when  $B_{t}\wedge y_{t} = -1$  occurs. In such rounds,  $f_{t}$  is fully determined by history (possibly with external randomness, which does not depend on data distribution),  $y_{t} = -1$  and  $\widehat{y}_t = -r_w(a_t)$  with  $r_w(a_t)$  sampled from the distribution defined in Eq (18).
For any algorithm that can successfully identify  $i$  under the data distribution  $\mathcal{D}_i$  with probability  $\frac{3}{4}$  for all  $i\in [n]$ , then  $\mathbf{P}_{\mathcal{D}_i}(i_{\mathrm{out}} = i)\geq \frac{3}{4}$  and  $\mathbf{P}_{\mathcal{D}_j}(i_{\mathrm{out}} = i)\leq \frac{1}{4}$  for all  $j\neq i$ . Recall that  $\mathcal{D}_i$  and  $\mathcal{D}_i^\prime$  are very close when the mixture parameter  $p$  is small. Combining with Eq (17), we have
$$
\begin{array}{l} \left| \mathbf {P} _ {i} \left(i _ {\mathrm {o u t}} = i\right) - \mathbf {P} _ {j} \left(i _ {\mathrm {o u t}} = i\right) \right| \\ \geq \left| \mathbf {P} _ {\mathcal {D} _ {i}} (i _ {\text {o u t}} = i) - \mathbf {P} _ {\mathcal {D} _ {j}} (i _ {\text {o u t}} = i) \right| - \left| \mathbf {P} _ {\mathcal {D} _ {i}} (i _ {\text {o u t}} = i) - \mathbf {P} _ {i} (i _ {\text {o u t}} = i) \right| - \left| \mathbf {P} _ {\mathcal {D} _ {j}} (i _ {\text {o u t}} = i) - \mathbf {P} _ {j} (i _ {\text {o u t}} = i) \right| \\ \geq \frac {1}{2} - \frac {1}{4} = \frac {1}{4}. \tag {19} \\ \end{array}
$$
Let  $\overline{w} = \frac{1}{n}\mathbf{1}$ . Let  $\mathrm{kl}(q,q')$  denote the KL divergence from  $\mathrm{Ber}(q)$  to  $\mathrm{Ber}(q')$ . Let  $H_{t-1} = (f_1, y_1, \widehat{y}_1, \ldots, f_{t-1}, y_{t-1}, \widehat{y}_{t-1})$  denote the history up to time  $t-1$ . Then we have
$$
\begin{array}{l} \mathbb {E} _ {i \sim \operatorname {U n i f} ([ n ])} \left[ \mathrm {T V} ^ {2} \left(\mathbf {P} _ {i}, \mathbf {P} _ {i + 1 \bmod n}\right) \right] \leq 4 \mathbb {E} _ {i \sim \operatorname {U n i f} ([ n ])} \left[ \mathrm {T V} ^ {2} \left(\mathbf {P} _ {i}, \overline {{\mathbf {P}}}\right) \right] \\ \leq 2 \mathbb {E} _ {i} \left[ \mathrm {D} _ {\mathrm {K L}} (\overline {{\mathbf {P}}} \| \mathbf {P} _ {i}) \right] \quad \text {(P i n s k e r ' s i n e q)} \\ = 2 \mathbb {E} _ {i} \left[ \sum_ {t = 1} ^ {T} \mathrm {D} _ {\mathrm {K L}} \left(\overline {{\mathbf {P}}} \left(f _ {t}, y _ {t}, \widehat {y} _ {t} \mid H _ {t - 1}\right) \| \mathbf {P} _ {i} \left(f _ {t}, y _ {t}, \widehat {y} _ {t} \mid H _ {t - 1}\right)\right) \right] \tag {Chain rule} \\ = 2 \mathbb {E} _ {i} \left[ \sum_ {t = 1} ^ {T} \overline {{\mathbf {P}}} \left(B _ {t} \wedge y _ {t} = - 1\right) \mathbb {E} _ {a _ {1: T} \sim \overline {{\mathbf {P}}}} \left[ \mathrm {D} _ {\mathrm {K L}} (\operatorname {B e r} (\langle \bar {w}, a _ {t} \rangle) \| \operatorname {B e r} (\langle w ^ {i}, a _ {t} \rangle)) \right] \right] \\ \end{array}
$$
$$
\begin{array}{l} = 6 (n - 1) \varepsilon \mathbb {E} _ {i} \left[ \sum_ {t = 1} ^ {T} \overline {{\mathbf {P}}} \left(B _ {t}\right) \mathbb {E} _ {a _ {1: T} \sim \overline {{\mathbf {P}}}} \left[ \mathrm {D} _ {\mathrm {K L}} (\operatorname {B e r} (\langle \bar {w}, a _ {t} \rangle) \| \operatorname {B e r} (\langle w ^ {i}, a _ {t} \rangle)) ] \right] \right. \\ = \frac {6 (n - 1) \varepsilon}{n} \sum_ {t = 1} ^ {T} \mathbb {E} _ {a _ {1: T} \sim \overline {{\mathbf {P}}}} \left[ \sum_ {i = 1} ^ {n} \mathrm {D} _ {\mathrm {K L}} (\operatorname {B e r} (\langle \bar {w}, a _ {t} \rangle) \| \operatorname {B e r} (\langle w ^ {i}, a _ {t} \rangle)) \right] \\ = \frac {6 (n - 1) \varepsilon}{n} \sum_ {t = 1} ^ {T} \mathbb {E} _ {a _ {1: T} \sim \overline {{\mathbf {P}}}} \left[ \sum_ {i: i \in a _ {t}} \operatorname {k l} \left(\frac {k _ {t}}{n}, \frac {\left(k _ {t} - 1\right) (1 - p)}{n - 1} + p\right) + \sum_ {i: i \notin a _ {t}} \operatorname {k l} \left(\frac {k _ {t}}{n}, \frac {k _ {t} (1 - p)}{n - 1}\right) \right] \\ = \frac {6 (n - 1) \varepsilon}{n} \sum_ {t = 1} ^ {T} \mathbb {E} _ {a _ {1: T} \sim \overline {{\mathbf {P}}}} \left[ k _ {t} \mathrm {k l} \left(\frac {k _ {t}}{n}, \frac {\left(k _ {t} - 1\right) (1 - p)}{n - 1} + p\right) + \left(n - k _ {t}\right) \mathrm {k l} \left(\frac {k _ {t}}{n}, \frac {k _ {t} (1 - p)}{n - 1}\right) \right] \tag {20} \\ \end{array}
$$
If  $k_{t} = 1$ , then
$$
k _ {t} \cdot \operatorname {k l} \left(\frac {k _ {t}}{n}, \frac {\left(k _ {t} - 1\right) (1 - p)}{n - 1} + p\right) = \operatorname {k l} \left(\frac {1}{n}, p\right) \leq \frac {1}{n} \log \left(\frac {1}{p}\right),
$$
$$
\left(n - k _ {t}\right) \cdot \mathrm {k l} \left(\frac {k _ {t}}{n}, \frac {k _ {t} (1 - p)}{n - 1}\right) = (n - 1) \cdot \mathrm {k l} \left(\frac {1}{n}, \frac {1 - p}{n - 1}\right) \leq \frac {1}{(1 - p) n (n - 2)},
$$
where the ineq holds due to  $\mathrm{kl}(q,q^{\prime})\leq \frac{(q - q^{\prime})^{2}}{q^{\prime}(1 - q^{\prime})}$ . If  $k_{t} = n - 1$ , it is symmetric to the case of  $k_{t} = 1$ . We have
$$
\begin{array}{l} k _ {t} \cdot \mathrm {k l} \left(\frac {k _ {t}}{n}, \frac {(k _ {t} - 1) (1 - p)}{n - 1} + p\right) = (n - 1) \mathrm {k l} \left(\frac {n - 1}{n}, \frac {n - 2}{n - 1} + \frac {1}{n - 1} p\right) = (n - 1) \mathrm {k l} \left(\frac {1}{n}, \frac {1 - p}{n - 1}\right) \\ \leq \frac {1}{(1 - p) n (n - 2)}, \\ \end{array}
$$
$$
\left(n - k _ {t}\right) \cdot \mathrm {k l} \left(\frac {k _ {t}}{n}, \frac {k _ {t} (1 - p)}{n - 1}\right) = \mathrm {k l} \left(\frac {n - 1}{n}, 1 - p\right) = \mathrm {k l} \left(\frac {1}{n}, p\right) \leq \frac {1}{n} \log \left(\frac {1}{p}\right).
$$
If  $1 < k_{t} < n - 1$ , then
$$
\begin{array}{l} k _ {t} \cdot \mathrm {k l} \left(\frac {k _ {t}}{n}, \frac {(k _ {t} - 1) (1 - p)}{n - 1} + p\right) = k _ {t} \cdot \mathrm {k l} \left(\frac {k _ {t}}{n}, \frac {k _ {t} - 1}{n - 1} + \frac {n - k _ {t}}{n - 1} p\right) \stackrel {(a)} {\leq} k _ {t} \cdot \mathrm {k l} \left(\frac {k _ {t}}{n}, \frac {k _ {t} - 1}{n - 1}\right) \\ \stackrel {(b)} {\le} k _ {t} \cdot \frac {\left(\frac {k _ {t}}{n} - \frac {k _ {t} - 1}{n - 1}\right) ^ {2}}{\frac {k _ {t} - 1}{n - 1} (1 - \frac {k _ {t} - 1}{n - 1})} = k _ {t} \cdot \frac {n - k _ {t}}{n ^ {2} (k _ {t} - 1)} \le \frac {k _ {t} \cdot}{n (k _ {t} - 1)} \le \frac {2}{n}, \\ \end{array}
$$
where inequality (a) holds due to that  $\frac{k_t - 1}{n - 1} +\frac{n - k_t}{n - 1} p\leq \frac{k_t}{n}$  and  $\mathrm{kl}(q,q^{\prime})$  is monotonically decreasing in  $q^{\prime}$  when  $q^{\prime}\leq q$  and inequality (b) adopts  $\mathrm{kl}(q,q^{\prime})\leq \frac{(q - q^{\prime})^{2}}{q^{\prime}(1 - q^{\prime})}$ , and
$$
\left(n - k _ {t}\right) \cdot \mathrm {k l} \left(\frac {k _ {t}}{n}, \frac {k _ {t} (1 - p)}{n - 1}\right) \leq \left(n - k _ {t}\right) \cdot \mathrm {k l} \left(\frac {k _ {t}}{n}, \frac {k _ {t}}{n - 1}\right) \leq \frac {k _ {t} (n - k _ {t})}{n ^ {2} (n - 1 - k _ {t})} \leq \frac {2 k _ {t}}{n ^ {2}},
$$
where the first inequality hold due to that  $\frac{k_t(1 - p)}{n - 1} \geq \frac{k_t}{n}$ , and  $\mathrm{kl}(q, q')$  is monotonically increasing in  $q'$  when  $q' \geq q$  and the second inequality adopts  $\mathrm{kl}(q, q') \leq \frac{(q - q')^2}{q'(1 - q')}$ . Therefore, we have
$$
\operatorname {Eq} (2 0) \leq \frac {6 (n - 1) \varepsilon}{n} \sum_ {t = 1} ^ {T} \mathbb {E} _ {a _ {1: T} \sim \overline {{\mathbf {P}}}} \left[ \frac {2}{n} \log \left(\frac {1}{p}\right) \right] \leq \frac {1 2 \varepsilon T \log (1 / p)}{n}.
$$
Combining with Eq (19), we have that there exists a universal constant  $c$  such that  $T \geq \frac{cn}{\varepsilon(\log(n / \varepsilon) + 1)}$ .
J Proof of Theorem 9
Proof. We will prove Theorem 9 by constructing an instance of  $\mathcal{Q}$  and  $\mathcal{H}$  and showing that for any learning algorithm, there exists a realizable data distribution s.t. achieving  $\varepsilon$  loss requires at least  $\widetilde{\Omega} \left( \frac{|\mathcal{H}|}{\varepsilon} \right)$  samples.
- Let feature vector space  $\mathcal{X} = \{0,1,\dots,n\}$  and let the space of feature-manipulation set pairs  $\mathcal{Q} = \{(0,\{0\} \cup s)|s\subset [n]\}$ . That is to say, every agent has the same original feature vector  $x = 0$  but has different manipulation ability according to  $s$ .
- Let the hypothesis class be a set of singletons over  $[n]$ , i.e.,  $\mathcal{H} = \{2\mathbb{1}_{\{i\}} - 1 | i \in [n]\}$ .
- We now define a collection of distributions  $\{\mathcal{D}_i | i \in [n]\}$  in which  $\mathcal{D}_i$  is realized by  $2\mathbb{1}_{\{i\}} - 1$ . For any  $i \in [n]$ , let  $\mathcal{D}_i$  put probability mass  $1 - 6\varepsilon$  on  $(0, \mathcal{X}, +1)$  and  $6\varepsilon$  uniformly over  $\{(0, \{0\} \cup s_{\sigma,i}, -1) | \sigma \in S_n\}$ , where  $S_n$  is the set of all permutations over  $n$  elements and  $s_{\sigma,i} := \{j | \sigma^{-1}(j) < \sigma^{-1}(i)\}$  is the set of elements appearing before  $i$  in the permutation  $(\sigma(1), \ldots, \sigma(n))$ . In other words, with probability  $1 - 6\varepsilon$ , we will sample  $(0, \mathcal{X}, +1)$  and with  $\varepsilon$ , we will randomly draw a permutation  $\sigma \sim \mathrm{Unif}(S_n)$  and return  $(0, \{0\} \cup s_{\sigma,i}, -1)$ . The data distribution  $\mathcal{D}_i$  is realized by  $2\mathbb{1}_{\{i\}} - 1$  since for negative examples  $(0, \{0\} \cup s_{\sigma,i}, -1)$ , we have  $i \notin s$  and for positive examples  $(0, \mathcal{X}, +1)$ , we have  $i \in \mathcal{X}$ .
Randomization and improperness of the output  $f_{\mathrm{out}}$  do not help Note that algorithms are allowed to output a randomized  $f_{\mathrm{out}}$  and to output  $f_{\mathrm{out}} \notin \mathcal{H}$ . We will show that randomization and improperness of  $f_{\mathrm{out}}$  don't make the problem easier. That is, supposing that the data distribution is  $\mathcal{D}_{i^*}$  for some  $i^* \in [n]$ , finding a (possibly randomized and improper)  $f_{\mathrm{out}}$  is not easier than identifying  $i^*$ . Since our feature space  $\mathcal{X}$  is finite, we can enumerate all hypotheses not equal to  $21_{\{i^*\}} - 1$  and calculate their strategic population loss as follows.
-  $21_{\emptyset} - 1$  predicts all points in  $\mathcal{X}$  by negative and thus  $\mathcal{L}^{\mathrm{str}}(21_{\emptyset} - 1) = 1 - 6\varepsilon$ ;
- For any  $a \subset \mathcal{X}$  s.t.  $0 \in a$ ,  $2\mathbb{1}_a - 1$  will predict 0 as positive and thus will predict any point drawn from  $\mathcal{D}_{i^*}$  as positive. Hence  $\mathcal{L}^{\mathrm{str}}(2\mathbb{1}_a - 1) = 6\varepsilon$ ;
- For any  $a \subset [n]$  s.t.  $\exists i \neq i^{*}, i \in a$ , we have  $\mathcal{L}^{\mathrm{str}}(2\mathbb{1}_a - 1) \geq 3\varepsilon$ . This is due to that when  $y = -1$ , the probability of drawing a permutation  $\sigma$  with  $\sigma^{-1}(i) < \sigma^{-1}(i^{*})$  is  $\frac{1}{2}$ . In this case, we have  $i \in s_{\sigma, i^{*}}$  and the prediction of  $2\mathbb{1}_a - 1$  is  $+1$ .
Under distribution  $\mathcal{D}_{i^*}$ , if we are able to find a (possibly randomized)  $f_{\mathrm{out}}$  with strategic loss  $\mathcal{L}^{\mathrm{str}}(f_{\mathrm{out}}) \leq \varepsilon$ , then we have  $\mathcal{L}^{\mathrm{str}}(f_{\mathrm{out}}) = \mathbb{E}_{h \sim f_{\mathrm{out}}}[\mathcal{L}^{\mathrm{str}}(h)] \geq \operatorname{Pr}_{h \sim f_{\mathrm{out}}} (h \neq 2\mathbb{1}_{\{i^*\}} - 1) \cdot 3\varepsilon$ . Thus,  $\operatorname{Pr}_{h \sim f_{\mathrm{out}}} (h = 2\mathbb{1}_{\{i^*\}} - 1) \geq \frac{2}{3}$  and then, we can identify  $i^*$  by checking which realization of  $f_{\mathrm{out}}$  has probability greater than  $\frac{2}{3}$ . In the following, we will focus on the sample complexity to identify the target function  $2\mathbb{1}_{\{i^*\}} - 1$  or simply  $i^*$ . Let  $i_{\mathrm{out}}$  denote the algorithm's answer to question of "what is  $i^*$ ?".
Smoothing the data distribution For technical reasons (appearing later in the analysis), we don't want to analyze distribution  $\{\mathcal{D}_i|i\in [n]\}$  directly as the probability of  $\Delta_t = i^*$  is 0 when  $f_{t}(i^{*}) = +1$ . Instead, we consider the mixture of  $\mathcal{D}_i$  and another distribution  $\mathcal{D}_i''$  to make the probability of  $\Delta_t = i^*$  be a small positive number. More specifically, let  $\mathcal{D}_i' = (1 - p)\mathcal{D}_i + p\mathcal{D}_i''$ , where  $\mathcal{D}_i''$  is defined by drawing  $(0,\mathcal{X}, + 1)$  with probability  $1 - 6\varepsilon$  and  $(0,\{0,i\} , - 1)$  with probability  $6\varepsilon$ . When  $p$  is extremely small, we will never sample from  $\mathcal{D}_i''$  when time horizon  $T$  is not too large and therefore, the algorithm behaves the same under  $\mathcal{D}_i'$  and  $\mathcal{D}_i$ . For any data distribution  $\mathcal{D}$ , let  $\mathbf{P}_{\mathcal{D}}$  be the dynamics of  $(x_{1},f_{1},\Delta_{1},y_{1},\widehat{y}_{1},\ldots ,x_{T},f_{T},\Delta_{T},y_{T},\widehat{y}_{T})$  under  $\mathcal{D}$ . According to Lemma 4, by setting  $p = \frac{\varepsilon}{16n^2}$ , when  $T\leq \frac{n}{\varepsilon}$ , we have that for any  $i,j\in [n]$
$$
\left| \mathbf {P} _ {\mathcal {D} _ {i}} \left(i _ {\text {o u t}} = j\right) - \mathbf {P} _ {\mathcal {D} _ {i} ^ {\prime}} \left(i _ {\text {o u t}} = j\right) \right| \leq \frac {1}{8}. \tag {21}
$$
Information gain from different choices of  $f_{t}$  In each round of interaction, the learner picks a predictor  $f_{t}$ , which can be out of  $\mathcal{H}$ . Suppose that the target function is  $2\mathbb{1}_{\{i^{*}\}} - 1$ . Here we enumerate all choices of  $f_{t}$  and discuss how much we can learn from each choice.
-  $f_{t} = 2\mathbb{1}_{\emptyset} - 1$  predicts all points in  $\mathcal{X}$  by negative. No matter what  $i^{*}$  is, we will observe  $\Delta_t = x_t = 0$ ,  $y_t \sim \mathrm{Rad}(1 - 6\varepsilon)$ ,  $\widehat{y}_t = -1$ . They are identically distributed for any  $i^{*} \in [n]$  and thus we cannot tell any information of  $i^{*}$  from this round.
-  $f_{t} = 2\mathbb{1}_{a_{t}} - 1$  for some  $a_{t} \subset \mathcal{X}$  s.t.  $0 \in a_{t}$ . Then no matter what  $i^{*}$  is, we will observe  $\Delta_{t} = x_{t} = 0$ ,  $y_{t} \sim \mathrm{Rad}(1 - 6\varepsilon)$ ,  $\widehat{y}_{t} = +1$ . Again, we cannot tell any information of  $i^{*}$  from this round.
-  $f_{t} = 21_{a_{t}} - 1$  for some some non-empty  $a_{t} \subset [n]$ . For rounds with  $y_{t} = +1$ , we have  $x_{t} = 0, \widehat{y}_{t} = +1$  and  $\Delta_{t} = \Delta(0, f_{t}, \mathcal{X}) \sim \mathrm{Unif}(a_{t})$ , which still do not depend on  $i^{*}$ . For rounds with  $y_{t} = -1$ , if the drawn example  $(0, \{0\} \cup s, -1)$  satisfies that  $s \cap a_{t} \neq \emptyset$ , the we would observe  $\Delta_{t} \in a_{t}$  and  $\widehat{y}_{t} = +1$ . At least we could tell that  $\mathbb{1}_{\{\Delta_{t}\}}$  is not the target function. Otherwise, we would observe  $\Delta_{t} = x_{t} = 0$  and  $\widehat{y}_{t} = -1$ .