SlowGuess/NeurIPS_2023_OCR huggingface dataset:
The  $K$ -armed bandit problem is an instance of structured bandits where  $\mathcal{A} = [K]$  and  $\mathcal{F} = [0,1]^K$ . The dissimilarity dimension of the  $K$ -armed bandit problem satisfies,
Proposition 31. Consider the action set  $\mathcal{A} = [K]$  and the function class  $\mathcal{F} = [0,1]^K$  as defined above. Let  $\alpha \in [0,1]$  and  $\mathcal{Z} = \mathcal{F} \times \mathcal{A}$ ,  $\rho = \rho_{\mathrm{bandits}}$  and  $\epsilon \in (0,1/2)$ . Then  $\mathsf{d}_{\rho}(\mathcal{Z},\alpha,\epsilon) \leq K$ .
Proof. Let  $c \leq \alpha - \epsilon$  and  $z_1, \ldots, z_d \in \mathcal{Z}$  with  $z_i = (f_i, a_i)$  be a maximal sequence such that,  $\rho(z_i | z_i) \geq \alpha$  while
$$
| \rho (z _ {i} | z _ {j}) - c | \leq \frac {\epsilon}{\sqrt {d}}.
$$
For  $i < j$ . Substituting the definition of  $\rho$ , this implies  $f_{i}(a_{i}) \geq \alpha$  for all  $i \in [K]$  while  $|f_{j}(a_{i}) - c| \leq \frac{\epsilon}{\sqrt{d}}$  for all  $i < j$ . By definition of  $c$ , if  $d \geq 2$
$$
f _ {j} \left(a _ {i}\right) \leq \alpha - \epsilon + \frac {\epsilon}{\sqrt {d}} \leq \alpha - \left(1 - \frac {1}{\sqrt {2}}\right) \epsilon <   \alpha - \frac {\epsilon}{4}, \quad \text {f o r a l l} i <   j. \tag {37}
$$
Let  $I_{i} = \{a_{\ell}\}_{\ell = 1}^{i}$  be the set of actions up to index  $i$  in the tuple sequence  $z_{1},\ldots ,z_{i}$ . Equation 37 implies that  $f_{j}(a)\leq \alpha -\frac{\epsilon}{4}$  for all  $j > i$ . Since  $a_{j}$  satisfies  $f_{j}(a_{j})\geq \alpha >\alpha -\frac{\epsilon}{4}$  this implies  $a_{j}\notin I_{i}$ . We conclude that  $a_{i}\neq a_{j}$  for all  $i < j$ . Since there are at most  $K$  different arm values, this implies  $d\le K$ .
D.1 Structured Bandits
We will now explain in detail how what Algorithm 1 reduces to in the structured bandits setting from Example 2. We write  $z_{i} = (f_{i},a_{i})$  for all  $i\in [T]$ . The large evaluation set  $\mathcal{Z}_{\alpha}$  can be reduced to the following set of functions,
$$
\mathcal{F}_{\alpha} = \{f\in \mathcal{F}\text{s.t.}\max_{a\in \mathcal{A}}f(a)\geq \alpha \} .
$$
Algorithm 5 Interactive Estimation for Structured Bandits
1: Input: action set  $\mathcal{A}$ , function class  $\mathcal{F}$ , optimality level  $\alpha$ , number of steps  $T$ . Compute large-evaluation function-action set  $\mathcal{F}_{\alpha} = \{f \in \mathcal{F} \text{ s.t. } \max_{a \in \mathcal{A}} f(a) \geq \alpha\}$ .
2: for  $t = 1, \dots, T$  do
3: Compute regression function
$$
f_{t} = \operatorname *{argmin}_{f\in \mathcal{F}_{\alpha}}\sum_{i = 1}^{t - 1}\left(f(a_{i}) - r_{i}\right)^{2}.
$$
4: Submit the query  $a_{t} = \operatorname{argmax}_{a \in A} f_{t}(a)$ .
5: Observe reward  $r_t$ .
6: end for
Since the  $\rho_{\mathrm{bandits}}$  function  $\rho_{\mathrm{bandits}}(z_i|z)$  is independent on  $a$  for  $z = (f,a)$ , the least squares equation  $\operatorname*{argmin}_{z\in \mathcal{Z}_\alpha}\sum_{i = 1}^{t - 1}\Bigl (\rho (z_i\mid z) - r_i\Bigr)^2$  reduces to,
$$
f _ {t} = \operatorname * {a r g m i n} _ {f \in \mathcal {F} _ {\alpha}} \sum_ {i = 1} ^ {t - 1} \left(f \left(a _ {i}\right) - r _ {i}\right) ^ {2}.
$$
finally, to ensure the action query has a self evaluation of at least  $\alpha$ , we output  $a_{t} = \operatorname{argmax}_{a \in \mathcal{A}} f_{t}(a)$ .
We will now explain in detail what the Optimistic Interactive Estimation Algorithm 3 reduces to in the structured bandits setting from Example 2. We write  $z_{i} = (f_{i},a_{i})$  for all  $i\in [T]$ .
The least squares objective  $(z_{t} = \operatorname*{argmin}_{z\in \mathcal{Z}_{\alpha}}\sum_{i = 1}^{t - 1}\Bigl (\rho (z_{i}\mid z) - r_{i}\Bigr)^{2})$  can be written as,
$$
\hat {f} _ {t} = \underset {f \in \mathcal {F}} {\operatorname {a r g m i n}} \sum_ {i = 1} ^ {t - 1} \left(f (a _ {i}) - r _ {i}\right) ^ {2}.
$$
The action component of the  $z$  element in this objective can be ignored since the  $\rho_{\mathrm{bandits}}$  evaluation function does not depend on it. The confidence ball  $\mathcal{Z}_t = \left\{z\in \mathcal{Z}:\sum_{i = 1}^{t - 1}\Bigl (\rho (z_i|z) - \rho (z_i|\hat{z}_t)\Bigr)^2\leq R\right\}$  reduces to,
$$
\mathcal {F} _ {t} = \left\{f \in \mathcal {F}: \sum_ {i = 1} ^ {t - 1} \left(f (a _ {i}) - \hat {f} _ {t} (a _ {i})\right) ^ {2} \leq R \right\}.
$$
The query can be reduced to the action component of  $z_{t}$ ,
$$
a_{t} = \operatorname *{argmax}_{f,a\in \mathcal{F}_{t}\times \mathcal{A}}f(a).
$$
Algorithm 6 summarizes this reduction and corresponds to the standard optimistic least squares for sturctured bandit problems from [24].
Algorithm 6 Optimistic Interactive Estimation for Structured Bandits
1: Input: action set  $\mathcal{A}$ , function class  $\mathcal{F}$ , confidence-set radius  $R$ , number of steps  $T$ .
2: for  $t = 1, \dots, T$  do
3: Compute confidence set
$$
\hat {f} _ {t} = \operatorname * {a r g m i n} _ {f \in \mathcal {F}} \sum_ {i = 1} ^ {t - 1} \left(f \left(a _ {i}\right) - r _ {i}\right) ^ {2}.
$$
4: Submit the query  $a_{t} = \operatorname{argmax}_{f, a \in \mathcal{F}_{t} \times \mathcal{A}} f(a)$ .
5: Observe reward  $r_t$ .
6: end for