SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
C _ {m, 1} ^ {Q} = \bigcup_ {i \in W _ {m}: Q _ {i} = 1} \left[ G _ {i}, G _ {i + 1}\right) \quad C _ {m, 2} ^ {Q} = \bigcup_ {i \in W _ {m}: Q _ {i} = - 1} \left[ G _ {i}, G _ {i + 1}\right).
$$
and  $C_m = \bigcup_{i\in W_m}\left[G_i,G_{i + 1}\right)$  we then have
$$
\eta_ {Q} (x) = \sum_ {m \in [ M ]} \mathbf {1} (x \in C _ {m, 1} ^ {Q}) \beta_ {m} ^ {1} + \mathbf {1} (x \in C _ {m, 2} ^ {Q}) \beta_ {m} ^ {2}.
$$
Finally let  $C_0^Q = [0,1] \setminus \{C_{m,1}^Q, C_{m,2}^Q : m \in [M]\}$  and note that for all  $x \in C_0^Q$ ,  $\eta(x) = 0$ . The following Lemma shows that, for a problem  $\nu \in \mathcal{B}$ , the gaps and complexity across our family problems,  $\nu^Q$  indexed by  $Q$ , does not differ too much from  $\nu$ .
Lemma B.4. Let  $Q \in \{0,1\}^K$ , for all  $m \in [M], i: [G_i, G_{i+1}) \in C_m$ , we have that  $\Delta_{U_m} / 4 < \Delta_i^{(Q)} < 3\Delta_{U_m}$ , where  $\Delta_i^{(Q)}$  is the gap of point  $i$  on problem  $\nu^Q$
Proof. We have immediately that  $3\Delta_{U_m} \leq 3\Delta_{U_m}$ , definition of  $\Delta_{U_m}$ . Then, by proposition B.3,
$$
\Delta_ {U _ {m}} / 3 \leq \frac {K \varepsilon p (1 - \mu_ {U _ {m}})}{3 | \{i : | \mu_ {i} - \mu_ {U _ {m}} | \leq \Delta_ {\} |} \leq \frac {K \varepsilon p (1 - \mu_ {U _ {m}})}{| W _ {m} |} \leq \frac {K \varepsilon p (1 - \mu_ {U _ {m}} ^ {\smile} 4 \Delta_ {U _ {m}} / 3)}{| W _ {m} |} + \frac {4 K \varepsilon p \Delta_ {U _ {m}}}{3 | W _ {m} |},
$$
which implies, in combination with our assumption  $K\varepsilon p \leq 1/8$  that,
$$
\Delta_ {U _ {m}} / 6 \leq \frac {K \varepsilon p \left(1 - \mu_ {U _ {m}} ^ {\checkmark} 4 \Delta_ {U _ {m}} / 3\right)}{\left| W _ {m} \right|}
$$
and thus that for all  $i\in W_m,\Delta_i^{(Q)}\geq \Delta_{U_m} / 6$
Lemma B.5. Given,  $\nu \in \mathcal{B}$ , we have that for all  $\nu^Q$ ,  $i \in [K]$ ,  $\Delta_i^{(Q)} \geq \Delta_i / 2$ , where  $\Delta_i^{(Q)}$  is the gap of point  $i$  on problem  $\nu^Q$ . Furthermore,  $\sum H_{i,Q}^{(1)} \geq c \sum H_i^{(1)}$  where  $H_{i,Q}^{(1)}$  is the complexity of point  $i$  on problem  $\nu^Q$ , for some absolute constant  $c > 0$ .
Proof. We prove the first statement. For all  $i: [G_i, G_{i+1}) \in C_0$  we have that  $\Delta_i^{(Q)} = 1$  and otherwise the statement follows from Lemma B.5. The second statement then follows by combination of Lemma B.5 and Propositions B.2 and B.1.
Step 2: showing that one suffers  $\varepsilon$  regret on a well chosen event Let  $Q^i$  be the transformation of  $Q$  that flips the  $i$ th coordinate,
$$
Q _ {a} ^ {k} = \left\{ \begin{array}{l} Q _ {a} \text {I f} a \neq k, \\ 1 - Q _ {a} \text {I f} a = k. \end{array} \right.
$$
We remind the reader that we denote the scoring function outputted by the learner as  $\hat{s}$ . Now define,
$$
z _ {m} := \min  \left(z: H _ {s _ {\hat {s}}} (z) \geq \frac {\lambda \left(\bigcup_ {n = 1} ^ {m - 1} C _ {n} \cup C _ {m , 1} ^ {Q}\right) \left(1 - \kappa \left(\bigcup_ {n = 1} ^ {m - 1} C _ {n} \cup C _ {m , 1} ^ {Q}\right)\right)}{(1 - p)}\right),
$$
define the event,
$$
\xi_ {i, m} := \left\{\left\{x \in \left[ G _ {i}, G _ {i + 1}\right): \hat {s} (x) > z _ {m} \right\} \leq \frac {1}{2 K} \right\}.
$$
and then the events,
$$
\mathcal {E} _ {1} ^ {m} := \left\{\sum_ {i: [ G _ {i}, G _ {i + 1}) \in C _ {m}, Q _ {i} = 1} \mathbf {1} (\xi_ {i, m}) \leq \frac {3 K \lambda (C _ {m})}{4} \right\}, \quad \mathcal {E} _ {0} ^ {m} := \left\{\sum_ {i: [ G _ {i}, G _ {i + 1}) \in C _ {m}, Q _ {i} = 0} \mathbf {1} (\xi_ {i, m}) \geq \frac {K \lambda (C _ {m})}{4} \right\}.
$$
Let  $\hat{Z}_m\subset [0,1]$  be the largest set such that,  $\forall x\in \hat{Z}_m,y\notin \hat{Z}_m,\hat{s} (x)\leq \hat{s} (y)$ , and,
$$
\lambda (\hat {Z} _ {m}) (1 - \kappa (\hat {Z} _ {m})) \leq \lambda \left(\bigcup_ {n = 1} ^ {m - 1} C _ {n} \cup C _ {m, 1}\right) \left(1 - \kappa \left(\bigcup_ {n = 1} ^ {m - 1} C _ {n} \cup C _ {m, 1}\right)\right).
$$
Note that  $\hat{Z}_m$  is not necessarily unique, in this case we choose an arbitrary such  $\hat{Z}_m$ . Furthermore, define,
$$
\hat {Z} _ {m} ^ {0} = \left\{x \in \hat {Z} _ {m}: x \in \bigcup_ {n = 1} ^ {m - 1} C _ {n} \right\}, \qquad \hat {Z} _ {m} ^ {1} = \left\{x \in \hat {Z} _ {m}: x \in C _ {m, 1} \right\},
$$
$$
\hat {Z} _ {m} ^ {2} = \left\{x \in \hat {Z} _ {m}: x \in C _ {m, 2} \cup \bigcup_ {n = m + 1} ^ {M} C _ {n} \right\}.
$$
And let  $\hat{G}_m = \bigcup_{n=1}^{m-1} C_n \setminus \hat{Z}_m^0$ . Note that under event  $\mathcal{E}_1^m$ , we have
$$
\lambda \left(Z _ {m} ^ {1}\right) \leq K \lambda \left(C _ {m}\right) / 4. \tag {27}
$$
Now, via definition of  $\hat{Z}_m$ , we have the following,
$$
\lambda \left(\hat {Z} _ {m} ^ {1}\right) (1 - \kappa \left(C _ {m, 1}\right)) + \lambda \left(\hat {Z} _ {m} ^ {2}\right) (1 - \kappa \left(C _ {m, 2}\right)) \leq \lambda \left(C _ {m, 1}\right) (1 - \kappa \left(C _ {m, 1}\right)) + \lambda \left(\hat {G} _ {m}\right) (1 - \kappa \left(\hat {G} _ {m}\right)), \tag {28}
$$
Thus, for a problem  $\nu^{Q}:\sum_{i:[G_{i},G_{i + 1})\in C_{m}}Q_{i}\geq K\lambda (C_{m}) / 2,$  on event  $\mathcal{E}_1^m$  , via combination of Equations (27) and (28),
$$
\begin{array}{l} \lambda \left(\hat {Z} _ {m} ^ {2}\right) \left(1 - \kappa \left(C _ {m, 2}\right)\right) \leq 3 \lambda \left(C _ {m, 1}\right) \left(1 - \kappa \left(C _ {m, 1}\right)\right) / 4 + \lambda \left(\hat {G} _ {m}\right) \left(1 - \kappa \left(\hat {G} _ {m}\right)\right) (29) \\ \leq \left(3 \lambda \left(C _ {m, 1}\right) / 4 + \lambda \left(\hat {G} _ {m}\right)\right) \left(1 - \kappa \left(C _ {m, 1}\right)\right), (30) \\ \end{array}
$$
where the final inequality comes from the fact  $\kappa(C_{m,1}) \geq \kappa(\hat{G}_m)$ . To complete Step: 2 we now lower bound  $d_{\infty}(\hat{s}, \eta)$  on event  $\mathcal{E}_1^m$ . Firstly note that,
$$
\begin{array}{l} \operatorname {R O C} \left(\frac {\left(1 - \hat {Z} _ {m}\right) \lambda \left(\hat {Z} _ {m}\right)}{1 - p}, \eta\right) = \frac {\lambda \left(\bigcup_ {n = 1} ^ {m - 1} C _ {n} \cup C _ {m , 1}\right) \kappa \left(\bigcup_ {n = 1} ^ {m - 1} C _ {n} \cup C _ {m , 1}\right)}{p} \\ = \frac {\lambda \left(\bigcup_ {n = 1} ^ {m - 1} C _ {n}\right) \kappa \left(\bigcup_ {n = 1} ^ {m - 1} C _ {n}\right)}{p} + \frac {\lambda \left(C _ {m , 1}\right) \kappa \left(C _ {m , 1}\right)}{p} \\ \end{array}
$$
therefore, for a problem  $\nu^{Q}:\sum_{i:P_{i}\in C_{m}}Q_{i}\geq K\lambda (C_{m}) / 2$  , on event  $\mathcal{E}_1^m$
$$
\begin{array}{l} d _ {\infty} (\hat {s}, \eta) \geq \frac {\lambda (\hat {G} _ {m}) \kappa (\hat {G} _ {m})}{p} + \frac {\lambda (C _ {m , 1}) \kappa (C _ {m , 1})}{p} - \frac {\kappa (C _ {m , 1}) \lambda (\hat {Z} _ {m} ^ {1})}{p} - \frac {\kappa (C _ {m , 2}) \lambda (\hat {Z} _ {m} ^ {2})}{p} \\ \geq \frac {\lambda (\hat {G} _ {m}) \kappa (\hat {G} _ {m})}{p} + \frac {3 \lambda (C _ {m , 1}) \kappa (C _ {m , 1})}{4 p} - \frac {\kappa (C _ {m , 2}) \lambda (\hat {Z} _ {m} ^ {2})}{p} \\ \geq \frac {\lambda (\hat {G} _ {m}) \kappa (\hat {G} _ {m})}{p} + \frac {3 \lambda (C _ {m , 1}) \kappa (C _ {m , 1})}{4 p} - \frac {\kappa (C _ {m , 2}) (3 \lambda (C _ {m , 1} / 4 + \lambda (\hat {G} _ {m}))}{p} \frac {1 - \kappa (C _ {m , 1})}{1 - \kappa (C _ {m , 2})} \\ \geq \frac {3 \lambda \left(C _ {m , 1}\right)}{4 p} \left(\kappa \left(C _ {m, 1}\right) - \kappa \left(C _ {m, 1}\right) \frac {1 - \kappa \left(C _ {m , 1}\right)}{1 - \kappa \left(C _ {m , 2}\right)}\right) \\ = \frac {3 \lambda \left(C _ {m , 1}\right)}{4 p} \left(\frac {\kappa \left(C _ {m , 1}\right) - \kappa \left(C _ {m , 2}\right)}{1 - \kappa \left(C _ {m , 2}\right)}\right) \\ = \frac {3 \lambda \left(C _ {m , 1}\right)}{4 p} \left(\frac {8 / 3 \Delta_ {U _ {m}}}{1 - \kappa \left(C _ {m , 2}\right)}\right) \geq \frac {3 \lambda \left(C _ {m , 1} \cup C _ {m , 2}\right)}{8 p} \left(\frac {8 / 3 \Delta_ {U _ {m}}}{1 - \kappa \left(C _ {m , 2}\right)}\right) \geq \varepsilon \\ \end{array}
$$
where the first inequality follows from Equation (27) and the second from (29).
Thus, as we assume policy  $\pi$  is  $\mathrm{PAC}(\delta ,\varepsilon)$ , on all problems  $\nu^Q$ , we must have that, on all problems  $\nu^{Q}:\sum_{i:[G_{i},G_{i + 1})\in C_{m}}Q_{i}\geq K\lambda (C_{m}) / 2,$
$$
\mathbb {P} _ {\nu^ {Q}, \pi} \left(\mathcal {E} _ {1} ^ {m}\right) \leq \delta , \tag {31}
$$
Via similar reasoning we can show that on all problems  $\nu^{Q}:\sum_{i:[G_{i},G_{i + 1})\in C_{m}}Q_{i}\leq K\lambda (C_{m}) / 2,$  we must have that,
$$
\mathbb {P} _ {\nu^ {Q}, \pi} \left(\mathcal {E} _ {0} ^ {m}\right) \leq \delta . \tag {32}
$$
Step 4: bounding the probability of the sum of  $\xi_i^m$  Now, for  $m\in [M]$  , via the Azuma hoeffding inequality applied to the martingale,
$$
\sum_ {i: \left[ G _ {i}, G _ {i + 1}\right) \in C _ {m}, Q _ {i} = 0} \left[ \mathbf {1} \left(\xi_ {i, m}\right) - \mathbb {P} _ {Q} \left(\xi_ {i, m}\right) \right],
$$
we have that,
$$
\mathbb {P} _ {Q} \left(\sum_ {i: \left[ G _ {i}, G _ {i + 1}\right) \in C _ {m}, Q _ {i} = 0} \left[ \mathbf {1} \left(\xi_ {i, m}\right) - \mathbb {P} _ {Q} \left(\xi_ {i, m}\right) \right] \geq K \lambda \left(C _ {m}\right) \log \left(\frac {1}{1 - \delta}\right)\right) \leq 1 - \delta . \tag {33}
$$
Thus via combination of Equations (32) and (33) we must have that,  $\forall Q:\sum_{i:[G_i,G_{i + 1})\in C_m}^K Q_i\leq K\lambda (C_m) / 2,$
$$
\sum_ {i: \left[ G _ {i}, G _ {i + 1}\right) \in C _ {m}, Q _ {i} = 0} \mathbb {P} _ {Q} \left(\xi_ {i, m}\right) \leq \lambda \left(C _ {m}\right) \left(\frac {K}{4} + K \log \left(\frac {1}{1 - \delta}\right)\right) \leq \frac {3 K \lambda \left(C _ {m}\right)}{8}, \tag {34}
$$
where the second inequality comes from our assumption  $\delta \leq 1 - \exp(-1/8)$ . Via similar reasoning we also have that,  $\forall Q: \sum_{i: [G_i, G_{i+1}) \in C_m} Q_a \geq K\lambda(C_m)/2$
$$
\sum_ {i: \left[ G _ {i}, G _ {i + 1}\right) \in C _ {m}, Q _ {i} = 1} \mathbb {P} _ {Q} \left(\xi_ {i, m}\right) \geq \lambda \left(C _ {m}\right) \left(\frac {K}{2} - K \log \left(\frac {1}{1 - \delta}\right)\right) \geq \frac {5 K \lambda \left(C _ {m}\right)}{8}. \tag {35}
$$
Step 5: applying a Fano type inequality We first define the class of problems upon which we will apply Fano.
$$
\mathfrak {Q} = \left\{Q: \forall m \in [ M ], \sum_ {i: [ G _ {i}, G _ {i + 1}) \in C _ {m}} Q _ {i} = \frac {K \lambda (C _ {m})}{2} \right\}
$$
$$
\mathfrak {Q} _ {0} = \left\{Q: \forall m \in [ M ], \sum_ {i: \left[ G _ {i}, G _ {i + 1}\right) \in C _ {m}} Q _ {i} = \frac {K \lambda \left(C _ {m}\right)}{2} - 1 \right\}
$$
We remind the reader that for  $i\in [K]$ , we write  $\tau_{i} = \sum_{t = 1}^{\tau}\mathbf{1}(a_{t}\in [G_{i},G_{i + 1}))$  and for  $m\in [M]$ $\tau_{(m)} = \sum_{i:[G_i,G_{i + 1})\in C_m}T_i$ . We see that, for all  $Q\in \mathfrak{Q}_0,i:Q_i = 0$ , there exists a unique  $\tilde{Q}\in \mathfrak{Q}$  such that  $\tilde{Q}^i = Q$ , therefore,
$$
\sum_ {Q \in \mathfrak {Q}} \sum_ {m \in [ M ]} \sum_ {i: \left[ G _ {i}, G _ {i + 1}\right) \in C _ {m}, Q _ {i} = 1} \mathbb {P} _ {Q ^ {i}} \left(\xi_ {i, m}\right) = \sum_ {Q \in \mathfrak {D} _ {0}} \sum_ {m \in [ M ]} \sum_ {i: \left[ G _ {i}, G _ {i + 1}\right) \in C _ {m}, Q _ {i} = 0} \mathbb {P} _ {Q} \left(\xi_ {i, m}\right),
$$
and thus, using the data processing inequality and the convexity of the relative entropy we have,
$$
\begin{array}{l} \operatorname {k l} \left(\underbrace {\frac {1}{| \mathfrak {Q} |} \sum_ {Q \in \mathfrak {Q}} \sum_ {m \in [ M ]} \frac {2}{K \lambda \left(C _ {m}\right)} \sum_ {i : P _ {i} \in C _ {m} , Q _ {i} = 1} \mathbb {P} _ {Q ^ {i}} \left(\xi_ {i , m}\right)} _ {\leq 6 / 8}, \underbrace {\frac {1}{| \mathfrak {Q} |} \sum_ {Q \in \mathfrak {Q}} \sum_ {m \in [ M ]} \frac {2}{K \lambda \left(C _ {m}\right)} \sum_ {i : P _ {i} \in C _ {m} , Q _ {i} = 1} \mathbb {P} _ {Q} \left(\xi_ {i , m}\right)} _ {\geq 1 0 / 8}\right) \\ \leq \frac {1}{| \mathfrak {Q} |} \sum_ {Q \in \mathfrak {Q}} \sum_ {m \in [ M ]} \frac {2}{K \lambda \left(C _ {m}\right)} \sum_ {i: P _ {i} \in C _ {m}, Q _ {i} = 1} \mathbb {E} _ {Q} \left[ \tau_ {m} \right] \frac {\operatorname {k l} \left(\mu_ {U _ {m}} - 4 / 3 \Delta_ {U _ {m}} , \mu_ {U _ {m}} + 4 / 3 \Delta_ {U _ {m}}\right)}{2} \\ \leq \frac {1}{| \mathfrak {D} |} \sum_ {Q \in \mathfrak {D}} \sum_ {m \in [ M ]} \frac {\mathbb {E} _ {Q} [ \tau_ {(m)} ] \operatorname {k l} (\mu_ {U _ {m}} - 4 / 3 \Delta_ {U _ {m}} , \mu_ {U _ {m}} + 4 / 3 \Delta_ {U _ {m}})}{K \lambda (C _ {m})} \\ \leq \max _ {Q \in \mathfrak {Q}} \sum_ {m \in [ M ]} \frac {\mathbb {E} _ {Q} [ \tau_ {(m)} ] \mathrm {k l} (\mu_ {U _ {m}} - 4 / 3 \Delta_ {U _ {m}} , \mu_ {U _ {m}} + 4 / 3 \Delta_ {U _ {m}})}{K \lambda (C _ {m})}. \\ \end{array}
$$
Then using the Pinsker inequality  $\mathrm{kl}(x,y)\geq 2(x - y)^2$ , we obtain
$$
\frac {1}{| \mathfrak {Q} _ {m} |} \sum_ {Q \in \mathfrak {Q} _ {m}} \sum_ {m \in [ M ]} \frac {2}{K \lambda (C _ {m})} \sum_ {i: P _ {i} \in C _ {m};: Q _ {i} = 1} \mathbb {P} _ {Q} (\xi_ {i, m}) \leq \frac {3}{8} + \sqrt {\max _ {Q \in \mathfrak {Q}} \sum_ {m \in [ M ]} \frac {\mathbb {E} _ {Q} [ \tau_ {(m)} ] \operatorname {k l} (\mu_ {U _ {m}} - 4 / 3 \Delta_ {U _ {m}} , \mu_ {U _ {m}} + 4 / 3 \Delta_ {U _ {m}})}{K \lambda (C _ {m})}}
$$
and therefore,
$$
\max  _ {Q \in \mathfrak {Q}} \sum_ {m \in [ M ]} \frac {\mathbb {E} _ {Q} \left[ \tau_ {(m)} \right] \operatorname {k l} \left(\mu_ {U _ {m}} - \Delta_ {m} , \mu_ {U _ {m}} + \Delta_ {m}\right)}{K \lambda \left(C _ {m}\right)} \geq \frac {9}{6 4}
$$
$$
\max  _ {Q \in \mathfrak {D}} \sum_ {m \in [ M ]} \mathbb {E} _ {Q} [ \tau_ {(m)} ] \geq c ^ {\prime} \sum_ {m \in [ M ]} \frac {K \lambda (C _ {m})}{\operatorname {k l} (\mu_ {U _ {m}} - 4 / 3 \Delta_ {U _ {m}} , \mu_ {U _ {m}} + 4 / 3 \Delta_ {U _ {m}})}
$$
where  $c' > 0$  is an absolute constant. The proof now follows, as  $\forall m \in [M], i: [G_i, G_{i+1}) \subset C_m$ ,  $H_i \geq \frac{1}{\mathrm{kl}(\mu_{U_m} - 4/3\Delta_{U_m}, \mu_{U_m} + 4/3\Delta_{U_m})}$ .