SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
\begin{array}{l} \Pr [ \neg \mathcal {C} ] = \Pr [ \exists t: r _ {t} \notin [ 0, 1 ] ] \leq \sum_ {t \in [ T ]} \Pr [ r _ {t} \notin [ 0, 1 ] ] \quad (\text {u n i o n b o u n d}) \\ \leq \sum_ {t \in [ T ]} \Pr \left[ | \varepsilon_ {t} | \geq \lambda \right] \leq 2 \exp \left(- \lambda^ {2} / \sigma^ {2}\right) \cdot T \leq \frac {2}{T} \quad (\text {s u b s t i t u t i n g} \lambda) \\ \end{array}
$$
Plugging  $\operatorname*{Pr}[\neg \mathcal{C}]$  to Equation (5) we get:
$$
\mathbb {E} \left[ \operatorname {R e g} ^ {\varepsilon} (T) \right] \leq \mathbb {E} \left[ \operatorname {R e g} ^ {\varepsilon} (T) | \mathcal {C} \right] + 2 \tag {6}
$$
So for the remainder of the proof we will condition on the clean event  $\mathcal{C}$  and compute  $\mathbb{E}[\mathrm{Reg}^{\varepsilon}(T)|\mathcal{C}]$ . Conditioning on  $\mathcal{C}$  means that  $1 + \lambda - r_t(a) \in [0, \lambda]$ , where  $\lambda = \sigma \sqrt{\log T}$ .
We first compute the first and the second moments of estimator  $\widehat{\ell}_t(\cdot)$ . For the first moment:
$$
\mathbb {E} \left[ \widehat {\ell} _ {t} (e) \right] = \sum_ {e ^ {\prime} \in \mathcal {E}} q _ {t} \left(e ^ {\prime}\right) \cdot \frac {\ell_ {t} (e) \cdot \mathbb {1} \{e = e ^ {\prime} \}}{q _ {t} (e)} = \ell_ {t} (e) \tag {7}
$$
For the second moment:
$$
\mathbb {E} \left[ \widehat {\ell} _ {t} ^ {2} (e) \right] = \sum_ {e ^ {\prime} \in \mathcal {E}} q _ {t} \left(e ^ {\prime}\right) \frac {\ell_ {t} ^ {2} (e) \cdot \mathbf {1} \{e = e ^ {\prime} \}}{q _ {t} ^ {2} (e)} = \frac {\ell_ {t} ^ {2} (e)}{q _ {t} (e)} \leq \frac {\lambda^ {2}}{q _ {t} (e)} \tag {8}
$$
where for the first inequality, we have used the fact that  $\ell_t(e) \leq \lambda$  (since we conditioned on  $\mathcal{C}$ ) and the last one is due to the fact that  $q_t(e) \geq \gamma / |\mathcal{E}|$ .
We define the weight assigned to grid point  $e \in \mathcal{E}$  at round  $t$  as:  $w_{t}(e) = w_{t - 1}(e) \cdot \exp (-\eta \widehat{\ell}_{t}(e))$  and  $w_{0}(e) = 1, \forall e \in \mathcal{E}$ . Let  $W_{t} = \sum_{e \in \mathcal{E}} w_{t}(e)$  be the potential function. Then,
$$
W _ {0} = \sum_ {e \in \mathcal {E}} w _ {0} (e) = | \mathcal {E} | \tag {9}
$$
Using  $e^{*}$  to denote the best-fixed policy in hindsight, we have:
$$
W _ {T} = \sum_ {e \in \mathcal {E}} w _ {T} (e) \geq w _ {T} \left(e ^ {*}\right) = \exp \left(- \eta \sum_ {t \in [ T ]} \widehat {\ell} _ {t} \left(e ^ {*}\right)\right) \tag {10}
$$
We next analyze how much the potential changes per-round:
$$
\begin{array}{l} \log \left(\frac {W _ {t + 1}}{W _ {t}}\right) = \log \left(\frac {\sum_ {e \in \mathcal {E}} w _ {t} (e) \exp \left(- \eta \widehat {\ell} _ {t} (e)\right)}{W _ {t}}\right) = \log \left(\sum_ {e \in \mathcal {E}} p _ {t} (e) \exp \left(- \eta \widehat {\ell} _ {t} (e)\right)\right) \\ \leq \log \left(\sum_ {e \in \mathcal {E}} p _ {t} (e) \cdot \left(1 - \eta \widehat {\ell} _ {t} (e) + \eta^ {2} \widehat {\ell} _ {t} ^ {2} (e)\right)\right) \quad (e ^ {- x} \leq 1 - x + x ^ {2}, x > 0) \\ = \log \left(1 - \eta \sum_ {e \in \mathcal {E}} p _ {t} (e) \widehat {\ell} _ {t} (e) + \eta^ {2} \sum_ {e \in \mathcal {E}} p _ {t} (e) \widehat {\ell} _ {t} ^ {2} (e)\right) \quad (\sum_ {e \in \mathcal {E}} p _ {t} (e) = 1) \\ \leq - \eta \sum_ {e \in \mathcal {E}} p _ {t} (e) \widehat {\ell} _ {t} (e) + \eta^ {2} \sum_ {e \in \mathcal {E}} p _ {t} (e) \widehat {\ell} _ {t} ^ {2} (e) \\ = - \eta \sum_ {e \in \mathcal {E}} \frac {q _ {t} (e) - \gamma / | \mathcal {E} |}{(1 - \gamma)} \widehat {\ell} _ {t} (e) + \eta^ {2} \sum_ {e \in \mathcal {E}} \frac {q _ {t} (e) - \gamma / | \mathcal {E} |}{(1 - \gamma)} \widehat {\ell} _ {t} ^ {2} (e) \\ \leq - \eta \sum_ {e \in \mathcal {E}} \frac {q _ {t} (e) - \gamma / | \mathcal {E} |}{(1 - \gamma)} \widehat {\ell} _ {t} (e) + \eta^ {2} \sum_ {e \in \mathcal {E}} \frac {q _ {t} (e)}{(1 - \gamma)} \widehat {\ell} _ {t} ^ {2} (e) \tag {11} \\ \end{array}
$$
where the second inequality is due to the fact that  $\log x \leq x - 1$  for  $x \geq 0$ . In order for this inequality to hold we need to verify that:
$$
1 - \eta \sum_ {e \in \mathcal {E}} p _ {t} (e) \widehat {\ell} _ {t} (e) + \eta^ {2} \sum_ {e \in \mathcal {E}} p _ {t} (e) \widehat {\ell} _ {t} ^ {2} (e) \geq 0,
$$
or equivalently, that:
$$
1 - \eta \sum_ {e \in \mathcal {E}} p _ {t} (e) \widehat {\ell} _ {t} (e) \geq 0 \tag {12}
$$
We do so after we explain how to tune  $\eta$  and  $\gamma$ .
We return to Equation (11); summing up for all rounds  $t \in [T]$  in Equation (11) we get:
$$
\log \left(\frac {W _ {T}}{W _ {0}}\right) \leq - \eta \sum_ {t \in [ T ]} \sum_ {e \in \mathcal {E}} \frac {q _ {t} (e) - \gamma / | \mathcal {E} |}{(1 - \gamma)} \widehat {\ell} _ {t} (e) + \eta^ {2} \sum_ {t \in [ T ]} \sum_ {e \in \mathcal {E}} \frac {q _ {t} (e)}{(1 - \gamma)} \widehat {\ell} _ {t} ^ {2} (e) \tag {13}
$$
Using Equation (9) and Equation (10) we have that:  $\log (W_T / W_0)\geq -\eta \sum_{t\in [T]}\widehat{\ell}_t(e^*) - \log |\mathcal{E}|.$  Combining this with the upper bound on  $\log (W_T / W_0)$  from Equation (13) and multiplying both sides by  $(1 - \gamma) / \eta$  we get:
$$
\sum_ {t \in [ T ]} \sum_ {e \in \mathcal {E}} \left(q _ {t} (e) - \frac {\gamma}{| \mathcal {E} |}\right) \widehat {\ell} _ {t} (e) - (1 - \gamma) \sum_ {t \in [ T ]} \widehat {\ell} _ {t} (e ^ {*}) \leq \eta \sum_ {t \in [ T ]} \sum_ {e \in \mathcal {E}} q _ {t} (e) \widehat {\ell} _ {t} ^ {2} (e) + (1 - \gamma) \frac {\log (| \mathcal {E} |)}{\eta}
$$
We can slightly relax the right hand side using the fact that  $\gamma < 1$  and get:
$$
\sum_ {t \in [ T ]} \sum_ {e \in \mathcal {E}} \left(q _ {t} (e) - \frac {\gamma}{| \mathcal {E} |}\right) \widehat {\ell} _ {t} (e) - (1 - \gamma) \sum_ {t \in [ T ]} \widehat {\ell} _ {t} \left(e ^ {*}\right) \leq \eta \sum_ {t \in [ T ]} \sum_ {e \in \mathcal {E}} q _ {t} (e) \widehat {\ell} _ {t} ^ {2} (e) + \frac {\log (| \mathcal {E} |)}{\eta}
$$
Taking expectations (wrt the draw of the algorithm) on both sides of the above expression and using our derivations for the first and second moment (Equation (7) and Equation (8) respectively) we get:
$$
\sum_ {t \in [ T ]} \sum_ {e \in \mathcal {E}} \left(q _ {t} (e) - \frac {\gamma}{| \mathcal {E} |}\right) \ell_ {t} (e) - (1 - \gamma) \sum_ {t \in [ T ]} \ell_ {t} (e ^ {*}) \leq \eta \sum_ {t \in [ T ]} \sum_ {e \in \mathcal {E}} q _ {t} (e) \frac {\lambda^ {2}}{q _ {t} (e)} + \frac {\log (| \mathcal {E} |)}{\eta}
$$
Using the fact that  $\ell_t(\cdot) \in [0, \lambda]$  the above becomes:
$$
\mathbb {E} \left[ \operatorname {R e g} ^ {\varepsilon} (T) | \mathcal {C} \right] = \sum_ {t \in [ T ]} \sum_ {e \in \mathcal {E}} q _ {t} (e) \ell_ {t} (e) - \sum_ {t \in [ T ]} \ell_ {t} \left(e ^ {*}\right) \leq \eta T \lambda^ {2} | \mathcal {E} | + \frac {\log (| \mathcal {E} |)}{\eta} + \gamma T
$$
Tuning  $\eta = \sqrt{\frac{\log(|\mathcal{E}|)}{T\lambda^2|\mathcal{E}|}}$  and  $\gamma = 2\eta \lambda |\mathcal{E}|$ , we get that:
$$
\mathbb {E} \left[ \operatorname {R e g} ^ {\varepsilon} (T) | \mathcal {C} \right] \leq 3 \sqrt {T | \mathcal {E} | \lambda^ {2} \log (| \mathcal {E} |)} = 3 \sqrt {T | \mathcal {E} | \sigma \log (T) \log (| \mathcal {E} |)} \tag {14}
$$
Before we proceed to bounding the discretization error that we incur by playing policies only on the grid, we verify that Equation (12) holds for the chosen  $\eta$  and  $\gamma$  parameters. Note that when  $\widehat{\ell_t} (e) = 0$ , then Equation (12) holds. So we focus on the case where  $\widehat{\ell_t} (e) = \ell_t(e) / q_t(e)$ .
$$
\eta \sum_ {e \in \mathcal {E}} p _ {t} (e) \frac {\ell_ {t} (e)}{q _ {t} (e)} \leq \eta \sum_ {e \in \mathcal {E}} p _ {t} (e) \frac {\ell_ {t} (e) \cdot | \mathcal {E} |}{\gamma} \leq \eta \sum_ {e \in \mathcal {E}} p _ {t} (e) \frac {\lambda \cdot | \mathcal {E} |}{\gamma} = \eta \frac {\lambda | \mathcal {E} |}{\gamma} = \frac {1}{2}
$$
where the first inequality is due to the fact that  $q_{t}(e) \geq \gamma / |\mathcal{E}|, \forall e \in \mathcal{E}$ , the second is because  $\ell_{t}(e) \leq \lambda$ , the first equality is because  $\sum_{e \in \mathcal{E}} p_{t}(e) = 1$ , and the last equality is because of the values that we chose for parameters  $\eta$  and  $\gamma$ .
The final step in proving the theorem is to bound the strategic discretization error that we incur because our algorithm only chooses policies on the grid, while  $\pmb{\theta}^{(1)},\pmb{\theta}^{(0)}$  (and hence, the actual optimal policy) may not correspond to any grid point. Let  $a_{t}^{*}$  correspond to the action chosen by the optimal policy.
$$
S D E (T) = \sum_ {t \in [ T ]} \mathbb {E} \left[ r _ {t} \left(a _ {t} ^ {*}\right) \right] - \sum_ {t \in [ T ]} \mathbb {E} \left[ r _ {t} \left(a _ {t, e ^ {*}}\right) \right] = \sum_ {t \in [ T ]} \left\langle \boldsymbol {\theta} ^ {\left(a _ {t} ^ {*}\right)} - \boldsymbol {\theta} ^ {\left(a _ {t, e ^ {*}}\right)}, \mathbf {x} _ {t} \right\rangle
$$
We separate the  $T$  rounds into 3 groups: in group  $G_{1}$ , we have rounds  $t \in [T]$  such that  $a_{t}^{*} = a_{t,e^{*}}$ . In group  $G_{2}$ , we have rounds  $t \in [T]$  such that  $a_{t}^{*} = 0$  but  $a_{t,e^{*}} = 1$ . In group  $G_{3}$ , we have rounds  $t \in [T]$ , such that  $a_{t}^{*} = 1$  but  $a_{t,e^{*}} = 0$ . With these groups in mind, one can rewrite the above equation as:
$$
S D E (T) = \sum_ {t \in G _ {1}} \left\langle \boldsymbol {\theta} ^ {(a _ {t} ^ {*})} - \boldsymbol {\theta} ^ {(a _ {t, e ^ {*}})}, \mathbf {x} _ {t} \right\rangle + \sum_ {t \in G _ {2}} \left\langle \boldsymbol {\theta} ^ {(a _ {t} ^ {*})} - \boldsymbol {\theta} ^ {(a _ {t, e ^ {*}})}, \mathbf {x} _ {t} \right\rangle + \sum_ {t \in G _ {3}} \left\langle \boldsymbol {\theta} ^ {(a _ {t} ^ {*})} - \boldsymbol {\theta} ^ {(a _ {t, e ^ {*}})}, \mathbf {x} _ {t} \right\rangle
$$
For all the rounds in  $G_{1}$ , the strategic discretization error is equal to 0. Hence the strategic discretization error becomes:
$$
S D E (T) = \underbrace {\sum_ {t \in G _ {2}} \left\langle \boldsymbol {\theta} \left(a _ {t} ^ {*}\right) - \boldsymbol {\theta} \left(a _ {t , e ^ {*}}\right) , \mathbf {x} _ {t} \right\rangle} _ {S D E \left(G _ {2}\right)} + \underbrace {\sum_ {t \in G _ {3}} \left\langle \boldsymbol {\theta} \left(a _ {t} ^ {*}\right) - \boldsymbol {\theta} \left(a _ {t , e ^ {*}}\right) , \mathbf {x} _ {t} \right\rangle} _ {S D E \left(G _ {3}\right)} \tag {15}
$$
We first analyze  $SDE(G_2)$ :
$$
S D E (G _ {2}) = \sum_ {t \in G _ {2}} \left\langle \boldsymbol {\theta} ^ {(0)} - \boldsymbol {\theta} ^ {(1)}, \mathbf {x} _ {t} \right\rangle
$$
Let us denote by  $\widehat{\pmb{\theta}}^{(1)}$  and  $\widehat{\pmb{\theta}}^{(0)}$  the points such that  $e^* = \widehat{\pmb{\theta}}^{(1)} - \widehat{\pmb{\theta}}^{(0)}$ . Adding and subtracting  $\langle e^*, \mathbf{x}_t \rangle$  in the above, we get:
$$
\begin{array}{l} S D E \left(G _ {2}\right) = \sum_ {t \in G _ {2}} \left(\left\langle \boldsymbol {\theta} ^ {(0)} - \widehat {\boldsymbol {\theta}} ^ {(0)}, \mathbf {x} _ {t} \right\rangle + \left\langle \widehat {\boldsymbol {\theta}} ^ {(1)} - \boldsymbol {\theta} ^ {(1)}, \mathbf {x} _ {t} \right\rangle + \left\langle \widehat {\boldsymbol {\theta}} ^ {(0)} - \widehat {\boldsymbol {\theta}} ^ {(1)}, \mathbf {x} _ {t} \right\rangle\right) \\ \leq \sum_ {t \in G _ {2}} \left(\left| \left\langle \boldsymbol {\theta} ^ {(0)} - \widehat {\boldsymbol {\theta}} ^ {(0)}, \mathbf {x} _ {t} \right\rangle \right| + \left| \left\langle \widehat {\boldsymbol {\theta}} ^ {(1)} - \boldsymbol {\theta} ^ {(1)}, \mathbf {x} _ {t} \right\rangle \right| + \left\langle \widehat {\boldsymbol {\theta}} ^ {(0)} - \widehat {\boldsymbol {\theta}} ^ {(1)}, \mathbf {x} _ {t} \right\rangle\right) (x \leq | x |) \\ \leq 2 \varepsilon T + \sum_ {t \in G _ {2}} \underbrace {\left\langle \widehat {\boldsymbol {\theta}} ^ {(0)} - \widehat {\boldsymbol {\theta}} ^ {(1)} , \mathbf {x} _ {t} \right\rangle} _ {Q _ {t}} \quad (\text {C a u c h y - S c h w a r z}) \\ \end{array}
$$
Finally, we show that  $Q_{t} \leq 0$ . For the rounds where  $a_{t,e^{*}} = 1$  but  $a_{t}^{*} = 0$ , it can be the case that  $\mathbf{x}_t \neq \mathbf{x}_t'$  (as the agents only strategize in order to get assigned action 1. But since  $a_{t,e^{*}} = 1$ , then from the algorithm:
$$
\left\langle \widehat {\boldsymbol {\theta}} ^ {(1)} - \widehat {\boldsymbol {\theta}} ^ {(0)}, \mathbf {x} _ {t} ^ {\prime} \right\rangle \geq \delta \| e ^ {*} \| \Leftrightarrow \left\langle \widehat {\boldsymbol {\theta}} ^ {(0)} - \widehat {\boldsymbol {\theta}} ^ {(1)}, \mathbf {x} _ {t} ^ {\prime} \right\rangle \leq - \delta \| e ^ {*} \| \tag {16}
$$
Adding and subtracting  $\mathbf{x}_t^\prime$  from quantity  $Q_{t}$  we have:
$$
\begin{array}{l} Q _ {t} = \left\langle \widehat {\pmb {\theta}} ^ {(0)} - \widehat {\pmb {\theta}} ^ {(1)}, \mathbf {x} _ {t} - \mathbf {x} _ {t} ^ {\prime} \right\rangle + \left\langle \widehat {\pmb {\theta}} ^ {(0)} - \widehat {\pmb {\theta}} ^ {(1)}, \mathbf {x} _ {t} ^ {\prime} \right\rangle \\ \leq \left\langle \widehat {\boldsymbol {\theta}} ^ {(0)} - \widehat {\boldsymbol {\theta}} ^ {(1)}, \mathbf {x} _ {t} - \mathbf {x} _ {t} ^ {\prime} \right\rangle - \delta \| e ^ {*} \| \tag {Equation(16)} \\ \leq \left\| \widehat {\boldsymbol {\theta}} ^ {(0)} - \widehat {\boldsymbol {\theta}} ^ {(1)} \right\| \cdot \left\| \mathbf {x} _ {t} - \mathbf {x} _ {t} ^ {\prime} \right\| - \delta \left\| e ^ {*} \right\| \quad (\text {C a u c h y - S c h w a r z}) \\ \leq \left\| e ^ {*} \right\| \cdot \delta - \delta \left\| e ^ {*} \right\|. \\ \end{array}
$$
As a result:
$$
S D E \left(G _ {2}\right) \leq 2 \varepsilon T \tag {17}
$$
Moving on to the analysis of  $SDE(G_3)$ :
$$
S D E (G _ {3}) = \sum_ {t \in G _ {3}} \left\langle \boldsymbol {\theta} ^ {(0)} - \boldsymbol {\theta} ^ {(1)}, \mathbf {x} _ {t} \right\rangle
$$
Again, we use  $\widehat{\pmb{\theta}}^{(1)}$  and  $\widehat{\pmb{\theta}}^{(0)}$  the points that  $e^* = \widehat{\pmb{\theta}}^{(1)} - \widehat{\pmb{\theta}}^{(0)}$ . Adding and subtracting  $\langle e^*, \mathbf{x}_t \rangle$  and following the same derivations as in  $SDE(G_3)$ , we have that:
$$
S D E \left(G _ {3}\right) \leq 2 \varepsilon T + \sum_ {t \in G _ {3}} \underbrace {\left\langle \widehat {\boldsymbol {\theta}} ^ {(1)} - \widehat {\boldsymbol {\theta}} ^ {(0)} , \mathbf {x} _ {t} \right\rangle} _ {Q _ {t}} \tag {18}
$$
Since  $a_{t,e^*} = 0$ , then it must have been the case that  $\mathbf{x}_t' = \mathbf{x}_t$ ; this is because the agent would not spend effort to strategize if they would still be assigned action 0. For this reason, it must be that  $Q_t \leq 0$ .
Combining the upper bounds for  $SDE(G_2)$  and  $SDE(G_3)$  in Equation (15), we have that  $SDE(T) \leq 4\varepsilon T$ .
Putting everything together, we have that the regret is comprised by the regret incurred on the discretized grid and the strategic discretization error, i.e.,
$$
\mathbb {E} [ \mathrm {R e g} (T) ] \leq 3 \sqrt {T | \mathcal {E} | \sigma \log (T) \log (| \mathcal {E} |)} + 4 \varepsilon T = 3 \sqrt {T d \left(\frac {1}{\varepsilon}\right) ^ {d} \sigma \log (T) \log (1 / \varepsilon)} + 4 \varepsilon T
$$
Tuning  $\varepsilon = \left(\frac{d\sigma\log T}{T}\right)^{1 / (d + 2)}$  we get that the regret is:
D Extension to trembling hand best-response
Observe that when lazy tiebreaking (Definition 2.1), if agent  $t$  modifies their context they modify it by an amount  $\delta_L$  such that
$$
\delta_{L,t}:= \min_{0\leq \eta \leq \delta}\eta
$$
$$
\begin{array}{l} \text {s . t .} \pi_ {t} \left(\mathbf {x} _ {t} ^ {\prime}\right) = 1 \end{array}
$$
$$
\left\| \mathbf {x} _ {t} ^ {\prime} - \mathbf {x} _ {t} \right\| _ {2} = \eta .
$$
We define  $\gamma$ -trembling hand tiebreaking as  $\delta_{TH,t} = \delta_{L,t} + \alpha_t$ , where  $\alpha_t \in [0, \min\{\delta - \delta_{L,t}, \gamma\}]$  may be chosen arbitrarily. Our results in Section 3 may be extended to trembling hand tiebreaking by considering the following redefinition of a clean point:
Condition D.1 (Sufficient condition for  $\mathbf{x}' = \mathbf{x}$ ). Given a shifted linear policy parameterized by  $\beta^{(1)} \in \mathbb{R}^d$ , we say that a context  $\mathbf{x}'$  is clean if  $\langle \beta^{(1)}, \mathbf{x}' \rangle > (\delta + \gamma) \| \beta^{(1)} \|_2 + r_0$ .
No further changes are required. This will result in a slightly worse constant in Theorem 3.3 (i.e. all instances of  $\delta$  will be replaced by  $\delta + \gamma$ ). Our algorithms and results in Section 4 do not change. Our definition of trembling hand best-response is similar in spirit to the  $\epsilon$ -best-response in Haghtalab et al. [26]. Specifically, Haghtalab et al. [26] study a Stackelberg game setting in which the follower best-responds  $\epsilon$ -optimally. In our trembling hand setting, the strategic agent can also be thought of as  $\epsilon$ -best responding (using the language of [26]), although it is important to note that an  $\epsilon$ -best response for the agent in our setting will cause the agent to only strategize more than necessary.