SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
\left(\frac {K}{\tilde {\tau} _ {i + 1} - \tilde {\tau} _ {i}}\right) ^ {\frac {1}{2 + d}} <   \delta_ {t _ {i}} ^ {\tilde {a} _ {i}, a _ {i}} (x _ {i}) - \delta_ {\tilde {\tau} _ {i + 1}} ^ {\tilde {a} _ {i}, a _ {i}} (x _ {i}) = \sum_ {t = t _ {i}} ^ {\tau_ {i + 1} - 1} \delta_ {t} ^ {\tilde {a} _ {i}, a _ {i}} (x _ {i}) - \delta_ {t + 1} ^ {\tilde {a} _ {i}, a _ {i}} (x _ {i}).
$$
For each round  $t = 2, \dots, T$ , define the function  $H_{i} : \mathcal{X} \times [0,1]^{K} \to [-1,1]$  by  $H_{i}(X,Y) \doteq Y^{\tilde{a}_{i}} - Y^{a_{i}}$  which is the realized difference in rewards between arms  $\tilde{a}_{i}$  and  $a_{i}$  within the reward vector  $Y$ . Then, using this notation and summing the above display over worst-case phases  $i \in [\tilde{L}_{\mathrm{pop}}]$ , we get:
$$
\sum_ {i = 1} ^ {\tilde {L} _ {\text {p o p}}} \left(\frac {K}{\tilde {\tau} _ {i + 1} - \tilde {\tau} _ {i}}\right) ^ {\frac {1}{2 + d}} <   \sum_ {i = 1} ^ {\tilde {L} _ {\text {p o p}}} \sum_ {t = \tau_ {i - 1}} ^ {\tau_ {i}} | \mathbb {E} _ {(X _ {t - 1}, Y _ {t - 1}) \sim \mathcal {D} _ {t - 1}} [ H _ {i} (X _ {t - 1}, Y _ {t - 1}) ] - \mathbb {E} _ {(X _ {t}, Y _ {t}) \sim \mathcal {D} _ {t}} [ H _ {i} (X _ {t}, Y _ {t}) ]. \tag {37}
$$
We next recall from the variational representation of the total variation distance [Polyanskiy and Wu, 2022, Theorem 7.24] that for any measurable function  $H: \mathcal{X} \times [0,1]^K \to [-1,1]$ ,
$$
\left\| \mathcal {D} _ {t} - \mathcal {D} _ {t - 1} \right\| _ {\mathrm {T V}} \geq \frac {1}{2} \left(\mathbb {E} _ {\left(X _ {t - 1}, Y _ {t - 1}\right) \sim \mathcal {D} _ {t - 1}} [ H \left(X _ {t - 1}, Y _ {t - 1}\right) ] - \mathbb {E} _ {\left(X _ {t}, Y _ {t}\right) \sim \mathcal {D} _ {t}} [ H \left(X _ {t}, Y _ {t}\right) ]\right). \tag {38}
$$
Thus, plugging (38) into (37), we get
$$
\sum_ {i = 1} ^ {\tilde {L} _ {\mathrm {p o p}}} \left(\frac {K}{\tilde {\tau} _ {i + 1} - \tilde {\tau} _ {i}}\right) ^ {\frac {1}{2 + d}} \leq 2 \sum_ {t = 2} ^ {T} \| \mathcal {D} _ {t} - \mathcal {D} _ {t - 1} \| _ {\mathrm {T V}}. \tag {39}
$$
Remark 7. Note that it is crucial in this argument that the functions  $H_{i}$  do not depend on the realized rewards  $Y_{t}$  or contexts  $X_{t}$  at any particular round  $t$ . Rather,  $H_{i}$  only depends on the arms  $\tilde{a}_{i}, a_{i}$  which in turn only depend on the mean reward sequence  $\{f_{t}\}_{t\in [T]}$ . In other words, the above step does not follow if  $a_{i}, \tilde{a}_{i}$  were defined in terms of the experienced significant shifts  $\tau_{i}(X_{T})$ .
Now, by Hölder's inequality for  $p \in (0, 1)$  and  $q \in \left(0, \frac{1 + d}{2 + d}\right)$ :
$$
\begin{array}{l} \sum_ {i = 1} ^ {\tilde {L} _ {\mathrm {p o p}}} (\tilde {\tau} _ {i + 1} - \tilde {\tau} _ {i}) ^ {\frac {1 + d}{2 + d}} K ^ {\frac {1}{2 + d}} \leq T ^ {\frac {1 + d}{2 + d}} K ^ {\frac {1}{2 + d}} \\ + \left(\sum_ {i} K ^ {\frac {1}{2 + d}} (\tilde {\tau} _ {i + 1} - \tilde {\tau} _ {i}) ^ {- q / p}\right) ^ {p} \left(\sum_ {i} K ^ {\frac {1}{2 + d}} (\tilde {\tau} _ {i + 1} - \tilde {\tau} _ {i}) ^ {\left(\frac {1 + d}{2 + d} + q\right) \cdot \frac {1}{1 - p}}\right) ^ {1 - p}. \\ \end{array}
$$
In particular, letting  $p = \frac{1}{3 + d}$  and  $q = \frac{1}{(2 + d)(3 + d)}$  and plugging in our earlier bound (39) makes the above R.H.S.
$$
T ^ {\frac {1 + d}{2 + d}} \cdot K ^ {\frac {1}{2 + d}} + V _ {T} ^ {\frac {1}{3 + d}} \cdot K ^ {\frac {1}{3 + d}} \cdot T ^ {\frac {2 + d}{3 + d}}.
$$
We first note that it suffices to show (3) for integer  $L \in [0, T] \cap \mathbb{N}$  as lower bounds for all other  $L$  follow via approximation and modifying the constant  $c > 0$  in (3). Thus, going forward, fix  $V \in [0, T]$  and  $L \in \mathbb{Z} \cap [0, T]$ .
At a high level, our construction will repeat  $L + 1$  times a hard environment for stationary contextual bandits. In particular, within each stationary phase of length  $T / (L + 1)$  one is forced to pay a regret of  $\left( T / (L + 1) \right)^{\frac{1 + d}{2 + d}}$ , summing to a total regret lower bound of  $(L + 1) \cdot \left( T / (L + 1) \right)^{\frac{1 + d}{2 + d}} \approx (L + 1)^{\frac{1}{2 + d}} \cdot T^{\frac{1 + d}{2 + d}}$ .
To obtain the lower bound expressed in terms of total variation budget  $V$  in (3), we will choose  $L \propto V^{\frac{2 + d}{3 + d}} \cdot T^{\frac{1}{3 + d}}$  and argue that the actual total variation  $V_T$  (Definition 5) is at most  $V$  in the constructed environments for this choice of  $L$ . This is similar to the arguments of the analogous dynamic regret lower bound [Besbes et al., 2019, Theorem 1] for the non-contextual bandit problem.
We start by establishing a lower bound for stationary Lipschitz contextual bandits. The construction is identical to that of Rigollet and Zeevi [2010, Theorem 4.1]. We provide the details here to (1) highlight a minor novelty in circumventing the reliance of the cited result on a positive "margin parameter"  $\alpha > 0$  and (2) to assist in later calculating the total variation  $V_T$ .
Remark 8. There are other stationary lower bound results for Lipschitz contextual bandits using similar constructions and arguments, but with context marginal measures  $\mu_{X}$  of finite support [Foster and Rakhlin, 2020, Theorem 2; Slivkins, 2014, Theorem 7]. To contrast, our construction involves setting  $\mu_{X}$  to be uniform on  $[0,1]^d$ , thus satisfying the strong density assumption (Assumption 2).
Proposition 16. Suppose there are  $K = 2$  arms. Then, there exists a finite family of stationary Lipschitz contextual bandit environments  $\mathcal{E}(n)$  over  $n$  rounds such that for any algorithm  $\pi$  taking as input random variable  $U$ , we have for some constant  $c > 0$  and environment  $\mathcal{E}$  generated uniformly and independently (of  $\pi, U$ ) at random from  $\mathcal{E}(n)$ :
$$
\mathbb {E} _ {\mathcal {E} \sim \mathrm {U n i f} (\mathcal {E} (n)), U} [ R (\pi , X _ {T}) ] \geq c \cdot n ^ {\frac {1 + d}{2 + d}}.
$$
Proof. Let the covariates  $X_{t}$  be uniformly distributed on  $[0,1]^d$  at each round  $t \in [n]$ , so that  $\mu_X \equiv \mathrm{Unif}\{[0,1]^d\}$ . For ease of presentation, let us reparametrize the two arms as  $+1$  and  $-1$ .
At each round  $t \in [n]$ , let arm -1 have reward  $Y_{t}^{-1} \sim \mathrm{Ber}(1/2)$  and let arm +1 have reward  $Y_{t}^{+1} \sim \mathrm{Ber}(f(X_{t}))$  where  $f: \mathcal{X} \to [0,1]$  is some mean reward function to be defined. Let
$$
M \doteq \left\lceil \left(\frac {n}{3 e}\right) ^ {\frac {1}{2 + d}} \right\rceil .
$$
We next partition  $\mathcal{X} = [0,1]^d$  into a regular grid of bins with centers  $\mathcal{Q} = \{q_1,\dots ,q_{M^d}\}$ , where  $q_{k}$  denotes the center of bin  $B_{k}$ ,  $k = 1,\ldots ,M^{d}$ . Concretly, re-indexing the bins, for each index  $\mathbf{k}\doteq (k_1,\dots ,k_d)\in \{1,\dots ,M\} ^d$ , we define the bin  $B_{\mathbf{k}}$  coordinate-wise as:
$$
B _ {\mathbf {k}} \doteq \left\{x \in \mathcal {X}: \frac {k _ {\ell} - 1}{M} \leq x _ {\ell} \leq \frac {k _ {\ell}}{M}, \ell = 1, \dots , d \right\}.
$$
Define  $C_{\phi} \doteq 1/4$ . Then, let  $\phi: \mathbb{R}^d \to \mathbb{R}_+$  be the smooth function defined by:
$$
\phi (x) \doteq \left\{ \begin{array}{l l} 1 - \| x \| _ {\infty} & 0 \leq \| x \| _ {\infty} \leq 1 \\ 0 & \| x \| _ {\infty} > 1 \end{array} \right..
$$
It's straightforward to verify  $\phi$  is 1-Lipschitz over  $\mathbb{R}^d$ .
Next, to ease notation, define the integer  $m \doteq M^d$ . Also, define  $\Sigma_m \doteq \{-1, 1\}^m$  and for any  $\omega \in \Omega_m$ , define the function  $f_\omega$  on  $[0, 1]^d$  via
$$
f _ {\omega} (x) \doteq 1 / 2 + \sum_ {j = 1} ^ {m} \omega_ {j} \cdot \phi_ {j} (x),
$$
where  $\phi_j(x) \doteq M^{-1} \cdot C_\phi \cdot \phi(M \cdot (x - q_j)) \cdot \mathbf{1}\{x \in B_j\}$ . Then, the optimal arm at context  $x \in \mathcal{X}$  in this environment is given by  $\pi_f^*(x) \doteq \operatorname{sgn}(f(x) - 1/2)$  (where we use the convention  $\operatorname{sgn}(0) \doteq 1$ ). If  $x \in B_j$ , then  $\pi_{f_\omega}^*(x) = \omega_j$ .
Then, define the family  $\mathcal{C}$  of environments induced by  $f_{\omega}$  for  $\omega \in \Omega_m$ . Note that  $f_{\omega}$  is also 1-Lipschitz for all  $\omega$ . Next, let  $\operatorname{Int}(B_j)$  be the  $\ell_{\infty}$  ball centered at  $q_k$  of radius  $\frac{1}{2M}$  (i.e.,  $\operatorname{Int}(B_j)$  is a ball of half the  $\ell_{\infty}$  radius contained in  $B_j$ ). Then, by the definition of  $\phi(x)$  above, we have for any  $x \in \operatorname{Int}(B_j)$  and any  $j \in [m]$ :
$$
\left| f _ {\omega} (x) - 1 / 2 \right| \geq M ^ {- 1} \cdot C _ {\phi} / 2.
$$
Next, we bound the average regret w.r.t. a uniform prior over the family  $\mathcal{C}$  of environments as:
$$
\mathbb {E} _ {f \in \mathcal {C}} \mathbb {E} \sum_ {t = 1} ^ {n} | f (X _ {t}) - 1 / 2 | \cdot \mathbf {1} \left\{\pi_ {t} (X _ {t}) \neq \pi^ {*} (X _ {t}) \right\} \geq
$$
$$
\frac {C _ {\phi}}{2 M} \mathbb {E} _ {f \in \mathcal {C}} \mathbb {E} \sum_ {t = 1} ^ {n} \sum_ {j = 1} ^ {m} \mathbf {1} \left\{\pi_ {t} \left(X _ {t}\right) \neq \pi^ {*} \left(X _ {t}\right), X _ {t} \in \operatorname {I n t} \left(B _ {j}\right) \right\}, \tag {40}
$$
where the outer expectation is over an  $f\in \mathcal{C}$  chosen uniformly at random from  $\mathcal{C}$ .
As recall  $M \propto n^{\frac{1}{2 + d}}$ , it will suffice to show the expectation on the above R.H.S. is of order  $\Omega(n)$ . This will follow essentially the same steps as the reduction to hypothesis testing in the proof of Theorem 4.1 in Rigollet and Zeevi [2010]. In particular, we note the KL divergence calculations for the induced distributions over observed data and decisions allows for the additional randomness  $U$  without consequence, as it's ignorable by use of KL chain rule. The only slight modification we make in their argument is to account for the fact that we only count rounds when  $X_{t} \in \operatorname{Int}(B_{j})$  rather than when  $X_{t} \in B_{j}$ . This will, in fact, only affect constants in the lower bound as  $B_{j}$  and  $\operatorname{Int}(B_{j})$  have similar masses.
Going into details, the following notation will be useful.
Notation 3. Let  $\omega_{[-j]}$  be  $\omega$  with the  $j$ -th entry removed, and let  $\omega_{[-j]}^i = (\omega_1,\ldots ,\omega_{j - 1},i,\omega_{j + 1},\ldots ,\omega_m)$  for  $i\in \{\pm 1\}$ . Also, let  $\mathbb{P}_{\pi ,f},\mathbb{E}_{\pi ,f}$  denote the joint measure and expectation, respectively, over all the randomness of  $\pi$  and observations within an environment induced by mean reward function  $f$ .
Then, the aforementioned reduction to hypothesis testing in the proof of Theorem 4.1 of Rigollet and Zeevi [2010] yields the following lower bound on (40):
$$
\frac {C _ {\phi}}{2 ^ {m + 1} \cdot M} \sum_ {j = 1} ^ {m} \frac {n}{4 M ^ {d}} \sum_ {\omega_ {[ - j ]} \in \Omega_ {m - 1}} \exp \left(- \frac {4}{3 M ^ {2}} \cdot N _ {j, \pi} (\omega)\right) + \tilde {N} _ {j, \pi} (\omega), \tag {41}
$$
where, letting  $X\sim \mu_X$  be an independently drawn context,
$$
N _ {j, \pi} (\omega) \doteq \mathbb {E} _ {\pi , f _ {\omega_ {[ - j ]} ^ {- 1}}} \mathbb {E} _ {X} \left[ \sum_ {t = 1} ^ {n} \mathbf {1} \{\pi_ {t} (X) = 1, X \in B _ {j} \} \right]
$$
$$
\tilde {N} _ {j, \pi} (\omega) \doteq \mathbb {E} _ {\pi , f _ {\omega_ {[ - j ]} ^ {- 1}}} \mathbb {E} _ {X} \left[ \sum_ {t = 1} ^ {n} \mathbf {1} \{\pi_ {t} (X) = 1, X \in \operatorname {I n t} (B _ {j}) \} \right].
$$
We next claim  $\tilde{N}_{j,\pi}(\omega) = N_{j,\pi} / 2^d$ , which will follow from swapping the order of expectations and summation in the above formulas. In particular, we have
$$
\begin{array}{l} \tilde {N} _ {j, \pi} (\omega) = \sum_ {t = 1} ^ {n} \mathbb {P} _ {\pi , f _ {\omega_ {[ - j ]} ^ {- 1}}} \left(\pi_ {t} (X) = 1 | X \in \operatorname {I n t} (B _ {j})\right) \cdot \mu_ {X} \left(X \in \operatorname {I n t} (B _ {j})\right) \\ = \sum_ {t = 1} ^ {n} \mathbb {P} _ {\pi , f _ {\omega_ {[ - j ]} ^ {- 1}}} (\pi_ {t} (X) = 1 | X \in \operatorname {I n t} (B _ {j})) \cdot \left(\frac {1}{2 M}\right) ^ {d} \\ = \frac {1}{2 ^ {d}} \sum_ {t = 1} ^ {n} \mathbb {P} _ {\pi , f _ {\omega_ {[ - j ]} ^ {- 1}}} (\pi_ {t} (X) = 1 | X \in B _ {j}) \cdot \mu_ {X} (X \in B _ {j}) \\ = N _ {j, \pi} (\omega) / 2 ^ {d} \\ \end{array}
$$
Thus, (41) becomes lower bounded by
$$
\begin{array}{l} \frac {C _ {\phi}}{2 ^ {m + 1} \cdot M} \sum_ {j = 1} ^ {m} \frac {n}{4 M ^ {d}} \sum_ {\omega_ {[ - j ]} \in \Omega_ {m - 1}} \exp \left(- \frac {4}{3 M ^ {2}} \cdot N _ {j, \pi} (\omega)\right) + \frac {N _ {j , \pi} (\omega)}{2 ^ {d}} \geq \\ \frac {C _ {\phi} \cdot m}{8 \cdot 2 ^ {d} \cdot M} \inf  _ {z \geq 0} \left\{\frac {n}{4 M ^ {d}} \exp \left(- \frac {4}{3 M ^ {2}} \cdot z\right) + z \right\}. \\ \end{array}
$$
The above R.H.S. is optimized at  $z^{*} \doteq \frac{3M^{2}}{4}\log\left(\frac{n}{3M^{2+d}}\right)$ . Plugging this into the above along with our choice of  $M$  defined earlier gives us a lower bound of  $\Omega(n^{\frac{1+d}{2+d}})$ .
Given Proposition 16, the  $(L + 1) \cdot \left(\frac{T}{L + 1}\right)^{\frac{1 + d}{2 + d}}$  lower bound immediately follows by lower bounding the total regret over a random environment formed by concatenating  $L + 1$  i.i.d. sampled environments from  $\mathrm{Unif}(\mathcal{E}(T / (L + 1)))$ . Any such resultant environment clearly has at most  $L$  global shifts. Note that the average regret (w.r.t. the random environment) over any stationary phase of length  $\frac{T}{L + 1}$  is lower bounded by  $\left(\frac{T}{L + 1}\right)^{\frac{1 + d}{2 + d}}$  regardless of the information learned prior to that phase, as such information can be formalized as exogenous randomness  $U$  in Proposition 16.
Next, we tackle the lower bound  $V^{\frac{1}{3 + d}} \cdot T^{\frac{2 + d}{3 + d}}$  in terms of total-variation budget  $V$ . First, if  $V < O(T^{-\frac{1}{2 + d}})$ , then we're already done as the desired rate
$$
\left(T ^ {\frac {1 + d}{2 + d}} + T ^ {\frac {2 + d}{3 + d}} \cdot V ^ {\frac {1}{3 + d}}\right) \wedge \left(\left(L + 1\right) ^ {\frac {1}{2 + d}} T ^ {\frac {1 + d}{2 + d}}\right)
$$
is minimized by the first term which is of order  $O(T^{\frac{1 + d}{2 + d}})$ . Thus, using Proposition 16 with a single stationary phase  $\mathcal{E}(T)$  gives lower bound of the right order in this regime. Such an environment clearly has total-variation  $V_{T} = 0\leq V$ .
Suppose then that  $V \geq 2 \cdot T^{-\frac{1}{2 + d}}$ . Let  $\Delta \doteq \left[ \left( \frac{T}{V} \right)^{\frac{2 + d}{3 + d}} \right] \leq T$  and consider  $L + 1 = \rho \cdot T / \Delta$  stationary phases of length  $\Delta$ , for some fixed constant  $\rho > 0$ . Then, by the previous arguments we have the regret is lower bounded by
$$
(L + 1) ^ {\frac {1}{2 + d}} \cdot T ^ {\frac {1 + d}{2 + d}} = \frac {T}{\Delta^ {\frac {1}{2 + d}}} \geq \frac {T}{2 ^ {\frac {1}{3 + d}} (T / V) ^ {\frac {1}{3 + d}}} \propto T ^ {\frac {2 + d}{3 + d}} \cdot V ^ {\frac {1}{3 + d}}.
$$
Additionally,  $T^{\frac{2 + d}{3 + d}} \cdot V^{\frac{1}{3 + d}}$  dominates  $T^{\frac{1 + d}{2 + d}}$  since  $V \geq T^{-\frac{1}{2 + d}}$ . Thus, the regret lower bound is proven in terms of  $V$ .
It remains to verify that the total-variation  $V_{T}$  is at most  $V$  in the above constructed environments so that any such environment lies in the family  $\mathcal{P}(V,L,T)$ .
Clearly, the "instantaneous total-variation"  $\| \mathcal{D}_t - \mathcal{D}_{t - 1}\|_{\mathrm{TV}} = 0$  for all rounds  $t$  not being the start of a new stationary phase. On the other hand, for a round  $t$  marking the beginning of a new phase, we have that since conditioning increases the total-variation [Polyanskiy and Wu, 2022, Theorem 7.5(c)], the instantaneous total-variation is at most:
$$
\left\| \mathcal {D} _ {t} - \mathcal {D} _ {t - 1} \right\| _ {\mathrm {T V}} \leq \mathbb {E} _ {x \sim \mu_ {X}} \left[ \left\| \mathcal {D} _ {t} \left(Y _ {t} \mid X _ {t} = x\right) - \mathcal {D} _ {t - 1} \left(Y _ {t - 1} \mid X _ {t - 1} = x\right) \right\| _ {\mathrm {T V}} \right].
$$
Since  $Y_{t}^{a}|X_{t} = x \sim \mathrm{Ber}(f_{t}^{a}(x))$ , we have the R.H.S.'s inner TV quantity is just the total variation between Bernoulli's or  $\max_{a \in \{\pm 1\}} |f_{t}^{a}(x) - f_{t-1}^{a}(x)|$ . Carefully analyzing the variations in the constructed Lipschitz reward functions in the proof of Proposition 16 reveals this TV between Bernoulli's is at most  $\frac{(3e)^{\frac{1}{2 + d}}}{2} \cdot \left( \frac{L + 1}{T} \right)^{\frac{1}{2 + d}}$ . Then, summing the instantaneous total-variation over phases, we have
$$
\begin{array}{l} V _ {T} \leq (L + 1) \cdot \frac {(3 e) ^ {\frac {1}{2 + d}}}{2} \cdot \left(\frac {L + 1}{T}\right) ^ {\frac {1}{2 + d}} \\ \leq \rho^ {\frac {3 + d}{2 + d}} (L + 1) ^ {\frac {3 + d}{2 + d}} \cdot \left(\frac {(3 e) ^ {\frac {1}{2 + d}}}{2}\right) \cdot T ^ {- \frac {1}{2 + d}} \\ <   T \cdot \left(\frac {1}{\Delta}\right) ^ {\frac {3 + d}{2 + d}} \\ \leq V, \\ \end{array}
$$
where the third inequality follows by letting  $\rho$  be a small enough constant.