SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Our work is based on the label shift assumption which restricts the applicability of our methods to scenarios where this assumption holds. However, as noted in Section 1, the problem of adaptation to changing data distribution is intractable without imposing assumptions on the nature of the shift.
Furthermore, as noted in Remark 9, our methods in the SOLS settings have a memory requirement that scales linearly with time, which may not be feasible in scenarios where memory is limited. This is reminiscent to FTL / FTRL type algorithms from online learning. We leave the task of deriving theoretical guarantees with reduced storage complexity under non-convex losses as an important future direction.
Offline total variation denoising The offline problem of Total Variation (TV) denoising constitutes estimating the ground truth under the observation model in Definition 1 with the caveat that all observations are revealed ahead of time. This problem is well studied in the literature of locally adaptive non-parametric regression [20, 21, 23, 52, 69, 45, 66, 73, 60, 34]. The optimal total squared error (TSE) rate for estimation is known to be  $\tilde{O}(T^{1/3}V_T^{2/3} + 1)$  [22]. Estimating sequences of bounded TV has received a lot of attention in literature mainly because of the fact that these sequences exhibit spatially varying degree of smoothness. Most signals of scientific interest are known to contain spatially localised patterns [44]. This property also makes the task of designing optimal estimators particularly challenging because the estimator has to efficiently detect localised patterns in the ground truth signal and adjust the amount of smoothing to be applied to optimally trade-off bias and variance.
Non-stationary online learning The problem of online regression can be casted into the dynamic regret minimisation framework of online learning. We assume the notations in Definition 1. In this framework, at each round the learner makes a decision  $\hat{\theta}_t$ . Then the learner suffers a squared error loss  $\ell_t(\hat{\theta}_t) = \| z_t - \hat{\theta}_t\|_2^2$ . The gradient of the loss at the point of decision,  $\nabla \ell_t(\hat{\theta}_t) = 2(\hat{\theta}_t - z_t)$ , is revealed to the learner. The expected dynamic regret against the sequence of comparators  $\theta_{1:T}$  is given by
$$
\begin{array}{l} R \left(\theta_ {1: T}\right) = \sum_ {t = 1} ^ {T} E \left[ \ell_ {t} \left(\hat {\theta} _ {t}\right) - \ell_ {t} \left(\theta_ {t}\right) \right] \\ = \sum_ {t = 1} ^ {T} E [ \| z _ {t} - \hat {\theta} _ {t} \| _ {2} ^ {2} ] - E [ \| z _ {t} - \theta_ {t} \| _ {2} ^ {2} ] \\ = \sum_ {t = 1} ^ {T} E \left[ \| \hat {\theta} _ {t} \| _ {2} ^ {2} - \| \theta_ {t} \| _ {2} ^ {2} - 2 z _ {t} ^ {T} \hat {\theta} _ {t} + 2 z _ {t} ^ {T} \theta_ {t} \right] \\ = \sum_ {t = 1} ^ {T} E [ \| \hat {\theta} _ {t} - \theta_ {t} \| _ {2} ^ {2} ], \\ \end{array}
$$
where in the last line we used the fact that the noise  $\epsilon_{t}$  (see Definition 1) is zero mean and independent of the online decisions  $\hat{\theta}_{t}$ . Due to this relation, we conclude that any algorithm that can optimally control the dynamic regret with respect to squared error losses  $\ell_t(x) = \| z_t - x\| _2^2$  can be directly used to control the TSE from the ground truth sequence  $\theta_{1:T}$ .
The minimax estimation rate is defined as follows:
$$
R^{*}(T,V_{T}) = \min_{\hat{\theta}_{1:T}}\max_{\substack{\theta_{1:T}\\ \sum_{i = 2}^{T}\| \theta_{i} - \theta_{i - 1}\|_{1}\leq V_{T}}}\sum_{t = 1}^{T}E[\| \hat{\theta}_{t} - \theta_{t}\|_{2}^{2}]
$$
Algorithms that can control the dynamic regret with respect to convex losses such as those presented in the works of Jadbabaie et al. [43], Besbes et al. [10], Yang et al. [78], Chen et al. [14], Zhang et al. [81], Goel and Wierman [32], Zhao et al. [84], Zhao and Zhang [83], Zhao et al. [85], Chang
Algorithm	Run-time	Memory
FLH-FTL [38]	O(T2)	O(T2)
Aligator [7]	O(T log T)	O(T)
Arrows [3]	O(T log T)	O(1)
Table 4: Run-time and memory complexity of various adaptively minimax optimal online regression algorithms (see Definition 1). For practical purposes, the storage requirement is negligible even for FLH-FTL. For example, with 10 classes and  $T = 1000$ , the storage requirement of FLH-FTL is only 40KB, which is insignificant compared to the storage capacity of most modern devices.
and Shahrampour [13], Jacobsen and Cutkosky [42], Baby and Wang [6] can lead to sub-optimal estimation rates of order  $O(\sqrt{T(1 + V_T)})$ .
On the other hand, algorithms presented in Hazan and Seshadhri [38], Daniely et al. [19], Baby and Wang [3], Baby et al. [7], Raj et al. [56], Baby and Wang [4, 5] exploit the curvature of the losses and attain the (near) optimal estimation rate of  $\tilde{O}(T^{1/3}V_T^{2/3} + 1)$ .
Online non-parametric regression The task of estimating a sequence of TV bounded sequence from noisy observations can be cast into the online non-parametric regression framework of Rakhlin and Sridharan [57]. Results on online non-parametric regression against reference class of Lipschitz sequences, Sobolev sequences and isotonic sequences can be found in [25, 47, 48] respectively. However as noted in Baby and Wang [3], these classes feature sequences that are more regular than TV bounded sequences. In fact they can be embedded inside a TV bounded sequence class [59]. So the minimax optimality of an algorithm for TV class implies minimax optimality for the smoother sequence classes as well.
Offline label shift Dataset shifts are predominantly studied under two scenarios: covariate shift and label shift [65]. Under covariate shift,  $p(y|x)$  doesn't change, while in label shift  $p(x|y)$  remains invariant. Scholkopf et al. [63] articulates connections between label shift and covariate shift with anti-causal and causal models respectively. Covariate shift is well explored in past [80, 79, 17, 16, 33]. The offline label shift assumption has been extensively studied in the domain adaptation literature [61, 65, 80, 51, 36, 26, 1, 30] and is also related to problems of estimating mixture proportions of different classes in unlabeled data where previously unseen classes may appear (e.g., positive and unlabeled learning; Elkan and Noto [24], Bekker and Davis [9], Garg et al. [27, 28], Roberts et al. [58]). Classical approaches suffer from the curse of dimensionality, failing in settings with high dimensional data where deep learning prevails. More recent methods work leverage black-box predictors to produce sufficient low-dimensional representations for identifying target distributions of interest [51, 2, 61, 1]. In our work, we leverage black box predictors to estimate label marginals at each step which we track with online regression oracles to trade off variance with (small) bias.
C Omitted proofs from Section 3
In the next two lemmas, we verify Assumption 2 for some important loss functions.
Lemma 11 (cross-entropy loss). Consider a sample  $(x,y)\sim Q$ . Let  $p\in \mathbb{R}_+^K$  and  $\tilde{p} (x)\in \Delta_K$  be a distribution that assigns a weight proportional  $\frac{p(i)}{q_0(i)} f_0(i|x)$  to the label  $i$ . Let  $\ell (\tilde{p} (x),y) = \sum_{i = 1}^{K}\mathbb{I}\{y = i\} \log (1 / p(x)[i])$  be the cross-entropy loss. Let  $L(p)\coloneqq E_{(x,y)\sim Q}[\ell (p(x),y)]$  be its population analogue. Then  $L(p)$  is  $2\sqrt{K} /\mu$  Lipschitz in  $\| \cdot \| _2$  norm over the clipped box  $\mathcal{D}\coloneqq \{p\in \mathbb{R}_{+}^{K}:\mu \leq p(i)\leq 1\forall i\in [K]\}$  which is compact and convex. Further, the true marginals  $q_{t}\in \mathcal{D}$  whenever  $q_{t}(i)\geq \mu$  for all  $i\in [K]$ .
$$
\begin{array}{l} L (p) = - \sum_ {i = 1} ^ {K} E [ E [ Q _ {t} (i | x) \log (\tilde {p} (x) [ i ]) | x ] ] \\ = E \left[ \log \left(\sum_ {i = 1} ^ {K} w _ {i} p (i)\right) \right] - \sum_ {i = 1} ^ {K} E \left[ E \left[ Q _ {t} (i | x) \log \left(w _ {i} p (i)\right) | x \right] \right], \\ \end{array}
$$
where we define  $w_{i}\coloneqq f_{0}(i|x) / q_{0}(i)$ . Then we can see that
$$
\nabla L (p) [ i ] = E \left[ \frac {w _ {i}}{\sum_ {j = 1} ^ {K} w _ {j} p (j))} \right] - E \left[ \frac {Q _ {t} (i | x)}{p (i)} \right].
$$
So if  $\min_i p(i) \geq \mu$ , we have that  $\frac{w_i}{\sum_{j=1}^K w_j p(j)} \leq 1 / \mu$  and  $Q_t(i|x) / p(i) \leq 1 / \mu$ . So by triangle inequality,  $|\nabla L(p)[i]| \leq 1 / \mu + 1 / \mu$ .
Lemma 12 (binary 0-1 loss). Consider a sample  $(x,y)\sim Q$ . Let  $p\in \mathbb{R}_+^K$  and  $\tilde{p} (x)\in \Delta_K$  be a distribution that assigns a weight proportional  $\frac{p(i)}{q_0(i)} f_0(i|x)$  to the label  $i$ . Let  $\hat{y} (x)$  be a sample obtained from the distribution  $\tilde{p} (x)$ . Consider the binary 0-1 loss  $\ell (\hat{y} (x),y) = \mathbb{I}(\hat{y} (x)\neq y)$ . Let  $L(p)\coloneqq E_{(x,y)\sim Q,\hat{y} (x)\sim \tilde{p} (x)}I(\hat{y} (x)\neq y)$  be its population analogue. Let  $q_{0}(i)\geq \alpha >0$ . Then  $L(p)$  is  $2K^{3 / 2} / (\alpha \tau)$  Lipschitz in  $\| \cdot \| _2$  norm over the domain  $\mathcal{D}\coloneqq \{p\in \mathbb{R}_{+}^{K}: \sum_{i = 1}^{K}p(i)f_{0}(i|x)\geq$ $\tau$ ,  $p(i)\leq 1\forall i\in [K]\}$  which is compact and convex. Further, the true marginals  $q_{t}\in \mathcal{D}$  whenever  $q_{t}(i)\geq \mu$  for all  $i\in [K]$ .
$$
L (p) = \sum_ {i = 1} ^ {K} E [ Q (y \neq i | x) \tilde {p} (x) [ i ] ].
$$
Denote  $\tilde{p}(x)[i] = p(i)w_i / \sum_{j=1}^{K} p(j)w_j$  with  $w_j \coloneqq f_0(i|x) / q_0(i)$ . Then we see that
$$
\begin{array}{l} \left| \frac {\partial \tilde {p} (x) [ i ]}{\partial p (i)} \right| = \left| \frac {w _ {i}}{\sum_ {j = 1} ^ {K} w _ {j} p (j)} - \frac {(w _ {i} p (i)) w _ {i}}{\left(\sum_ {j = 1} ^ {K} w _ {j} p (j)\right) ^ {2}} \right| \\ \leq \frac {1}{\alpha \tau} + \frac {w _ {i}}{\sum_ {j = 1} ^ {K} w _ {j} p (j)} \\ \leq 2 / (\alpha \tau). \\ \end{array}
$$
$$
\begin{array}{l} \left| \frac {\partial \tilde {p} (x) [ i ]}{\partial p (j)} \right| = \frac {w _ {i} p (i) w _ {j}}{\left(\sum_ {j = 1} ^ {K} w _ {j} p (j)\right) ^ {2}} \\ \leq 1 / (\alpha \tau). \\ \end{array}
$$
Thus we conclude that  $\| \nabla \tilde{p} (x)[i]\| _2\leq 2\sqrt{K} /(\alpha \tau)$ , where the gradient is taken with respect to  $p\in \mathbb{R}_+^K$
$$
\begin{array}{l} \| \nabla L (p) \| _ {2} \leq \sum_ {i = 1} ^ {K} \| \nabla \tilde {p} (x) [ i ] \| _ {2} \\ \leq 2 K ^ {3 / 2} / (\alpha \tau). \\ \end{array}
$$
Remark 13. The condition  $\sum_{i=1}^{K} f_0(i|x)p(i) \geq \tau$  is closely related to Condition 1 of Garg et al. [26]. Note that this is strictly weaker than imposing the restriction that the distribution  $p(i) \geq \mu$  for each  $i$ .
Remark 14. We emphasize that the conditions in Lemmas 11 and 12 are only sufficient conditions that imply bounded gradients. However, they are not necessary for satisfying bounded gradients property.
Lemma 15. Let  $\mu, \nu \in \Delta_K$  be such that  $\mu[i] = q_t(i)$ . Let  $s_t = C^{-1}f_0(x_t)$ , where  $C$  is the confusion matrix defined in Assumption 1. We have that  $E[s_t] = \mu$  and  $\operatorname{Var}(s_t) \leq 1 / \sigma_{\min}^2(C)$
Proof. Let  $\tilde{q}_t(\hat{y}_t) = E_{x_t\sim Q_t^X,\hat{y} (x_t)\sim f_0(x_t)}\mathbb{I}\{\hat{y} (x_t) = \hat{y}_t\}$  be the probability that the classifier  $f_{0}$  predicts the label  $\hat{y}_t$ . Here  $Q_{t}^{X}(x)\coloneqq \sum_{i = 1}^{K}Q_{t}(x,i)$ . Let's denote  $Q_{t}(\hat{y} (x_{t}) = \hat{y}_{t}|y_{t} = i)\coloneqq$ $E_{x_t\sim Q_t(\cdot |y = i),\hat{y} (x_t)\sim f_0(x_t)}\mathbb{I}\{\hat{y} (x_t) = \hat{y}_t\}$ . By law of total probability, we have that
$$
\begin{array}{l} \tilde {q} _ {t} (\hat {y} _ {t}) = \sum_ {i = 1} ^ {K} Q _ {t} (\hat {y} (x _ {t}) = \hat {y} _ {t} | y _ {t} = i) q _ {t} (i) \\ = \sum_ {i = 1} ^ {K} Q _ {0} (\hat {y} (x _ {t}) = \hat {y} _ {t} | y _ {t} = i) q _ {t} (i), \\ \end{array}
$$
where the last line follows by the label shift assumption.
Let  $\mu, \nu \in \mathbb{R}^{K}$  be such that  $\mu[i] = q_{t}(i)$  and  $\nu[i] = \tilde{q}_{t}(i)$ . Then the above equation can be represented as  $\nu = C\mu$ . Thus  $\mu = C^{-1}\nu$ .
Given a sample  $x_{t} \in Q_{t}$ , the vector  $f_{0}(x_{t})$  forms an unbiased estimate of  $\nu$ . Hence we have that the vector  $\hat{\mu} := C^{-1}f_0(x_t)$  is an unbiased estimate of  $\mu$ . Moreover,
$$
\begin{array}{l} \left\| \hat {\mu} \right\| _ {2} \leq \left\| C ^ {- 1} \right\| _ {2} \left\| f _ {0} \left(x _ {t}\right) \right\| \\ \leq 1 / \sigma_ {\min } (C). \\ \end{array}
$$
Hence the variance of the estimate  $\hat{\mu}$  is bounded by  $1 / \sigma_{min}^2 (C)$ .
We have the following performance guarantee for online regression due to Baby et al. [7].
Proposition 16 (Baby et al. [7]). Let  $s_t = C^{-1}f_0(x_t)$ . Let  $\hat{q}_t \coloneqq ALG(s_{1:t-1})$  be the online estimate of the true label marginal  $q_t$  produced by the Aligator algorithm by taking  $s_{1:t-1}$  as input at a round  $t$ . Then we have that
$$
\sum_ {t = 1} ^ {T} E \left[ \| \hat {q} _ {t} - q _ {t} \| _ {2} ^ {2} \right] = \tilde {O} (K ^ {1 / 3} T ^ {1 / 3} V _ {T} ^ {2 / 3} (1 / \sigma_ {m i n} ^ {4 / 3} (C)) + K),
$$
where  $V_{T} \coloneqq \sum_{t=2}^{T} \| q_{t} - q_{t-1} \|_{1}$ . Here  $\tilde{O}$  hides dependencies in absolute constants and polylogarithmic factors of the horizon. Further this result is attained without prior knowledge of the variation  $V_{T}$ .
By following the arguments in Baby and Wang [4], a similar statement can be derived also for the FLH-FTL algorithm of Hazan and Seshadhri [38] (Algorithm 7).
Proof. Owing to our carefully crafted reduction from the problem of online label shift to online regression, the proof can be conducted in just a few lines. Let  $\tilde{q}_t$  be the value of  $\mathrm{ALG}(s_{1:t-1})$  computed at line 2 of Algorithm 2. Recall that the dynamic regret was defined as:
$$
R _ {\text {d y n a m i c}} (T) = \sum_ {t = 1} ^ {T} L _ {t} \left(\hat {q} _ {t}\right) - L _ {t} \left(q _ {t}\right) \leq \sum_ {t = 1} ^ {T} G \| \hat {q} _ {t} - q _ {t} \| _ {2} \tag {4}
$$
Continuing from Eq.(4), we have
$$
\begin{array}{l} E \left[ R _ {\text {d y n a m i c}} (T) \right] \leq \sum_ {t = 1} ^ {T} G \cdot E \left[ \| \hat {q} _ {t} - q _ {t} \| _ {2} \right] \\ \leq \sum_ {t = 1} ^ {T} G \cdot E [ \| \tilde {q} _ {t} - q _ {t} \| _ {2} ] \\ \leq \sum_ {t = 1} ^ {T} G \sqrt {E \| \tilde {q} _ {t} - q _ {t} \| _ {2} ^ {2}} \\ \leq G \sqrt {T \sum_ {t = 1} ^ {T} E [ \| \tilde {q} _ {t} - q _ {t} \| _ {2} ^ {2} ]} \\ = \tilde {O} \left(K ^ {1 / 6} T ^ {2 / 3} V _ {T} ^ {1 / 3} \left(1 / \sigma_ {\min } ^ {2 / 3} (C)\right) + \sqrt {K T} / \sigma_ {\min } (C)\right), \\ \end{array}
$$
where the second line is due to non-expansivity of projection, the third line is due to Jensen's inequality, fourth line by Cauchy-Schwartz and last line by Proposition 16. This finishes the proof.
Next, we provide matching lower bounds (modulo log factors) for the regret in the unsupervised label shift setting. We start from an information-theoretic result which will play a central role in our lower bound proofs.
Proposition 17 (Theorem 2.2 in Tsybakov [68]). Let  $\mathbb{P}$  and  $\mathbb{Q}$  be two probability distributions on  $\mathcal{H}$ , such that  $KL(\mathbb{P}||\mathbb{Q}) \leq \beta < \infty$ . Then for any  $\mathcal{H}$ -measurable real function  $\phi: \mathcal{H} \to \{0,1\}$ ,
$$
\max  \{\mathbb {P} (\phi = 1), \mathbb {Q} (\phi = 0) \} \geq \frac {1}{4} \exp (- \beta).
$$
Theorem 5. Let  $V_{T} \leq 64T$ . There exists a loss function, a domain  $\mathcal{D}$  (in Assumption 2), and a choice of adversarial strategy for generating the data such that for any algorithm, we have  $\sum_{t=1}^{T} E\left([L_{t}(\hat{q}_{t})] - L_{t}(q_{t})\right) = \Omega\left(\max\{T^{2/3} V_{T}^{1/3}, \sqrt{T}\}\right)$ , where  $\hat{q}_{t} \in \mathcal{D}$  is the weight estimated by the algorithm and  $q_{t} \in \mathcal{D}$  is the label marginal at round  $t$  chosen by the adversary. Here the expectation is taken with respect to the randomness in the algorithm and the adversary.
Proof. We start with a simple observation about KL divergence. Consider distributions with density  $P(x,y) = P_0(x|y)p(y)$  and  $Q(x,y) = P_0(x|y)q(y)$  where  $(x,y)\in \mathbb{R}\times [K]$ . Note that these distributions are consistent with the label shift assumption. We note that
$$
\begin{array}{l} \operatorname {K L} (P | | Q) = \sum_ {i = 1} ^ {K} \int_ {\mathbb {R}} P _ {0} (x | i) p (i) \log \left(\frac {P _ {0} (x | i) p (i)}{P _ {0} (x | i) q (i)}\right) d x \\ = \sum_ {i = 1} ^ {K} \int_ {\mathbb {R}} P _ {0} (x | i) p (i) \log \left(\frac {p (i)}{q (i)}\right) d x \\ = \sum_ {i = 1} ^ {K} p (i) \log \left(\frac {p (i)}{q (i)}\right) \\ \end{array}
$$
Thus we see that under the label shift assumption, the KL divergence is equal to the KL divergence between the marginals of the labels.
Next, we define a problem instance and an adversarial strategy. We focus on a binary classification problem where the labels is either 0 or 1. As noted before, the KL divergence only depends on the
marginal distribution of labels. So we fix the density  $Q_0(x|y)$  to be any density such that under the uniform label marginals  $(q_0(1) = q_0(0) = 1/2)$  we can find a classifier with invertible confusion matrix (recall from Fig. 1 that  $Q_0$  corresponds to the data distribution of the training data set).
Divide the entire time horizon  $T$  is divided into batches of size  $\Delta$ . So there are  $M \coloneqq T / \Delta$  batches (we assume divisibility). Let  $\Theta = \left\{\frac{1}{2} - \delta, \frac{1}{2} + \delta\right\}$  be a set of success probabilities, where each probability can define a Bernoulli trial. Here  $\delta \in (0, 1/4)$  which will be tuned later.
The problem instance is defined as follows:
- For batch  $i \in  \left\lbrack  M\right\rbrack$  ,adversary selects a probability  ${\widehat{q}}_{i} \in  \Theta$  uniformly at random.
- For any round  $t$  that belongs to the  $i^{th}$  batch, sample a label  $y_{t} \sim \mathrm{Ber}(q_{t})$  and co-variate  $x_{t} \sim Q_{0}(\cdot | y_{t})$ . Here  $q_{t} = \bar{q}_{i}$ . The co-variate  $x_{t}$  is revealed.
- Let  $\hat{q}_t$  be any estimate of  $q_t$  at round  $t$ . Define the loss as  $L_t(\hat{q}_t) \coloneqq \mathbb{I}\{q_t \geq 1/2\}(1 - \hat{q}_t) + \mathbb{I}\{q_t < 1/2\} \hat{q}_t$ .
We take the domain  $\mathcal{D}$  in Assumption 2 as  $[1/2 - \delta, 1/2 + \delta]$ . It is easy to verify that  $L_{t}(\hat{q}_{t})$  is Lipschitz over  $\mathcal{D}$ . Note that unlike Besbes et al. [10], we do not have an unbiased estimate of the gradient of loss functions.
Let's compute an upperbound on the total variation incurred by the true marginals. We have
$$
\begin{array}{l} \sum_ {t = 2} ^ {T} \left| q _ {t} - q _ {t - 1} \right| = \sum_ {i = 2} ^ {M} \left| \overset {\prime} {q} _ {i} - \overset {\prime} {q} _ {i - 1} \right| \\ \leq 2 \delta M \\ \leq V _ {T}, \\ \end{array}
$$
where the last line is obtained by choosing  $\delta = V_T / (2M) = V_T\Delta /(2T)$ .
Since at the beginning of each batch, the sampling probability is chosen uniformly at random, the loss function in the current batch is independent of the history available at the beginning of the batch. So only the data in the current batch alone is informative in minimising the loss function in that batch. Hence it is sufficient to consider algorithms that only use the data within a batch alone to make predictions at rounds that falls within that batch.
Now we proceed to bound the regret incurred within batch 1. The computation is identical for any other batches.
Let  $\mathbb{P}$  be the joint probability distribution in which labels  $(y_{1},\ldots ,y_{\Delta})$  within batch 1 are sampled with success probability  $1 / 2 - \delta$  (i.e  $q_{t} = 1 / 2 - \delta$ )
$$
\mathbb {P} \left(y _ {1}, \dots , y _ {\Delta}\right) = \Pi_ {i = 1} ^ {\Delta} \left(1 / 2 - \delta\right) ^ {y _ {i}} \left(1 / 2 + \delta\right) ^ {1 - y _ {i}}.
$$
Define an alternate distribution  $\mathbb{Q}$  such that
$$
\mathbb {Q} (y _ {1}, \ldots , y _ {\Delta}) = \Pi_ {i = 1} ^ {\Delta} (1 / 2 + \delta) ^ {y _ {i}} (1 / 2 - \delta) ^ {1 - y _ {i}}.
$$
According to the above distribution the data are independently sampled from Bernoulli trials with success probability  $1 / 2 + \delta$ . (i.e  $q_{t} = 1 / 2 + \delta$ )
Moving forward, we will show that by tuning  $\Delta$  appropriately, any algorithm won't be able to detect between these two alternate worlds with constant probability resulting in sufficiently large regret.
We first bound the KL distance between these two distributions. Let
$$
\begin{array}{l} \operatorname {K L} (1 / 2 - \delta | | 1 / 2 + \delta) := (1 / 2 + \delta) \log \left(\frac {1 / 2 + \delta}{1 / 2 - \delta}\right) + (1 / 2 - \delta) \log \left(\frac {1 / 2 - \delta}{1 / 2 + \delta}\right) \\ \le_ {(a)} (1 / 2 + \delta) \frac {2 \delta}{1 / 2 + \delta} - (1 / 2 - \delta) \frac {2 \delta}{1 / 2 + \delta} \\ = \frac {1 6 \delta^ {2}}{1 - 4 \delta^ {2}} \\ \le_ {(b)} \frac {6 4 \delta^ {2}}{3}, \\ \end{array}
$$
where in line (a) we used the fact that  $\log(1 + x) \leq x$  for  $x > -1$  and observed that  $-4\delta / (1 + 2\delta) > -1$  as  $\delta \in (0, 1/4)$ . In line (b) we used  $\delta \in (0, 1/4)$ .
Since  $\mathbb{P}$  and  $\mathbb{Q}$  are product of the marginals due to independence we have that
$$
\begin{array}{l} \operatorname {K L} (\mathbb {P} | | \mathbb {Q}) = \sum_ {t = 1} ^ {\Delta} \operatorname {K L} (1 / 2 - \delta | | 1 / 2 + \delta) \\ \leq (6 4 \Delta / 3) \cdot \delta^ {2} \\ = 1 6 / 3 \\ := \beta , \tag {5} \\ \end{array}
$$
where we used the choices  $\delta = \Delta V_{T} / (2T)$  and  $\Delta = (T / V_{T})^{2 / 3}$ .
Suppose at the beginning of batch, we reveal the entire observations within that batch  $y_{1:\Delta}$  to the algorithm. Note that doing so can only make the problem easier than the sequential unsupervised setting. Let  $\hat{q}_t$  be any measurable function of  $y_{1:\Delta}$ . Define the function  $\phi_t \coloneqq \mathbb{I}\{\hat{q}_t \geq 1/2\}$ . Then by Proposition 17, we have that
$$
\max  \left\{\mathbb {P} \left(\phi_ {t} = 1\right), \mathbb {Q} \left(\phi_ {t} = 0\right) \right\} \geq \frac {1}{4} \exp (- \beta), \tag {6}
$$
where  $\beta$  is as defined in Eq.(5).
Notice that if  $q_{t} = 1 / 2 - \delta$ , then  $L_{t}(\hat{q}_{t}) \geq 1 / 2$  for any  $\hat{q}_t \geq 1 / 2$ . Similarly if  $q_{t} = 1 / 2 + \delta$ , we have that  $L_{t}(\hat{q}_{t}) \geq 1 / 2$  for any  $\hat{q}_t < 1 / 2$ .
Further note that  $L_{t}(q_{t}) = 1 / 2 - \delta$  by construction.
For notational clarity define  $L_{t}^{p}(x) \coloneqq x$  and  $L_{t}^{q}(x) \coloneqq 1 - x$ . We can lower-bound the instantaneous regret as:
$$
\begin{array}{l} E \left[ L _ {t} \left(\hat {q} _ {t}\right) \right] - L _ {t} \left(q _ {t}\right) = _ {(a)} \frac {1}{2} \left(E _ {\mathbb {P}} \left[ L _ {t} ^ {p} \left(\hat {q} _ {t}\right) \right] - L _ {t} ^ {p} (1 / 2 - \delta)\right) + \frac {1}{2} \left(E _ {\mathbb {Q}} \left[ L _ {t} ^ {q} \left(\hat {q} _ {t}\right) \right] - L _ {t} ^ {q} (1 / 2 + \delta)\right) \\ \geq_ {(b)} \frac {1}{2} \left(E _ {\mathbb {P}} \left[ L _ {t} ^ {p} (\hat {q} _ {t}) \mid \hat {q} _ {t} \geq 1 / 2 \right] - L _ {t} ^ {p} (1 / 2 - \delta) \mathbb {P} \left(\phi_ {t} = 1\right) \right. \\ + \frac {1}{2} \left(E _ {\mathbb {Q}} \left[ L _ {t} ^ {q} (\hat {q} _ {t}) \right| \hat {q} _ {t} <   1 / 2 \right] - L _ {t} ^ {q} (1 / 2 + \delta) \mathbb {Q} \left(\phi_ {t} = 0\right) \\ \geq_ {(c)} \frac {1}{2} \delta \mathbb {P} (\phi_ {t} = 1) + \frac {1}{2} \delta \mathbb {Q} (\phi_ {t} = 0) \\ \geq \delta / 2 \max  \left\{\mathbb {P} \left(\phi_ {t} = 1\right), \mathbb {Q} \left(\phi_ {t} = 0\right) \right\} \\ \geq_ {(d)} \frac {\delta}{8} \exp (- \beta), \\ \end{array}
$$