SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Comparison with [8]: The seminal work of [8] started the work on non-stationary estimation in the absence of corruptions. They establish a regret lower bound of  $\Omega (\Phi^{1 / 3}T^{2 / 3})$  for non-stationary estimation in the presence of non-stationarities. They also established an upper bound by showing that SGD without clipping yields matching regret upper bounds in expectation. The upper bound in Theorem 5.1 on the other hand gives high-probability regret bounds that hold even in the presence of corruptions, in addition to non-stationarities. Expressing an upper bound in high-probability is both technically challenging and algorithmically insightful. The insight we make is that for having high-probability bounds, we need to have clipped SGD. On the other hand, since [8] only give bounds holding in expectation, they do not need any clipping in their algorithms. The necessity of clipping in heavy-tailed settings is not an artifact of analysis, but is crucial for good empirical performance, as noted in recent works of [55]. Thus a conceptual contribution we make is that even in the absence of corruptions, a different algorithm compared to [8], namely that of clipping gradients is required to get regret bounds holding in high-probability. From a technical perspective, the proofs for high-probability bounds need different techniques as compared to [8]. For instance, we need several martingale and induction arguments to arrive at high-probability bounds in heavy tails while [8] have a much more simpler proofs just based on convexity, since they only give bounds in expectation.
Robust Stochastic Optimization The paper of [44] proves high-probability bounds for optimizing convex functions under heavy tailed gradients. However, they do not consider corruptions or drifts and assume finite diameter. In contrast our bounds explicitly depend on drifts and corruptions and can handle infinite diameter in the absence of corruptions. The work of [25] give high-probability bounds for offline clipped gradient descent based algorithms for optimizing both convex and strongly convex functions. However, they do not consider drift or corruptions. The work of [60] studies convergence of gradient descent methods in the absence of second moments, but only give bounds in expectation and do not consider drift or corruptions. The paper of [15] give high-probability bounds for stochastic optimization, but their analysis does not consider drift or corruptions. None of these papers simultaneously consider drifts, corruptions and heavy-tailed noise. The paper of [63] extend this to heavy-tailed noise under only the  $p \in (1, 2]$ th moment and non-convex functions. However, they only give results in expectation and in a setting without drift and corruptions.
Robust Statistics There is a growing literature [18, 37, 39, 10] that give near-optimal offline algorithms in the presence of corruptions but no drifts. Since our focus is to devise online algorithms that are adaptive to distribution shift, these algorithms are not directly applicable to our setting.
Dynamic Regret Minimization A parallel line of work concerns algorithms being adaptive to drifts for online convex optimization problems [27] where the data are arbitrary rather than sampled from a distribution [67, 65, 56, 5]. However, these algorithms assume that there are no noise in the gradients and no corruptions on the data stream.
Gradient clipping The work of [45] showed clipping gradients as a practical work-around to the 'exploding gradients' problem. Theoretically, [62] study the effect of clipping in offline training under restrictive noise assumptions and do not consider impact of drifts and corruptions.
11 Key algorithmic challenge
All distribution changes initially look like corruptions
In this section, we give a simple example to see whay developing algorithms that are robust to drift and corruptions are challenging. Consider a simple example of estimating the mean in the absence of noise, i.e., when variance is 0. We will show two possible underlying scenarios that yield the same observation to the forecaster. In both scenarios, the observed samples are all 0 in the first  $T - \Lambda_T$  rounds and is  $R \neq 0$  in all of the last  $\Lambda_T$  rounds. The first scenario is one where the true mean in the first  $T - \Lambda_T$  samples is 0 and the true mean in the last  $\Lambda_T$  samples is  $R$ , i.e., this scenario has no adversarial corruptions and a cumulative distribution shift of  $R$ . In the second scenario, the true mean for all rounds is 0, except the last  $\Lambda_T$  samples are adversarily corrupted, i.e., this scenario has no distribution shift but has  $\Lambda_T$  adversially corrupted samples. If the true system is scenario 1, then the regret of a forecaster is the distance from the ground-truth  $R$ , while in the second scenario is the distance from the un-corrupted mean 0. Thus, any algorithm will incur in at-least one of the scenarios, a regret of at-least  $\frac{R\Lambda_T}{2}$ . This example highlights the main tension in the problem: a
(c)
Figure 3: Over  $T = 10000$  steps, we vary  $\Phi_T$  and  $\Lambda_T$  keeping the clipping value  $\lambda = 2T^{0.25}$  for linear regression. We observe that the best choice of  $\alpha$  decreases as the distribution shift  $\Phi_T$  and/or corruptions  $\Lambda_T$  increases.
distribution shift initially presents as outliers and the online algorithm needs samples and thus incurs regret in order to learn this to be the 'new normal'.
12 Additional details on simulations
We give more details on experiments reported in Figures 2 and 3 for synthetic data.
Mean-Estimation Setup At each time  $t$ , we sample a 0 mean univariate random variable  $Y_{t}$  from the Pareto distribution with mean 0, variance  $\sigma^2 \coloneqq 5$  and tail parameter 3, so that it has a finite second moment. The un-corrupted  $d$ -dimensional vector at time  $t$  is denoted by  $Z_{t} \coloneqq \frac{(Y_{t} - \mu_{t})}{\sqrt{d}}\mathbf{v}_{t}$ , where  $\mathbf{v}_t$  is a unit vector sampled uniformly on the sphere of  $d = 256$  dimensions. The true means  $\mu_t$  are chosen by running a random walk, i.e.,  $\mu_0 = 0$  and  $\mu_t \coloneqq \mu_{t-1} + w_t$ , where  $w_t \sim \mathcal{N}\left(0, \frac{1}{T^{1-\kappa}}\mathbb{I}_d\right)$ . This ensures that  $\Phi_T \approx T^\kappa$ . In Figures 2a, 2b, 2c, 2d, we set  $\kappa \in \{0.5, 0.25, 0.15, 0\}$  respectively. The instants of corruption are chosen at integer multiples of  $\lfloor \sqrt{T} \rfloor$  and the observed sample  $X_{t} = 0$  if  $t = k\lfloor \sqrt{T} \rfloor$ , for some non-negative integer  $k$ . Otherwise,  $X_{t} = Z_{t}$ . For a given underlying mean vector, we run the streaming setup 30 times, and plot the average along with 95% confidence interval on the regret estimates.
Linear Regression Setup: We generate the covariate  $X_{t} \in \mathbb{R}^{256}$  from a generalized Pareto distribution with mean 0, variance  $\sigma^2 \coloneqq 1$ , i.e.  $m = 1$ . The true parameter at time 0 was set to  $\theta_0 \coloneqq [d^{-0.5}, \dots, d^{-0.5}]$ . At each time  $t$ , the true parameter  $\theta_t^* \coloneqq \mu_{t-1} + w_t$  where  $w_t \sim \mathcal{N}(0, \frac{1}{T^{1-\kappa}}\mathbb{I}_d)$ . This ensures that  $\Phi_T \approx T^\kappa$ . In Figures 3a, 3b, 3c, 3d, we set  $\kappa \in \{0.5, 0.25, 0.15, 0\}$  respectively. At each time  $t$ , the response  $y_t = \langle X_t, \theta_t^* \rangle + n_t$ , where  $n_t$  is a Pareto distribution with 0 mean, variance  $\sigma^2 = 3$  and shape parameter 2.5. The instants of corruption are chosen at integer multiples of  $\lfloor \sqrt{T} \rfloor$ . At instants of corruption, the covariates are sampled from a normal distribution with identity covariance and mean 100. The response variable for a corrupted time point is set to 0. For a given set of underlying mean vector, we run the streaming setup 30 times, and plot the average along with 95% confidence interval on the regret estimates.
Methods: We compare clipped-SGD with learning rates  $\eta_t \coloneqq 1 / mT^\alpha$  and clipping values  $\lambda = 2T^\beta$  for a variety of  $\alpha$  and  $\beta$ . Further we implement the doubling trick given in Algorithm 2 so that clipped-SGD does not require knowledge of the time-horizon  $T$ . In addition, we also compare against the clipped-SGD version of [55] which uses the time-varying learning rate of  $1/(t)$  learning and clipping value of  $2\sqrt{T}$ . Note that the algorithm of [55] has strictly more information than the one we implement as it has knowledge of time horizon  $T$  for setting the clipping value, while the one we propose does not and relies on the doubling trick. Further, we do not assume that the set  $\Theta$  is known and consider the un-projected clipped-SGD for our algorithm, while for that of [55], we consider the smallest  $L2$  ball around the horizon that contains all the  $(\theta_t^*)_{t=1}^T$ . Thus, the algorithm we implement is truly parameter-free as it does not have any problem specific parameters in its implementation.
Observations: All plots in Figure 2 show both mean-estimation and linear regression for varying amounts of distribution shift  $\Phi_T$ , while keeping the number of corruptions  $\Lambda_T = \sqrt{T}$  and clipping value  $\lambda = 2T^{0.25}$ . For both mean-estimation in Figures 2a, 2b, 2c, 2d and linear regression in Figures
(a)
(e)
Figure 4: Over  $T = 10000$  steps, we vary  $\Phi_T$  while keeping the number of corruptions  $\Lambda_T = \sqrt{T}$  and the clipping value  $\lambda = 2T^{0.25}$  the same. The top rows  $(a) - (d)$  are for the synthetic mean-estimation and the bottom figures  $(e) - (h)$  are for linear regression.
3a, 3b, 3c, 3d, we show that as the amount of distribution shift  $\Phi_T$  increases, the learning rate needs to be more aggressive to be adaptive to the distribution shift.
Sample paths In Figure 4, we plot a particular sample path over time for a variety of settings of  $\alpha$ .
13 Real-data experiments
The key result of this section is that even on real-data with real performance metrics, adapting the learning-rate to the amount of distribution shift on a data-stream is beneficial. We consider two tasks to demonstrate this: online binary classification and online anomaly detection.
Dataset	Stream-length T	Task	Dimension d
ElectricityNSW [26, 4]	45312	Binary classification	13
MiniBoone [49, 4]	130065	Binary Classification	50
MNIST [34]	11811	Anomaly Detection	784
Table 3: Real data-sets used.
13.1 Classification setting
Datasets: We consider two binary classification datasets available from the UCI repository [4] (i) NSW Electricity data [26] and (ii) MiniBoone data [49]. The NSW Electricity dataset consists of 45312 samples with each data-point having 6 numeric features and one categorical feature representing the day of week. After one-hot-encoding the categorical feature, we have 13 numeric features and a binary target. The MiniBoone dataset has 130065 samples with each data point having 50 numeric features and a binary target. We choose these two as they have been shown to be good binary classification benchmark for data-streams with drift [40]. In particular, the MiniBoone dataset is extreme where the first 36499 has target of 1 while all the last 93565 data points have target of 0. For both the datasets, the order of streaming is the same order in which they are collected. This is consistent with the evaluation protocol of streaming algorithms in the presence of drifts [40].
(c)
Figure 5: The plot of Regret with different choice of learning rate. In Figures  $(a)$  and  $(b)$ , we plot the log(Regret), where lower is better, while in  $(c)$  we plot the Precision-Recall area under curve, where higher is better. We see for the Electricity and MNIST dataset that  $\alpha \approx 0.5$  has the best performance, while for the MiniBoone dataset in  $(b)$ , we see that  $\alpha = 0$  has the best performance. The result for MiniBoone is not surprising as that dataset is has very little noise ( $\sigma \approx 0$ ) compared to the Electricity dataset. Thus, the optimal choice of  $\alpha$  is lower in MiniBoone compared to the Electricity data. Furthermore, across datasets, tasks and models, our theoretical insights hold demonstrating that they are not artifacts of analysis.
Model and Methods: For both data-sets, we consider a simple one layer Logistic regression as the model to train. We set the gradient clipping value  $\lambda = 1$  for both datasets. For both these datasets, we deviate from our setup and employ the classical online learning evaluation [40]. At each time  $t$ , the covariate is first shown to the algorithm which then predicts the target. The prediction of the logistic regression is then threshold at 0.5. The instantaneous loss at time  $t$  is the indicator loss whether if predictor after applying the threshold matches the target. The covariate and label pair is then used to take one step of clipped SGD in order to get the model used at the next time step. We experiment with different learning rates and plot their impact in Figure 5.
Observations: We observe in Figure 5 that for the Electricity dataset which has more shifts and is noisy  $\sigma >0$ , the optimal choice of  $\alpha \approx 0.5$ , while for the MiniBoone dataset has just one change point (i.e.,  $\Phi_T\approx 0$ ) and is noiseless  $\sigma \approx 0$  (cf [40]). Thus the optimal  $\alpha \approx 0$ .
13.2 Online Anomaly Detection
Dataset : We considered the standard MNIST dataset [34]. We modified it to a stream containing drift and corruptions/anomalies as follows. We introduce abrupt change points by first streaming in all 0's followed by 1's and so on with the last of the stream being 9's. This way, the stream has 9 abrupt change-points. In addition, within a change-point, the images are slowly rotating clockwise at a fixed angle of 1 degree rotation to model slowly varying change in between two abrupt changes. Anomalies are introduced at random with probability 0.05 by sampling a digit other than the current segment of time. For instance 5c, the third point on the stream is an anomaly since the digit is different from 0 which is the inlier digit till the first break-point. Similarly, the last but one the data point on the stream is an anomaly since it is different from 9, the inlier digit in that segment.
Methods: We flatten the image into a vector of 784 and consider a simple 2 layer auto-encoder with hidden dimension 512 and activation function of ReLU [66]. We initialize the network to be random at the beginning of the stream and consider clipped SGD with clip-value set to 5 for various learning rates as shown in Figure 5c. At each time  $t$ , the sample  $X_{t}$  is revealed to the algorithm, a single clipped gradient step is taken on this sample to update the model and then the anomaly score is produced from the resulting model.
**Observation:** In Figure 5c, we plot the anomaly detection accuracy measured through the average precision recall score, where higher the score indicates better AD performance. We see that neither too small nor too high a value of  $\alpha$  obtains the highest performance.
Figure 6: The data stream of MNIST digits with drifts and corruptions. The stream is ordered such that all digits labeled 0 arrive before digits labeled 1 and so-on. The abrupt change points are time instants when the next digit starts appearing. Within an abrupt change-point corresponding to a digit  $i \in \{0, \dots, 9\}$  and time-step  $t$ , with probability 0.95, a digit  $i$  image is sampled from the MNIST dataset without replacement, rotated by angle  $q_{t}$ . With probability 0.05, an image corresponding to a digit other than  $i$  is sampled, which is labeled (but unknown to the algorithm) as a true anomaly. These images are marked with the red-boundary in the figure. The rotation angle varies with time as  $q_{t+1} := q_{t} + \frac{1}{\sqrt{t}}$ , i.e., the rotations are slowly varying. The goal of the algorithm is to detect the true anomalies. In this experiment (see also Table 3),  $T = 1181$  and the number of anomalies was 563.
14 Removing the knowledge of time horizon  $T$  from Algorithm 1
In this section, we re-state the doubling trick from [11] to remove the dependence of the stream length  $T$  on the learning rate tuning in Theorems 4.3 and 5.1.
Algorithm 2 Online-Clipped-SGD without time horizon
1: Input: Learning rate exponent  $\alpha \in [0,1]$ , clipping exponent  $\beta \leq \frac{1}{2}$ ,  $\theta_0 \in \Theta$  initialization,  $m, M$ 
2: TIME-HORIZON  $\leftarrow 1$ ,  $\eta \leftarrow 1$ ,  $\lambda \leftarrow 2m(G + 1)$ 
3: for each time  $t = 1, 2, \dots, \mathbf{do}$ 
4: if  $t > \text{TIME-HORIZON then}$ 
5: TIME-HORIZON  $\leftarrow 2*\text{TIME-HORIZON}$ 
6:  $\eta \leftarrow \frac{1}{m\text{TIME-HORIZON}^{\alpha}}$ .
7:  $\lambda \leftarrow 2m(G + 1)\text{TIME-HORIZON}^{\beta}$ 
8: end if
9: Receive sample  $X_t$ 
10: Output  $\theta_t \leftarrow \mathcal{P}_{\Theta}(\theta_{t-1} - \eta\text{clip}(\nabla\mathcal{L}(X_t, \theta_{t-1}), \lambda))$ 
11: end for
Proposition 14.1 ([11]). If for a given  $\alpha, \beta, \delta, T$ , clipped-SGD satisfies  $\mathcal{R}_T \leq \Re(\alpha, \beta, T) \ln \left( \frac{T}{\delta} \right)$  with probability at-least  $1 - \delta$ , then Algorithm 2 will satisfy regret  $\mathcal{R}_T \leq \Re(\alpha, \beta, T)(\log_2(T))^2 \ln \left( \frac{2}{\delta} \right)$  with probability at-least  $1 - \delta$ .
Proof. Over a time interval of  $T$ , there are  $\log_2(T)$  times when Line 5 of Algorithm 2 is executed. Denote by the time-interval between two successive executions of Line 5 as a 'phase' of the algorithm. In the  $i$ th phase, the hypothesis of the proposition gives that with probability at-most  $\delta$ , the regret in phase  $i$  denoted by  $\mathcal{R}^{(i)} \geq \Re(\alpha, \beta, T) \ln \left(\frac{2^i}{\delta}\right)$ . Thus, taking an union bound over all the  $\log_2(T)$
phases, we get that with probability at-least  $1 - \log_2(T)\delta$ , the total regret  $\mathcal{R}_T$  is bounded by
$$
\begin{array}{l} \mathcal {R} _ {T} \leq \sum_ {i = 1} ^ {\log_ {2} (T)} \Re (\alpha , \beta , T) \ln \left(\frac {2 ^ {i}}{\delta}\right), \\ \leq \mathcal {R} _ {T} \leq \Re (\alpha , \beta , T) (\log_ {2} (T)) ^ {2} \ln \left(\frac {2}{\delta}\right). \\ \end{array}
$$
15 Proof of Proposition 6.1
The proof is by construction. For each time  $t = 1,2,\dots$ , denote by the function  $f_{t}(x)\coloneqq \frac{1}{2} (x - 1 - H_{t})^{2}$ , where  $H_{t}\coloneqq \sum_{s = 1}^{t - 1}\frac{1}{s}$ . Let the initial point  $\theta_0 = 0$  and let us set the clipping value of clipped gradient descent to be larger than or equal to 1.
We will argue by induction that  $\theta_t = H_t + \frac{1}{t}$  for all  $t = 1, 2, \dots$ . Thus, the loss suffered at time  $t$  is  $l_t := 1 - \frac{1}{t}$ , since the optimal point at all times  $t$  is  $1 + H_t$ .
For the base-case of time 1, we see that the gradient of  $f_{1}(\cdot)$  at  $x = \theta_0 = 0$  is  $-1$ . Thus,  $\theta_{1} = 1 = H_{2}$  and the loss at time 1 is  $0 = 1 - \frac{1}{1}$ . Assume the induction hypothesis is true for all  $s = 1, \dots, t$  for some  $t$ . Consider the time  $t + 1$ . The induction hypothesis tells that at time  $t$ ,  $\theta_t = H_{t + 1}$ . At time  $t + 1$ , the function revealed is  $(x - 1 - H_{t + 1})^2$  and thus the gradient at any  $x = H_{t + 1}$  is  $-1$ . The step size at time  $t + 1$  is  $\eta_{t + 1} = \frac{1}{t + 1}$  and thus  $\theta_{t + 1} = H_{t + 1} + \frac{1}{t + 1} = H_{t + 2}$ . Thus, the induction hypothesis holds.
This gives that the total cumulative regret is
$$
\begin{array}{l} \mathcal {R} _ {T} = \sum_ {t = 1} ^ {T} \left(1 - \frac {1}{t}\right), \\ \geq T - 2 \ln (T), \\ \geq \frac {T}{3}. \\ \end{array}
$$
16 Proofs of Lower Bounds on Estimation Regret
Before giving the proof of Proposition 2.6, we state and prove a simpler result that finite diameter is necessary in the presence of both non-stationarities and corruptions.
Proposition 16.1 (Finite diameter is necessary in general). There exists a strongly convex loss function  $\mathcal{L}$  such that for every  $\mathcal{D} > 0$ , domain  $\Theta = [-\mathcal{D}/2, \mathcal{D}/2] \subset \mathbb{R}$  and every  $T \in \mathbb{N}$ , such that for every algorithm, there exists a sequence of probability measures  $(\mathbb{P}_t)_{t=1}^T$  and corruptions  $(\mathcal{C}_t)_{t=1}^T$  where regret is at-least  $\Omega(\Lambda_T\mathcal{D})$  with probability 1.
We prove the following Proposition on mean-estimation, which will prove Proposition 16.1.
Proposition 16.2 (Mean estimation needs finite diameter). Let  $\Theta = [-\mathcal{D} / 2,\mathcal{D} / 2]\subset \mathbb{R}$ , time-horizon  $T$  and the number of corruptions be  $\Lambda_T\leq T$  and  $\mathcal{L}(Z,\theta) = \frac{1}{2} (Z - \theta)^2$ . For every algorithm, there exists a sequence of probability measures  $(\mathbb{P}_t)_t = 1$  and corruption functions  $(\mathcal{C}_t)_t = 1$ , such that the regret incurred is at-least  $\Omega (\Lambda_T\mathcal{D})$  with probability 1.
Proof. The crux of this lower-bound is that the time instants of corruptions are adversarially chosen. The proof follows by construction where the underlying ground-truth is one of two possibilities and there is no noise, i.e., the variance of the probability distributions chosen by the adversary are all 0. We will show that even against an oblivious adversary,  $\Omega (\Lambda_T\mathcal{D})$  regret is un-avoidable.
Assume that the adversary has only two choices - either an environment in which the true mean at all times is  $\mathcal{D} / 2$  or one in which the true mean in the first  $T - \Lambda_T$  samples is  $\mathcal{D} / 2$  and the true mean in the last  $\Lambda_T$  samples is  $-\mathcal{D} / 2$ . If the adversary picks the first scenario then it does not corrupt any sample and since there is no noise, all the  $T$  observed samples in the first scenario will be equal to  $\mathcal{D} / 2$ . If
the adversary picks the second scenario as the environment, then it will corrupt the last  $\Lambda_T$  samples to show as  $\mathcal{D} / 2$  instead of the true  $-\mathcal{D} / 2$ . Thus, in either choice of the adversary the observed set of  $T$  samples by the forecaster is deterministic  $\mathcal{D} / 2$  for all  $T$  samples. Thus, the forecaster cannot distinguish whether the underlying environment is from scenario 1 or 2 from observations, even if the forecaster has knowledge of the two possible situations from which the adversary is choosing.
Thus, for any deterministic choice of outputs by the algorithm, the adversary can choose one of these two situations such that the regret on the last  $\Lambda_T$  samples of the stream incurs regret at-least  $\frac{\Lambda_T\mathcal{D}}{4}$ .
16.1 Proof of Proposition 2.6
Similar to Proposition 2.6, consider two scenarios for mean-estimation. In one scenario, the uncorrupted samples are all drawn from a Dirac mass at 0, but the first  $\Lambda_T$  samples are corrupted with all  $d$  coordinates set to  $\mathcal{D} / \sqrt{d}$ . In the other scenario, there are no corruptions and the un-corrupted samples are all from Dirac mass at location with all coordinates  $\mathcal{D} / \sqrt{d}$ . In both situations, the first  $\Lambda_T$  samples are identical. Thus, no estimator in the first  $\Lambda_T$  samples can distinguish between these two scenarios and will incur regret at-least  $(\Lambda_T\mathcal{D}) / 4$ .
17 Useful convexity based inequalities
In this section, we collect some inequalities that we will repeatedly use throughout the proofs. Throughout the rest of the paper, for every  $t \in [T]$ , we denote by  $\mathcal{R}_t(\theta) \coloneqq \mathbb{E}_{Z \sim \mathbb{P}_t}[\mathcal{L}(Z, \theta)]$  and by  $\theta_t^* \coloneqq \arg \min_{\theta \in \Theta} \mathcal{R}_t(\theta)$ . Convexity of  $\mathcal{R}_t(\cdot)$  and the domain  $\Theta$  being convex implies that  $\theta_t^*$  is well-defined, exists and is unique. Recall that for all time  $t$ ,  $\mathcal{R}_t(\theta)$  is strongly convex with parameters  $M, m$  respectively. Thus, we know that