Title: Online Label Shift: Optimal Dynamic Regret meets Practical Algorithms
Venue: NeurIPS 2023
Accepted: yes
Keywords: online learning

label shift

distribution shift

unsupervised domain adaptation
Abstract: This paper focuses on supervised and unsupervised online label shift,
where the class marginals $Q(y)$ varies
but the class-conditionals $Q(x|y)$ remain invariant. In the unsupervised setting, our goal is to adapt a learner, trained on some offline labeled data, to changing label distributions given unlabeled online data. In the supervised setting, we must both learn a classifier and adapt to the dynamically evolving class marginals given only labeled online data. We develop novel algorithms that reduce the adaptation problem to online regression and guarantee optimal dynamic regret without any prior knowledge of the extent of drift in the label distribution. Our solution is based on bootstrapping the estimates of *online regression oracles* that track the drifting proportions. Experiments across numerous simulated and real-world online label shift scenarios demonstrate the superior performance of our proposed approaches, often achieving 1-3% improvement in accuracy while being sample and computationally efficient. Code is publicly available at https://github.com/Anon-djiwh/OnlineLabelShift

Online Label Shift: Optimal Dynamic Regret meets Practical Algorithms

Abstract

This paper focuses on supervised and unsupervised online label shift, where the class marginals Q ( y ) varies but the class-conditionals Q ( x | y ) remain invariant. In the unsupervised setting, our goal is to adapt a learner, trained on some offline labeled data, to changing label distributions given unlabeled online data. In the supervised setting, we must both learn a classifier and adapt to the dynamically evolving class marginals given only labeled online data. We develop novel algorithms that reduce the adaptation problem to online regression and guarantee optimal dynamic regret without any prior knowledge of the extent of drift in the label distribution. Our solution is based on bootstrapping the estimates of online regression oracles that track the drifting proportions. Experiments across numerous simulated and real-world online label shift scenarios demonstrate the superior performance of our proposed approaches, often achieving 1-3% improvement in accuracy while being sample and computationally efficient. Code is publicly available at this url.

1 Introduction

Supervised machine learning algorithms are typically developed assuming independent and identically distributed (iid) data. However, real-world environments evolve dynamically [55, 67, 46, 29]. Absent further assumptions on the nature of the shift, such problems are intractable. One line of research has explored causal structures such as covariate shift [64], label shift [61, 51], and missingness shift [86], for which the optimal target predictor is identified from labeled source and unlabeled target data. Let's denote the feature-label pair of an example by ( x, y ) . Label shift addresses the setting where the label marginal distribution Q ( y ) may change but the conditional distribution Q ( x | y ) remains fixed. Most prior work addresses the batch setting for unsupervised adaptation, where a single shift occurs between a source and target population [61, 51, 1, 2, 1, 26]. However, in the real world, shifts are more likely to occur continually and unpredictably, with data arriving in an online fashion. A nascent line of research tackles online distribution shift, typically in settings where labeled data is available in real time [4], seeking to minimize the dynamic regret .

Researchers have only begun to explore the role that structures like label shift might play in such online settings. Initial attempts to learn under unsupervised online label shifts were made by Wu et al. [76] and Bai et al. [8], both of which rely on reductions to Online Convex Optimization (OCO) [37, 54]. This line of research aims in updating a classification model based on online data so that the overall regret is controlled. However, Wu et al. [76] only control for static regret against a fixed

```

offline data

model

Nature

classifier (or model) in hindsight and makes the assumption of the convexity (of losses), which is often violated in practice. In the face of online label shift, where the class marginals can vary across rounds, a more fitting notion is to control the dynamic regret against a sequence of models in hindsight. Motivated by this observation, Bai et al. [8] control for the dynamic regret. However, their approach is based on updating model parameters (of the classifier) with online gradient descent and relying on convex losses limits the applicability of their methods (e.g. algorithms in Bai et al. [8] can not be employed with decision tree classifiers).

In this paper, we study the problem of learning classifiers under Online Label Shift (OLS) in both supervised and unsupervised settings (Fig.1). In both these settings, the distribution shifts are an online process that respects the label shift assumption. Our primary goal is to develop algorithms that sidestep convexity assumptions and at the same time optimally adapt to the non-stationarity in the label drift. In the Unsupervised Online Label Shift (UOLS) problem, the learner is provided with a pool of labeled offline data sampled iid from the distribution Q 0 ( x, y ) to train an initial model f 0 . Afterwards, at every online round t , few unlabeled data points sampled from Q t ( x ) are presented. The goal is to adapt f 0 to the non-stationary target distributions Q t ( x, y ) so that we can accurately classify the unlabelled data. By contrast, in Supervised Online Label Shift (SOLS), our goal is to learn classifiers from only the (labeled) samples that arrive in an online fashion from Q t ( x, y ) at each time step, while simultaneously adapting to the non-stationarity induced due to changing label proportions. While SOLS is similar to online learning under non-stationarity, UOLS differs from classical online learning as the test label is not seen during online adaptation. Below are the list of contributions of this paper.

Unsupervised adaptation. For the UOLS problem, we provide a reduction to online regression (see Defn. 1), and develop algorithms for adapting the initial classifier f 0 in a computationally efficient way leading to minimax optimal dynamic regret. Our approach achieves the best-ofboth worlds of Wu et al. [76], Bai et al. [8] by controlling the dynamic regret while allowing us to use expressive black-box models for classification (Sec. 3).

Supervised adaptation. We develop algorithms for SOLS problem that lead to minimax optimal dynamic regret without assuming convexity of losses (Sec. 4). Our theoretically optimal solution is based on weighted Empirical Risk Minimization (wERM) with weights tracked by online regression. Motivated by our theory, we also propose a simple continual learning baseline which achieves empirical performance competitive to the wERM from scratch at each time step across several semi-synthetic SOLS problems while being 15 × more efficient in computation cost.

Low switching regressors. We propose a black-box reduction method to convert an optimal online regression algorithm into another algorithm that switches decisions sparingly while maintaining minimax optimality . This method is relevant for online change point detection. We demonstrate its application in developing SOLS algorithms to train models only when significant distribution drift is detected, while maintaining statistical optimality (App. D and Algorithm 6).

Extensive empirical study. We corroborate our theoretical findings with experiments across numerous simulated and real-world OLS scenarios spanning vision and language datasets (Sec. 5). Our proposed algorithms often improve over the best alternatives in terms of both final accuracy and label marginal estimation. This advantage is particularly prominent with limited initial holdout data (in the UOLS problem) highlighting the sample efficiency of our approach.

Notes on technical novelties. Even-though online regression is a well studied technique, to the best of our knowledge, it is not used before to address the problem of online label shift. It is precisely the usage of regression which lead to tractable adaptation algorithms while side-stepping convexity assumptions thereby allowing us to use very flexible models for classification. This is in stark contrast

to OCO based reductions in [76] and [8]. We propose new theoretical frameworks and identify the right set of assumptions for materializing the reduction to online regression. It was not evident initially that this link would lead to minimax optimal dynamic regret rates as well as consistent empirical improvement over prior works. Proof of the lower bounds requires adapting the ideas from non-stationary stochastic optimization [10] in a non-trivial manner. Further, none of the proposed methods require the prior knowledge of the extent of distribution drift.

2 Problem Setup

Let X ⊆ R d be the input space and Y = [ K ] := { 1 , 2 , . . . , K } be the output space. Let Q be a distribution over X × Y and let q ( · ) denotes the corresponding label marginal. ∆ K is the K -dimensional simplex. For a vector v ∈ R K , v [ i ] is its i th coordinate. We assume that we have a hypothesis class H . For a function f ∈ H : X → ∆ K , we also use f ( i | x ) to indicate f ( x )[ i ] . With ℓ ( f ( x ) , y ) , we denote the loss of making a prediction with the classifier f on ( x, y ) . L denotes the expected loss, i.e., L = E ( x,y ) ∼ Q [ ℓ ( f ( x ) , y )] . ˜ O ( · ) hides dependencies in absolute constants and poly-logarithmic factors of horizon and failure probabilities.

In this work, we study online learning under distribution shift, where the distribution Q t ( x, y ) may continuously change with time. Throughout the paper, we focus on the label shift assumption where the distribution over label proportions q t ( y ) can change arbitrarily but the distribution of the covariate conditioned on a label value (i.e., Q t ( x | y ) ) is assumed to be invariant across all time steps. We refer to this setting as Online Label Shift (OLS). Here, we consider settings of unsupervised and supervised OLS settings captured in Frameworks 1 and 3 respectively. In both settings, at round t a sample ( x t , y t ) is drawn from a distribution with density Q t ( x t , y t ) . In the UOLS setting, the label is not revealed to the learner. However, we assume access to offline labeled data sampled iid from Q 0 which we use to train an initial classifier f 0 . The goal is to adapt the initial classifier f 0 to drifting label distributions. In contrast, for the SOLS setting, the label is revealed to the learner after making a prediction and the goal is to learn a classifier f t ∈ H for each time step.

Next, we formally define the concept of online regression which will be central to our discussions. Simply put, an online regression algorithm tracks a ground truth sequence from noisy observations.

Definition 1 (online regression) . Fix any T > 0 . The following interaction scheme is defined to be the online regression protocol.

At round t ∈ [ T ] , an algorithm predicts ˆ θ t ∈ R K .

A noisy version of ground truth z t = θ t + ϵ t is revealed where θ t , ϵ t ∈ R K , and ∥ ϵ t ∥ 2 , ∥ θ t ∥ 2 ≤ B . Further the noise ϵ t are independent across time with E [ ϵ t ] = 0 and Var( ϵ t [ i ]) ≤ σ 2 ∀ i ∈ [ K ] .

An online regression algorithm aims to control ∑ T t =1 ∥ ˆ θ t -θ t ∥ 2 2 . Moreover, the regression algorithm is defined to be adaptively minimax optimal if with probability at least 1 -δ , ∑ n t =1 ∥ ˆ θ t -θ t ∥ 2 2 = ˜ O ( T 1 / 3 V 2 / 3 T ) without knowing V T ahead of time. Here V T := ∑ T t =2 ∥ θ t -θ t -1 ∥ 1 is termed as the Total Variation (TV) of the sequence θ 1: T .

3 Unsupervised Online Label Shift

In this section, we develop a framework for handling the UOLS problem. We summarize the setup in Framework 1. Since in practice, we may need to work with classifiers such as deep neural networks or decision trees, we do not impose convexity assumptions on the (population) loss of the classifier as a function of the model parameters. Despite the absence of such simplifying assumptions, we provide performance guarantees for our label shift adaption techniques so that they are certified to be fail-safe.

Under the label shift assumption, we have Q t ( y | x ) as a re-weighted version of Q 0 ( y | x ) :

where the second equality is due to the label shift assumption. Hence, a reasonable strategy is to reweight the initial classifier f 0 with label proportions (estimate) at the current step, since we only have to correct the label distribution shift. This re-weighting technique is widely used for offline label shift correction [51, 2, 1] and for learning under label imbalance [41, 74, 18].

Framework 1 Unsupervised Online Label Shift (UOLS) protocol

Input : Initial classifier f 0 : X → ∆ K trained on offline labeled dataset { ( x i , y i ) } N i =1 sampled iid from Q 0 ;

1: f 1 = f 0

2: for each round t ∈ [ T ] do

3: Nature samples x t ∈ X and y t ∈ Y , with ( x t , y t ) ∼ Q t ; Only x t is revealed to the learner.

4: Learner predicts a label i ∼ f t ( x t ) ∈ ∆ K .

5: f t +1 = A ( f 0 , x 1: t ) , where A is strategy to adapt the classifier based on past data.

6: end for

Input : i) Online regression oracle ALG; ii) Initial classifier f 0 ; iii) The confusion matrix C ; iv) The label marginal q 0 ∈ D of the training distribution;

1: At round t , get the classifier covariate x t .

2: Let ̂ q t = Π D ( ALG ( s 1: t -1 )) , where Π D ( x ) = argmin y ∈D ∥ y -x ∥ 2 .

3: Sample a label i with probability ∝ ̂ q t ( i ) q 0 ( i ) f 0 ( i | x t ) .

4: Let s t = C -1 f 0 ( x t ) .

5: Update the online regression oracle with the estimate s t .

Our starting point in developing a framework is inspired by Wu et al. [76], Bai et al. [8] . For selfcontainedness, we briefly recap their arguments next. We refer interested readers to their papers for more details. Wu et al. [76] considers a hypothesis class of re-weighted initial classifier f 0 . The loss of a hypothesis is parameterised by the re-weighting vector. They use tools from OCO to optimise the loss and converge to a best fixed classifier. However as noted in Wu et al. [76], the losses are not convex with respect to the re-weight vector in practice. Hence usage of OCO techniques is not fully satisfactory in their problem formulation.

In a complementary direction, Bai et al. [8] abandons the idea of re-weighting. Instead, they update the parameters of a model at each round using online gradient descent and a loss function whose expected value is assumed to be convex with respect to model parameters. They provide dynamic regret guarantees against a sequence of changing model parameters in hindsight, and connects it to the variation of the true label marginals. More precisely, they provide algorithms with ∑ T t =1 L t ( w t ) -L t ( w ∗ t ) to be well controlled where w ∗ t is the best model parameter to be used at round t and L t is a (population level) loss function. However, there are some scopes for improvement in this direction as well. For example, the convexity assumption can be easily violated when working with interpretable models based on decision trees, or if we want to retrain few final layers of a deep classifier based on new data. Further as noted in the experiments (Sec. 5), their methods based on retraining the classifier require more data than re-weighting based methods. Our experiments also indicate that re-weighting can be computationally cheaper than re-training without sacrificing the classifier accuracy.

Thus, on the one hand, the work of Wu et al. [76] allows us to use the power of expressive initial classifiers while only controlling the static regret against a fixed hypothesis. On the other hand, the work of Bai et al. [8] allows controlling the dynamic regret while limiting the flexibility of deployed models. We next provide our framework for handling label shifts that achieves the best of both worlds by controlling the dynamic regret while allowing the use of expressive blackbox models.

In summary, we estimate the sequence of online label marginals and leverage the idea of re-weighting an initial classifier as in Wu et al. [76]. In particular, given an estimate ̂ q t ( y ) of the true label marginal at round t , we compute the output of the re-weighted classifier f t as ̂ q t ( y ) q 0 ( y ) f 0 ( y | x ) / Z where Z = ∑ y ̂ q t ( y ) q 0 ( y ) f 0 ( y | x ) . However, to get around the issue of non-convexity, we separate out the process of estimating the re-weighting vectors via a reduction to online regression which is a welldefined and convex problem with computationally efficient off-the-shelf algorithms readily available. Second, and more importantly, Wu et al. [76] competes with the best fixed re-weighted hypothesis. However, in the problem setting of label shift, the true label marginals are in fact changing. Hence, we control the dynamic regret against a sequence of re-weighted hypotheses in hindsight. All proofs for the next sub-section are deferred to App. C.

3.1 Proposed algorithm and performance guarantees

We start by presenting our assumptions. This is followed by the main algorithm for UOLS and its performance guarantees. Similar to the treatment in Bai et al. [8], we assume the following.

Assumption 1. Assume access to the true label marginals q 0 ∈ ∆ K of the offline training data and the true confusion matrix C ∈ R K × K with C ij = E x ∼ Q 0 ( ·| y = j ) , f 0 ( i | x ) . Further the minimum singular value σ min ( C ) = Ω(1) is bounded away from zero.

As noted in prior work [51, 26], the invertibility of the confusion matrix holds whenever the classifier f 0 has good accuracy and the true label marginal q 0 assigns a non-zero probability to each label. Though we assume perfect knowledge of the label marginals of the training data and the associated confusion matrix, this restriction can be easily relaxed to their empirical counterparts computable from the training data. The finite sample error between the empirical and population quantities can be bounded by O (1 / √ N ) where N is the number of initial training data samples. To this end, we operate in the regime where the time horizon obeys T = O ( √ N ) . However, similar to Bai et al. [8], we make this assumption mainly to simplify presentation without trivializing any aspect of the OLS problem.

Next, we present our assumptions on the loss function. Let p ∈ ∆ K . Consider a classifier that predicts a label ̂ y ( x ) , by sampling ̂ y ( x ) according to the distribution that assigns a weight p ( i ) q 0 ( i ) f 0 ( i | x ) to the label i . Define L t ( p ) to be any non-negative loss that ascertains the quality of the marginal p . For example, L t ( p ) = E [ ℓ (ˆ y ( x ) , y )] where the expectation is taken wrt the randomness in the draw ( x, y ) ∼ Q t and in sampling ˆ y ( x ) . Here ℓ is any classification loss (e.g. 0-1, cross-entropy).

Assumption 2 (Lipschitzness of loss functions) . Let D be a compact and convex domain. Assume that L t ( p ) is G Lipschitz with p ∈ D ⊆ ∆ K , i.e, L t ( p 1 ) -L t ( p 2 ) ≤ G ∥ p 1 -p 2 ∥ 2 for any p 1 , p 2 ∈ D . The constant G need not be known ahead of time.

We show in Lemmas 11 and 12 that the above assumption is satisfied under mild regularity conditions. Furthermore, the prior works such as Wu et al. [76] and Bai et al. [8] also require that losses are Lipschitz with a known Lipschitz constant apriori to set the step sizes for their OGD based methods.

The main goal here is to design appropriate re-weighting estimates such that the dynamic regret :

is controlled where ˆ q t ∈ ∆ K is the estimate of the true label marginal q t . Thus we have reduced the problem of handling OLS to the problem of online estimation of the true label marginals.

Under label shift, we can get an unbiased estimate of the true marginals at any round via the techniques in Lipton et al. [51], Azizzadenesheli et al. [2], Alexandari et al. [1]. More precisely, s t = C -1 f 0 ( x t ) has the property that E [ s t ] = q t (see Lemma 15). Further, the variance of the estimate s t is bounded by 1 /σ 2 min ( C ) . Unfortunately, these unbiased estimates can not be directly used to track the moving marginals q t . This is because the total squared error ∑ T t =1 E [ ∥ s t -q t ∥ 2 2 ] grows linearly in T as the sum of the variance of the point-wise estimates accumulates unfavorably over time.

To get around these issues, one can use online regression algorithms such as FLH [38] with online averaging base learners or the Aligator algorithm [7]. These algorithms use ensemble methods to (roughly) output running averages of s t where the variation in the true label marginals is small enough. The averaging within intervals where the true marginals change slowly helps to reduce the overall variance while injecting only a small bias. We use such online regression oracles to track the moving marginals and re-calibrate the initial classifier. Overall, Algorithm 2 summarizes our method which has the following performance guarantee.

Theorem 2. Suppose we run Algorithm 2 with the online regression oracle ALG as FLH-FTL (App. F) or Aligator [7]. Then under Assumptions 1 and 2, we have

where V T := ∑ T t =2 ∥ q t -q t -1 ∥ 1 and the expectation is taken with respect to randomness in the revealed co-variates. Further, this result is attained without prior knowledge of V T .

Remark 3. We emphasize that any valid online regression oracle ALG can be plugged into Algorithm 2. This implies that one can even use transformer-based time series models to track the moving marginals q t . Further, we have the flexibility of choosing the initial classifier to be any black-box model that outputs a distribution over the labels.

Framework 3 Supervised Online Label Shift (SOLS) protocol

input A hypothesis class H .

1: for each round t ∈ [ T ] do

2: Nature samples N iid data points x t, 1: N ∈ X and y t, 1: N ∈ Y , with each ( x t,i , y t,i ) ∼ Q t ; x t, 1: N is revealed to the learner.

3: For each i ∈ [ N ] , learner predicts a label f t ( x t,i ) .

4: The label y t,i ∈ Y for each i ∈ [ N ] is revealed.

5: f t +1 = A ( f t , { x 1: t, 1: N , y 1: t, 1: N } ) where algorithm A updates the classifier with past data.

6: end for

We next show that the performance guarantee in Theorem 2 is optimal (modulo factors of log T ) in a minimax sense.

Theorem 5. Let V T ≤ 64 T . There exists a loss function, a domain D (in Assumption 2), and a choice of adversarial strategy for generating the data such that for any algorithm, we have ∑ T t =1 E ([ L t (ˆ q t )] -L t ( q t )) = Ω ( max { T 2 / 3 V 1 / 3 T , √ T } ) , where ˆ q t ∈ D is the weight estimated by the algorithm and q t ∈ D is the label marginal at round t chosen by the adversary. Here the expectation is taken with respect to the randomness in the algorithm and the adversary.

4 Supervised Online Label Shift

In this section, we focus on the SOLS problem where the labels are revealed to the learner after it makes decisions. Framework 3 summarizes our setup. Let f ∗ t := argmin f ∈H L t ( f ) be the population minimiser. We aim to control the dynamic regret against the best sequence of hypotheses in hindsight:

If the SOLS problem is convex, it reduces to OCO [37, 54] and existing works provide ˜ O ( T 2 / 3 V 1 / 3 T ) dynamic regret guarantees [82]. However, in practice, since loss functions are seldom convex with respect to model parameters in modern machine learning, the performance bounds of OCO algorithms cease to hold true. In our work, we extend the generalization guarantees of ERM from statistical learning theory [11] to the SOLS problem. All proofs of next sub-section are deferred to App. E.

4.1 Proposed algorithms and performance guarantees

We start by providing a simple initial algorithm whose computational complexity and flexibility will be improved later. Note that due to the label shift assumption, for any j, t ∈ [ T ] , we have E ( x,y ) ∼ Q t [ ℓ ( f ( x ) , y )] = E ( x,y ) ∼ Q j [ q t ( y ) q j ( y ) ℓ ( f ( x ) , y ) ] . Here we assume that the true label marginals q t ( y ) > 0 for all t ∈ [ T ] and all y ∈ [ K ] . Based on this, we propose a simple weighted ERM approach (Algorithm 4) where we use an online regression oracle to estimate the label marginals from the (noisy) empirical label marginals computed with observed labeled data. With weighted

Algorithm 4 TrainByWeights to handle SOLS

input Online regression oracle ALG, hypothesis class H

1: At round t ∈ [ T ] , get estimated label marginal ˆ q t from ALG ( s 1: t -1 ) .

2: Update the hypothesis with weighted ERM:

3: Get co-variates x t, 1: N and make predictions with f t

4: Get labels y t, 1: N

5: Compute s t [ i ] = 1 N ∑ N j =1 I { y t,j = i } for all i ∈ [ K ] .

6: Update ALG with the empirical label marginals s t .

ERM and plug-in estimates of importance weights, we can obtain our classifier f t . One can expect that by adequately choosing the online regression oracle ALG, the risk of the hypothesis f t computed will be close to that of f ∗ t . Here the degree of closeness will also depend on the number of data points seen thus far. Consequently, Algorithm 4 controls the dynamic regret (Eq.(3)) in a graceful manner. We have the following performance guarantee:

Theorem 6. Suppose the true label marginal satisfies min t,k q t ( k ) ≥ µ > 0 . Choose the online regression oracle in Algorithm 4 as FLH-FTL (App. F) or Aligator from Baby et al. [7] with its predictions clipped such that ˆ q t [ k ] ≥ µ . Then with probability at least 1 -δ , Algorithm 4 produces hypotheses with R H dynamic = ˜ O ( T 2 / 3 V 1 / 3 T + √ T log( |H| /δ ) ) , where V T = ∑ T t =2 ∥ q t -q t -1 ∥ 1 . Further, this result is attained without any prior knowledge of the variation budget V T .

The above rate contains the sum of two terms. The second term is the familiar rate seen in the supervised statistical learning theory literature under iid data [11]. The first term reflects the price we pay for adapting to distributional drift in the label marginals. While we prove this result for finite hypothesis sets, the extension to infinite sets is direct by standard covering net arguments [70].

Remark 7. Theorem 6 requires that the estimates of the label marginals to be clipped from below by µ . This is done only to facilitate theoretical guarantees by enforcing that the importance weights used in Eq. (2) do not become unbounded. However, note that only the labels we actually observe enters the objective in Eq. (2) . In particular, if a label has very low probability of getting sampled at a round, then it is unlikely that it enters the objective. Due to this reason, in our experiments, we haven't used the clipping operation (see Section 5 and Appendix F for more details).

The proof of the theorem uses concentration arguments to establish that the risk of the hypothesis f t is close to the risk of the optimal f ∗ t . However, unlike the standard offline supervised setting with iid data, for any fixed hypothesis, the terms in the summation of Eq.(2) are correlated through the estimates of the online regression oracle. We handle it by introducing uncorrelated surrogate random variables and bounding the associated discrepancy. Next, we show (near) minimax optimality of the guarantee in Theorem 6.

Theorem 8. Let V T ≤ T/ 8 . There exists a choice of hypothesis class, loss function, and adversarial strategy of generating the data such that R H dynamic = Ω ( T 2 / 3 V 1 / 3 T + √ T log( |H| ) ) , where the expectation is taken with respect to randomness in the algorithm and adversary.

Remark 9. Though the rates in Theorems 5 and 8 are similar, we note that the corresponding regret definitions are different. Hence the minimax rates are not directly comparable between the supervised and unsupervised settings.

Even-though Algorithm 4 has attractive performance guarantees, it requires retraining with weighted ERM at every round. This can be computationally expensive. To alleviate this issue, we design a new online change point detection algorithm (Algorithm 5 in App. D) that can adaptively discover time intervals where the label marginals change slow enough. We show that the new online change point detection algorithm can be used to significantly reduce the number of retraining steps without sacrificing statistical efficiency (up to constants). Due to space constraints, we defer the exact details to App. D. We remark that our change point detection algorithm is applicable to general online regression problems and hence can be of independent interest to online learning community.

Remark 10. Algorithm 5 helps to reduce the run-time complexity. However, both Algorithms 4 and 5 have the drawback of storing all data points accumulated over the online rounds. This is reminiscent to FTL / FTRL type algorithms from online learning. We leave the task of deriving theoretical guarantees with reduced storage complexity under non-convex losses as an important future direction.

5 Experiments 2

5.1 UOLS Setup and Results

Setup Following the dataset setup of Bai et al. [8], we conducted experiments on synthetic and common benchmark data such as MNIST [50], CIFAR-10 [49], Fashion [77], EuroSAT [40], Arxiv [15], and SHL [31, 71]. For each dataset, the original data is split into labeled data available

10 2

10 3

10 4

Number of Holdout Data

0.12

0.13

0.14

0.15

0.16

0.17

Classification Error

ROGD

UOGD

FTH

FLH-FTL (ours)

(a)

10 2

10 3

10 4

Number of Holdout Data

0.10

0.15

0.20

0.25

0.30

Mean Square Error

ROGD

FTH

FLH-FTL (ours)

(b)

UOGD

ATLAS

ROGD

FLH-FTL

0.80

0.82

0.84

0.86

0.88

0.90

0.92

0.94

0.879

0.895

0.876

0.897

0.875

0.890

0.880

0.907

40k training data

49.75k training data

(c)

during offline training and validation, and the unlabeled data that we observe during online learning. We experiment with varying sizes of holdout offline data which is used to obtain the confusion matrix and update the model parameters to adapt to OLS to probe the sample efficiency of all the methods. In contrast to previous works [8, 76], we have chosen to use a smaller amount of holdout offline data for our main experiments. We made this decision because the standard practice for deployment involves training and validating models on training and holdout splits, respectively (e.g., with k-fold cross-validation). Then, the final model is deployed by training on all available data (i.e., the union of train and holdout) with the identified hyperparameters. However, to employ UOLS techniques in practice, practitioners must hold out data that was not seen during training to update the model during online adaptation. Therefore, methods that are efficient with respect to the amount of offline holdout data required might be preferable.

For all datasets except SHL, we simulate online label shifts with four types of shifts studied in Bai et al. [8]: monotone shift, square shift, sinusoidal shift, and Bernoulli shift. For SHL locomotion, we use the real-world shift occurring over time. For architectures, we use an MLP for Fashion, SHL and MNIST, Resnets [39] for EuroSAT, CINIC, and CIFAR, and DistilBERT [62, 75] based models for arXiv. For alternate approaches, along with a base classifier (which does no adaptation) and oracle classifier (which reweight using the true label marginals), we make comparisons with adaptation algorithms proposed in prior works [76, 8]. In particular, we compare with ROGD, FTH, FTFWH from Wu et al. [76] and UOGD, ATLAS from Bai et al. [8]. For brevity, we refer to our method as FLH-FTL (though strictly speaking, our methods are based on FLH from Hazan and Seshadhri [38] with online averages as base learners). We run all the online label shift experiments with the time horizon T = 1000 and at each step 10 samples are revealed. We repeat all experiments with 3 seeds to obtain means and standard deviations of the results. For other methods that perform reweighting correction on softmax predictions, we use the labeled holdout data to calibrate the model with temperature scaling, which tunes one temperature parameter [35]. We provide exact details about the datasets, label shift simulations, models, and prior methods in App. F.

Results Overall, across all datasets, we observe that our method FLH-FTL performs better than alternative approaches in terms of both classification error and mean squared error for estimating the label marginal. Note that methods that directly update the model parameters (i.e., UOGD, ATLAS) do not provide any estimate of the label marginal (Table 1). UOGD and ATLAS also require offline holdout labeled data (i.e., from time step 0) to make online updates to the model parameters. For this purpose, we use the same labeled data that we use to compute the confusion matrix.

As we increase the holdout offline labeled dataset size for updating the model parameters (and to compute the confusion matrix), we observe that classification error and MSE with FLH-FTL stay (relatively) constant whereas the classification errors of other alternatives improve (Fig. 2). This highlights that FLH-FTL can be much more sample efficient with respect to the size of the hold-out offline labeled data. Motivated by this observation, we perform an additional experiment in which we increase the offline training data and observe that we can overall improve the classification accuracy significantly with FLH-FTL (Fig. 2). We present results on SHL dataset with similar findings on semisynthetic datasets in App. G.5. Finally, we also experiment with a random forest model on the MNIST dataset. Note methods that update model parameters (e.g., UOGD and ATLAS) with OGD are not applicable here. Here, we also observe that we improve over existing applicable alternatives (Table 2).

5.2 SOLS setup and results

Setup For the supervised problem, we experiment with MNIST and CIFAR datasets. We simulate a time horizon of T = 200 . For each dataset, at each step, we observe 50 samples with Bernoulli shift. Motivated by our theoretical results with weighted ERM, we propose a simple baseline which continually trains the model at every step instead of starting ERM from scratch every time. We maintain a pool of all the labeled data received till that time step, and at every step, we randomly sample a batch with uniform label marginal to update the model. Finally, we re-weight the updated softmax outputs with estimated label marginal. We call this method Continual Training via ReSampling (CT-RS). Its relation as a close variant of weighted ERM is elaborated in App. F.1. To estimate the label marginal, we try FTH and ours FLH-FTL.

Results On both datasets, we observe that empirical performance with CT-RS improves over the naive continual training baseline. Additionally, CT-RS results are competitive with weighted ERM

while being 5-15 × faster in terms of computation cost (we include the exact computational cost in App. F.1). Moreover, as in UOLS setup, we observe that FLH-FTL improves over FTH for both target label marginal estimation and classification.

6 Conclusion

In this work, we focused on unsupervised and supervised online label shift settings. For both settings, we developed algorithms with minimax optimal dynamic regret. Experimental results on both real and semi-synthetic datasets substantiate that our methods improve over prior works both in terms of accuracy and target label marginal estimation.

In future work, we aim to expand our experiments to more real-world label shift datasets. Our work also motivates future work in exploiting other causal structures (e.g. covariate shift) for online distribution shift problems.

References

A. Alexandari, A. Kundaje, and A. Shrikumar. Adapting to label shift with bias-corrected calibration. In International Conference on Machine Learning (ICML) , 2021.

K. Azizzadenesheli, A. Liu, F. Yang, and A. Anandkumar. Regularized learning for domain adaptation under label shifts. In International Conference on Learning Representations (ICLR) , 2019.

D. Baby and Y.-X. Wang. Online forecasting of total-variation-bounded sequences. In Neural Information Processing Systems (NeurIPS) , 2019.

D. Baby and Y.-X. Wang. Optimal dynamic regret in exp-concave online learning. In COLT , 2021.

D. Baby and Y.-X. Wang. Optimal dynamic regret in proper online learning with strongly convex losses and beyond. AISTATS , 2022.

D. Baby and Y.-X. Wang. Second order path variationals in non-stationary online learning. AISTATS , 2023.

D. Baby, X. Zhao, and Y.-X. Wang. An optimal reduction of tv-denoising to adaptive online learning. AISTATS , 2021.

Y. Bai, Y.-J. Zhang, P. Zhao, M. Sugiyama, and Z.-H. Zhou. Adapting to online label shift with provable guarantees. In Advances in Neural Information Processing Systems , 2022.

J. Bekker and J. Davis. Learning from positive and unlabeled data: a survey. Machine Learning , 2020. URL https://doi.org/10.1007%2Fs10994-020-05877-5 .

O. Besbes, Y. Gur, and A. Zeevi. Non-stationary stochastic optimization. Operations research , 63(5):1227-1244, 2015.

O. Bousquet, S. Boucheron, and G. Lugosi. Introduction to statistical learning theory. In Advanced Lectures on Machine Learning , Lecture Notes in Computer Science. Springer, 2003.

N. Cesa-Bianchi and G. Lugosi. Prediction, Learning, and Games . Cambridge University Press, New York, NY, USA, 2006. ISBN 0521841089.

T.-J. Chang and S. Shahrampour. On online optimization: Dynamic regret analysis of strongly convex and smooth problems. Proceedings of the AAAI Conference on Artificial Intelligence , 2021.

X. Chen, Y. Wang, and Y.-X. Wang. Technical note - non-stationary stochastic optimization under lp, q-variation measures. Operations Research , 2018.

C. B. Clement, M. Bierbaum, K. P. O'Keeffe, and A. A. Alemi. On the use of arxiv as a dataset. arXiv preprint arXiv:1905.00075 , 2019.

C. Cortes and M. Mohri. Domain adaptation and sample bias correction theory and algorithm for regression. Theoretical Computer Science , 519, 2014.

C. Cortes, Y. Mansour, and M. Mohri. Learning Bounds for Importance Weighting. In Advances in Neural Information Processing Systems (NIPS) , 2010.

Y. Cui, M. Jia, T.-Y. Lin, Y. Song, and S. Belongie. Class-balanced loss based on effective number of samples. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages 9268-9277, 2019.

A. Daniely, A. Gonen, and S. Shalev-Shwartz. Strongly adaptive online learning. In International Conference on Machine Learning , pages 1405-1411, 2015.

D. Donoho and I. Johnstone. Ideal spatial adaptation by wavelet shrinkage. Biometrika , 81(3): 425-455, 1994.

D. Donoho and I. Johnstone. Minimax risk over ℓ p -balls for ℓ q -error. Probability Theory and Related Fields , 99(2):277-303, 1994.

D. Donoho, R. Liu, and B. MacGibbon. Minimax risk over hyperrectangles, and implications. Annals of Statistics , 18(3):1416-1437, 1990.

D. L. Donoho and I. M. Johnstone. Minimax estimation via wavelet shrinkage. The annals of Statistics , 26(3):879-921, 1998.

C. Elkan and K. Noto. Learning classifiers from only positive and unlabeled data. In International Conference Knowledge Discovery and Data Mining (KDD) , pages 213-220, 2008.

P. Gaillard and S. Gerchinovitz. A chaining algorithm for online nonparametric regression. In Conference on Learning Theory , pages 764-796, 2015.

S. Garg, Y. Wu, S. Balakrishnan, and Z. Lipton. A unified view of label shift estimation. In Advances in Neural Information Processing Systems (NeurIPS) , 2020.

S. Garg, Y. Wu, A. Smola, S. Balakrishnan, and Z. Lipton. Mixture proportion estimation and PU learning: A modern approach. In Advances in Neural Information Processing Systems (NeurIPS) , 2021.

S. Garg, S. Balakrishnan, and Z. Lipton. Domain adaptation under open set label shift. In Advances in Neural Information Processing Systems (NeurIPS) , 2022.

S. Garg, S. Balakrishnan, Z. Lipton, B. Neyshabur, and H. Sedghi. Leveraging unlabeled data to predict out-of-distribution performance. In International Conference on Learning Representations (ICLR) , 2022.

S. Garg, N. Erickson, J. Sharpnack, A. Smola, S. Balakrishnan, and Z. Lipton. Rlsbench: Domain adaptation under relaxed label shift. In International Conference on Machine Learning (ICML) , 2023.

H. Gjoreski, M. Ciliberto, L. Wang, F. J. O. Morales, S. Mekki, S. Valentin, and D. Roggen. The university of sussex-huawei locomotion and transportation dataset for multimodal analytics with mobile devices. IEEE Access , 6:42592-42604, 2018.

G. Goel and A. Wierman. An online algorithm for smoothed regression and lqr control. In Proceedings of the Twenty-Second International Conference on Artificial Intelligence and Statistics , 2019.

A. Gretton, A. J. Smola, J. Huang, M. Schmittfull, K. M. Borgwardt, and B. Schölkopf. Covariate Shift by Kernel Mean Matching. Journal of Machine Learning Research (JMLR) , 2009.

A. Guntuboyina, D. Lieu, S. Chatterjee, and B. Sen. Adaptive risk bounds in univariate total variation denoising and trend filtering. Annals of Statistics , 2020.

C. Guo, G. Pleiss, Y. Sun, and K. Q. Weinberger. On calibration of modern neural networks. In International conference on machine learning , pages 1321-1330. PMLR, 2017.

J. Guo, M. Gong, T. Liu, K. Zhang, and D. Tao. Ltf: A label transformation framework for correcting label shift. In International Conference on Machine Learning , pages 3843-3853. PMLR, 2020.

E. Hazan. Introduction to online convex optimization. Foundations and Trends® in Optimization , 2(3-4):157-325, 2016.

E. Hazan and C. Seshadhri. Adaptive algorithms for online decision problems. In Electronic colloquium on computational complexity (ECCC) , volume 14, 2007.

K. He, X. Zhang, S. Ren, and J. Sun. Deep Residual Learning for Image Recognition. In Computer Vision and Pattern Recognition (CVPR) , 2016.

P. Helber, B. Bischke, A. Dengel, and D. Borth. Eurosat: A novel dataset and deep learning benchmark for land use and land cover classification. IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing , 12(7):2217-2226, 2019.

C. Huang, Y. Li, C. C. Loy, and X. Tang. Learning deep representation for imbalanced classification. In Proceedings of the IEEE conference on computer vision and pattern recognition , pages 5375-5384, 2016.

A. Jacobsen and A. Cutkosky. Parameter-free mirror descent. COLT , 2022.

A. Jadbabaie, A. Rakhlin, S. Shahrampour, and K. Sridharan. Online optimization: Competing with dynamic comparators. In Artificial Intelligence and Statistics , pages 398-406, 2015.

I. M. Johnstone. Gaussian estimation: Sequence and wavelet models, 2017. URL: https: //imjohnstone.su.domains//GE12-27-11.pdf . Last visited on 2023/04/20.

S.-J. Kim, K. Koh, S. Boyd, and D. Gorinevsky. ℓ 1 trend filtering. SIAM Review , 51(2):339360, 2009.

P. W. Koh, S. Sagawa, H. Marklund, S. M. Xie, M. Zhang, A. Balsubramani, W. Hu, M. Yasunaga, R. L. Phillips, I. Gao, T. Lee, E. David, I. Stavness, W. Guo, B. A. Earnshaw, I. S. Haque, S. Beery, J. Leskovec, A. Kundaje, E. Pierson, S. Levine, C. Finn, and P. Liang. WILDS: A benchmark of in-the-wild distribution shifts. In International Conference on Machine Learning (ICML) , 2021.

W. M. Koolen, A. Malek, P. L. Bartlett, and Y. Abbasi. Minimax time series prediction. Advances in Neural Information Processing Systems (NIPS'15) , pages 2557-2565, 2015.

W. Kotłowski, W. M. Koolen, and A. Malek. Online isotonic regression. In Annual Conference on Learning Theory (COLT-16) , volume 49, pages 1165-1189. PMLR, 2016.

A. Krizhevsky and G. Hinton. Learning Multiple Layers of Features from Tiny Images. Technical report, Citeseer, 2009.

Y. LeCun, L. Bottou, Y. Bengio, and P. Haffner. Gradient-Based Learning Applied to Document Recognition. Proceedings of the IEEE , 86, 1998.

Z. Lipton, Y.-X. Wang, and A. Smola. Detecting and correcting for label shift with black box predictors. In International Conference on Machine Learning , 2018.

E. Mammen and S. van de Geer. Locally adaptive regression splines. Annals of Statistics , 25(1): 387-413, 1997.

M. Mohri, A. Rostamizadeh, and A. Talwalkar. Foundations of Machine Learning . The MIT Press, 2012.

F. Orabona. A modern introduction to online learning, 2019.

J. Quinonero-Candela, M. Sugiyama, A. Schwaighofer, and N. D. Lawrence. Dataset shift in machine learning . Mit Press, 2008.

A. Raj, P. Gaillard, and C. Saad. Non-stationary online regression, 2020.

A. Rakhlin and K. Sridharan. Online non-parametric regression. In Conference on Learning Theory , pages 1232-1264, 2014.

M. Roberts, P. Mani, S. Garg, and Z. Lipton. Unsupervised learning under latent label shift. In Advances in Neural Information Processing Systems (NeurIPS) , 2022.

V. Sadhanala, Y.-X. Wang, and R. Tibshirani. Total variation classes beyond 1d: Minimax rates, and the limitations of linear smoothers. Advances in Neural Information Processing Systems (NIPS-16) , 2016.

V. Sadhanala, Y.-X. Wang, and R. J. Tibshirani. Graph sparsification approaches for laplacian smoothing. In AISTATS'16 , pages 1250-1259, 2016.

M. Saerens, P. Latinne, and C. Decaestecker. Adjusting the Outputs of a Classifier to New a Priori Probabilities: A Simple Procedure. Neural Computation , 2002.

V. Sanh, L. Debut, J. Chaumond, and T. Wolf. Distilbert, a distilled version of bert: smaller, faster, cheaper and lighter. arXiv preprint arXiv:1910.01108 , 2019.

B. Schölkopf, D. Janzing, J. Peters, E. Sgouritsa, K. Zhang, and J. Mooij. On Causal and Anticausal Learning. In International Conference on Machine Learning (ICML) , 2012.

H. Shimodaira. Improving predictive inference under covariate shift by weighting the loglikelihood function. Journal of statistical planning and inference , 90(2):227-244, 2000.

A. Storkey. When training and test sets are different: characterizing learning transfer. Dataset shift in machine learning , 30:3-28, 2009.

R. J. Tibshirani. Adaptive piecewise polynomial estimation via trend filtering. Annals of Statistics , 42(1):285-323, 2014.

A. Torralba and A. A. Efros. Unbiased look at dataset bias. In Computer Vision and Pattern Recognition (CVPR), 2011 IEEE Conference on , pages 1521-1528. IEEE, 2011.

A. B. Tsybakov. Introduction to Nonparametric Estimation . Springer Publishing Company, Incorporated, 1st edition, 2008.

S. van de Geer. Estimating a regression function. Annals of Statistics , 18(2):907-924, 1990.

R. Vershynin. High-Dimensional Probability: An Introduction with Applications in Data Science . Cambridge University Press, 2018.

L. Wang, H. Gjoreski, M. Ciliberto, S. Mekki, S. Valentin, and D. Roggen. Enabling reproducible research in sensor-based transportation mode recognition with the sussex-huawei dataset. IEEE Access , 7:10870-10891, 2019.

Y.-X. Wang, J. Sharpnack, A. Smola, and R. Tibshirani. Trend filtering on graphs. In AISTATS'15 , pages 1042-1050, 2015.

Y.-X. Wang, J. Sharpnack, A. Smola, and R. J. Tibshirani. Trend filtering on graphs. Journal of Machine Learning Research , 17(105):1-41, 2016.

Y.-X. Wang, D. Ramanan, and M. Hebert. Learning to model the tail. Advances in neural information processing systems , 30, 2017.

T. Wolf, L. Debut, V. Sanh, J. Chaumond, C. Delangue, A. Moi, P. Cistac, T. Rault, R. Louf, M. Funtowicz, et al. Huggingface's transformers: State-of-the-art natural language processing. arXiv preprint arXiv:1910.03771 , 2019.

R. Wu, C. Guo, Y. Su, and K. Q. Weinberger. Online adaptation to label distribution shift. In Advances in Neural Information Processing Systems , 2021.

H. Xiao, K. Rasul, and R. Vollgraf. Fashion-mnist: a novel image dataset for benchmarking machine learning algorithms. arXiv preprint arXiv:1708.07747 , 2017.

T. Yang, L. Zhang, R. Jin, and J. Yi. Tracking slowly moving clairvoyant: optimal dynamic regret of online learning with true and noisy gradient. In International Conference on Machine Learning (ICML-16) , pages 449-457, 2016.

B. Zadrozny. Learning and Evaluating Classifiers Under Sample Selection Bias. In International Conference on Machine Learning (ICML) , 2004.

K. Zhang, B. Schölkopf, K. Muandet, and Z. Wang. Domain Adaptation Under Target and Conditional Shift. In International Conference on Machine Learning (ICML) , 2013.

L. Zhang, S. Lu, and Z.-H. Zhou. Adaptive online learning in dynamic environments. In Advances in Neural Information Processing Systems (NeurIPS-18) , pages 1323-1333, 2018.

L. Zhang, T. Yang, Z.-H. Zhou, et al. Dynamic regret of strongly adaptive methods. In International Conference on Machine Learning (ICML-18) , pages 5877-5886, 2018.

P. Zhao and L. Zhang. Improved analysis for dynamic regret of strongly convex and smooth functions. L4DC , 2021.

P. Zhao, Y. Zhang, L. Zhang, and Z.-H. Zhou. Dynamic regret of convex and smooth functions. NeurIPS , 2020.

P. Zhao, Y.-X. Wang, and Z.-H. Zhou. Non-stationary online learning with memory and nonstochastic control. AISTATS , 2022.

H. Zhou, S. Balakrishnan, and Z. C. Lipton. Domain adaptation under missingness shift. In Artificial Intelligence and Statistics (AISTATS) , 2022.

Appendix

A Limitations

Our work is based on the label shift assumption which restricts the applicability of our methods to scenarios where this assumption holds. However, as noted in Section 1, the problem of adaptation to changing data distribution is intractable without imposing assumptions on the nature of the shift.

Furthermore, as noted in Remark 9, our methods in the SOLS settings have a memory requirement that scales linearly with time, which may not be feasible in scenarios where memory is limited. This is reminiscent to FTL / FTRL type algorithms from online learning. We leave the task of deriving theoretical guarantees with reduced storage complexity under non-convex losses as an important future direction.

B Related work

Offline total variation denoising The offline problem of Total Variation (TV) denoising constitutes estimating the ground truth under the observation model in Definition 1 with the caveat that all observations are revealed ahead of time. This problem is well studied in the literature of locally adaptive non-parametric regression [20, 21, 23, 52, 69, 45, 66, 73, 60, 34]. The optimal total squared error (TSE) rate for estimation is known to be ˜ O ( T 1 / 3 V 2 / 3 T + 1) [22]. Estimating sequences of bounded TV has received a lot of attention in literature mainly because of the fact that these sequences exhibit spatially varying degree of smoothness. Most signals of scientific interest are known to contain spatially localised patterns [44]. This property also makes the task of designing optimal estimators particularly challenging because the estimator has to efficiently detect localised patterns in the ground truth signal and adjust the amount of smoothing to be applied to optimally trade-off bias and variance.

Non-stationary online learning The problem of online regression can be casted into the dynamic regret minimisation framework of online learning. We assume the notations in Definition 1. In this framework, at each round the learner makes a decision ˆ θ t . Then the learner suffers a squared error loss ℓ t ( ˆ θ t ) = ∥ z t -ˆ θ t ∥ 2 2 . The gradient of the loss at the point of decision, ∇ ℓ t ( ˆ θ t ) = 2( ˆ θ t -z t ) , is revealed to the learner. The expected dynamic regret against the sequence of comparators θ 1: T is given by

where in the last line we used the fact that the noise ϵ t (see Definition 1) is zero mean and independent of the online decisions ˆ θ t . Due to this relation, we conclude that any algorithm that can optimally control the dynamic regret with respect to squared error losses ℓ t ( x ) = ∥ z t -x ∥ 2 2 can be directly used to control the TSE from the ground truth sequence θ 1: T .

The minimax estimation rate is defined as follows:

Algorithms that can control the dynamic regret with respect to convex losses such as those presented in the works of Jadbabaie et al. [43], Besbes et al. [10], Yang et al. [78], Chen et al. [14], Zhang et al. [81], Goel and Wierman [32], Zhao et al. [84], Zhao and Zhang [83], Zhao et al. [85], Chang

and Shahrampour [13], Jacobsen and Cutkosky [42], Baby and Wang [6] can lead to sub-optimal estimation rates of order O ( √ T (1 + V T )) .

On the other hand, algorithms presented in Hazan and Seshadhri [38], Daniely et al. [19], Baby and Wang [3], Baby et al. [7], Raj et al. [56], Baby and Wang [4, 5] exploit the curvature of the losses and attain the (near) optimal estimation rate of ˜ O ( T 1 / 3 V 2 / 3 T +1) .

Online non-parametric regression The task of estimating a sequence of TV bounded sequence from noisy observations can be cast into the online non-parametric regression framework of Rakhlin and Sridharan [57]. Results on online non-parametric regression against reference class of Lipschitz sequences, Sobolev sequences and isotonic sequences can be found in [25, 47, 48] respectively. However as noted in Baby and Wang [3], these classes feature sequences that are more regular than TV bounded sequences. In fact they can be embedded inside a TV bounded sequence class [59] . So the minimax optimality of an algorithm for TV class implies minimax optimality for the smoother sequence classes as well.

Offline label shift Dataset shifts are predominantly studied under two scenarios: covariate shift and label shift [65]. Under covariate shift, p ( y | x ) doesn't change, while in label shift p ( x | y ) remains invariant. Schölkopf et al. [63] articulates connections between label shift and covariate shift with anti-causal and causal models respectively. Covariate shift is well explored in past [80, 79, 17, 16, 33]. The offline label shift assumption has been extensively studied in the domain adaptation literature [61, 65, 80, 51, 36, 26, 1, 30] and is also related to problems of estimating mixture proportions of different classes in unlabeled data where previously unseen classes may appear (e.g., positive and unlabeled learning; Elkan and Noto [24], Bekker and Davis [9], Garg et al. [27, 28], Roberts et al. [58]). Classical approaches suffer from the curse of dimensionality, failing in settings with high dimensional data where deep learning prevails. More recent methods work leverage black-box predictors to produce sufficient low-dimensional representations for identifying target distributions of interest [51, 2, 61, 1]. In our work, we leverage black box predictors to estimate label marginals at each step which we track with online regression oracles to trade off variance with (small) bias.

C Omitted proofs from Section 3

In the next two lemmas, we verify Assumption 2 for some important loss functions.

Lemma 11 (cross-entropy loss) . Consider a sample ( x, y ) ∼ Q . Let p ∈ R K + and ˜ p ( x ) ∈ ∆ K be a distribution that assigns a weight proportional p ( i ) q 0 ( i ) f 0 ( i | x ) to the label i . Let ℓ (˜ p ( x ) , y ) = ∑ K i =1 I { y = i } log(1 /p ( x )[ i ]) be the cross-entropy loss. Let L ( p ) := E ( x,y ) ∼ Q [ ℓ ( p ( x ) , y )] be its population analogue. Then L ( p ) is 2 √ K/µ Lipschitz in ∥·∥ 2 norm over the clipped box D := { p ∈ R K + : µ ≤ p ( i ) ≤ 1 ∀ i ∈ [ K ] } which is compact and convex. Further, the true marginals q t ∈ D whenever q t ( i ) ≥ µ for all i ∈ [ K ] .

Proof. We have

where we define w i := f 0 ( i | x ) /q 0 ( i ) . Then we can see that

So if min i p ( i ) ≥ µ , we have that w i ∑ K j =1 w j p ( j ) ≤ 1 /µ and Q t ( i | x ) /p ( i ) ≤ 1 /µ . So by triangle inequality, |∇ L ( p )[ i ] |≤ 1 /µ +1 /µ .

̸

Lemma 12 (binary 0-1 loss) . Consider a sample ( x, y ) ∼ Q . Let p ∈ R K + and ˜ p ( x ) ∈ ∆ K be a distribution that assigns a weight proportional p ( i ) q 0 ( i ) f 0 ( i | x ) to the label i . Let ˆ y ( x ) be a sample obtained from the distribution ˜ p ( x ) . Consider the binary 0-1 loss ℓ (ˆ y ( x ) , y ) = I (ˆ y ( x ) = y ) . Let L ( p ) := E ( x,y ) ∼ Q, ˆ y ( x ) ∼ ˜ p ( x ) I (ˆ y ( x ) = y ) be its population analogue. Let q 0 ( i ) ≥ α > 0 . Then L ( p ) is 2 K 3 / 2 / ( ατ ) Lipschitz in ∥·∥ 2 norm over the domain D := { p ∈ R K + : ∑ K i =1 p ( i ) f 0 ( i | x ) ≥ τ, p ( i ) ≤ 1 ∀ i ∈ [ K ] } which is compact and convex. Further, the true marginals q t ∈ D whenever q t ( i ) ≥ µ for all i ∈ [ K ] .

Proof. We have that

Similarly,

̸

̸

Denote ˜ p ( x )[ i ] = p ( i ) w i / ∑ K j =1 p ( j ) w j with w j := f 0 ( i | x ) /q 0 ( i ) . Then we see that

Thus we conclude that ∥∇ ˜ p ( x )[ i ] ∥ 2 ≤ 2 √ K/ ( ατ ) , where the gradient is taken with respect to p ∈ R K + .

Therefore,

Remark 13. The condition ∑ K i =1 f 0 ( i | x ) p ( i ) ≥ τ is closely related to Condition 1 of Garg et al. [26]. Note that this is strictly weaker than imposing the restriction that the distribution p ( i ) ≥ µ for each i .

Remark 14. We emphasize that the conditions in Lemmas 11 and 12 are only sufficient conditions that imply bounded gradients. However, they are not necessary for satisfying bounded gradients property.

Lemma 15. Let µ, ν ∈ ∆ K be such that µ [ i ] = q t ( i ) . Let s t = C -1 f 0 ( x t ) , where C is the confusion matrix defined in Assumption 1. We have that E [ s t ] = µ and Var ( s t ) ≤ 1 /σ 2 min ( C )

Proof. Let ˜ q t (ˆ y t ) = E x t ∼ Q t X , ˆ y ( x t ) ∼ f 0 ( x t ) I { ˆ y ( x t ) = ˆ y t } be the probability that the classifier f 0 predicts the label ˆ y t . Here Q X t ( x ) := ∑ K i =1 Q t ( x, i ) . Let's denote Q t (ˆ y ( x t ) = ˆ y t | y t = i ) := E x t ∼ Q t ( ·| y = i ) , ˆ y ( x t ) ∼ f 0 ( x t ) I { ˆ y ( x t ) = ˆ y t } . By law of total probability, we have that

where the last line follows by the label shift assumption.

Let µ, ν ∈ R K be such that µ [ i ] = q t ( i ) and ν [ i ] = ˜ q t ( i ) . Then the above equation can be represented as ν = Cµ . Thus µ = C -1 ν .

Given a sample x t ∈ Q t , the vector f 0 ( x t ) forms an unbiased estimate of ν . Hence we have that the vector ˆ µ := C -1 f 0 ( x t ) is an unbiased estimate of µ . Moreover,

Hence the variance of the estimate ˆ µ is bounded by 1 /σ 2 min ( C ) .

We have the following performance guarantee for online regression due to Baby et al. [7].

Proposition 16 (Baby et al. [7]) . Let s t = C -1 f 0 ( x t ) . Let ˆ q t := ALG ( s 1: t -1 ) be the online estimate of the true label marginal q t produced by the Aligator algorithm by taking s 1: t -1 as input at a round t . Then we have that

where V T := ∑ T t =2 ∥ q t -q t -1 ∥ 1 . Here ˜ O hides dependencies in absolute constants and polylogarithmic factors of the horizon. Further this result is attained without prior knowledge of the variation V T .

By following the arguments in Baby and Wang [4], a similar statement can be derived also for the FLH-FTL algorithm of Hazan and Seshadhri [38] (Algorithm 7).

Theorem 2. Suppose we run Algorithm 2 with the online regression oracle ALG as FLH-FTL (App. F) or Aligator [7]. Then under Assumptions 1 and 2, we have

where V T := ∑ T t =2 ∥ q t -q t -1 ∥ 1 and the expectation is taken with respect to randomness in the revealed co-variates. Further, this result is attained without prior knowledge of V T .

Proof. Owing to our carefully crafted reduction from the problem of online label shift to online regression, the proof can be conducted in just a few lines. Let ˜ q t be the value of ALG ( s 1: t -1 ) computed at line 2 of Algorithm 2. Recall that the dynamic regret was defined as:

Continuing from Eq.(4), we have

where the second line is due to non-expansivity of projection, the third line is due to Jensen's inequality, fourth line by Cauchy-Schwartz and last line by Proposition 16. This finishes the proof.

Next, we provide matching lower bounds (modulo log factors) for the regret in the unsupervised label shift setting. We start from an information-theoretic result which will play a central role in our lower bound proofs.

Proposition 17 (Theorem 2.2 in Tsybakov [68]) . Let P and Q be two probability distributions on H , such that KL ( P || Q ) ≤ β < ∞ , Then for any H -measurable real function ϕ : H → { 0 , 1 } ,

Theorem 5. Let V T ≤ 64 T . There exists a loss function, a domain D (in Assumption 2), and a choice of adversarial strategy for generating the data such that for any algorithm, we have ∑ T t =1 E ([ L t (ˆ q t )] -L t ( q t )) = Ω ( max { T 2 / 3 V 1 / 3 T , √ T } ) , where ˆ q t ∈ D is the weight estimated by the algorithm and q t ∈ D is the label marginal at round t chosen by the adversary. Here the expectation is taken with respect to the randomness in the algorithm and the adversary.

Proof. We start with a simple observation about KL divergence. Consider distributions with density P ( x, y ) = P 0 ( x | y ) p ( y ) and Q ( x, y ) = P 0 ( x | y ) q ( y ) where ( x, y ) ∈ R × [ K ] . Note that these distributions are consistent with the label shift assumption. We note that

Thus we see that under the label shift assumption, the KL divergence is equal to the KL divergence between the marginals of the labels.

Next, we define a problem instance and an adversarial strategy. We focus on a binary classification problem where the labels is either 0 or 1. As noted before, the KL divergence only depends on the

marginal distribution of labels. So we fix the density Q 0 ( x | y ) to be any density such that under the uniform label marginals ( q 0 (1) = q 0 (0) = 1 / 2 ) we can find a classifier with invertible confusion matrix (recall from Fig. 1 that Q 0 corresponds to the data distribution of the training data set).

Divide the entire time horizon T is divided into batches of size ∆ . So there are M := T/ ∆ batches (we assume divisibility). Let Θ = { 1 2 -δ, 1 2 + δ } be a set of success probabilities, where each probability can define a Bernoulli trial. Here δ ∈ (0 , 1 / 4) which will be tuned later.

The problem instance is defined as follows:

For batch i ∈ [ M ] , adversary selects a probability ˚ q i ∈ Θ uniformly at random.

For any round t that belongs to the i th batch, sample a label y t ∼ Ber ( q t ) and co-variate x t ∼ Q 0 ( ·| y t ) . Here q t =˚ q i . The co-variate x t is revealed.

Let ˆ q t be any estimate of q t at round t . Define the loss as L t (ˆ q t ) := I { q t ≥ 1 / 2 } (1 -ˆ q t ) + I { q t < 1 / 2 } ˆ q t .

We take the domain D in Assumption 2 as [1 / 2 -δ, 1 / 2 + δ ] . It is easy to verify that L t (ˆ q t ) is Lipschitz over D . Note that unlike Besbes et al. [10], we do not have an unbiased estimate of the gradient of loss functions.

Let's compute an upperbound on the total variation incurred by the true marginals. We have

where the last line is obtained by choosing δ = V T / (2 M ) = V T ∆ / (2 T ) .

Since at the beginning of each batch, the sampling probability is chosen uniformly at random, the loss function in the current batch is independent of the history available at the beginning of the batch. So only the data in the current batch alone is informative in minimising the loss function in that batch. Hence it is sufficient to consider algorithms that only use the data within a batch alone to make predictions at rounds that falls within that batch.

Now we proceed to bound the regret incurred within batch 1. The computation is identical for any other batches.

Let P be the joint probability distribution in which labels ( y 1 , . . . , y ∆ ) within batch 1 are sampled with success probability 1 / 2 -δ (i.e q t = 1 / 2 -δ )

Define an alternate distribution Q such that

According to the above distribution the data are independently sampled from Bernoulli trials with success probability 1 / 2 + δ . (i.e q t = 1 / 2 + δ )

Moving forward, we will show that by tuning ∆ appropriately, any algorithm won't be able to detect between these two alternate worlds with constant probability resulting in sufficiently large regret.

We first bound the KL distance between these two distributions. Let

where in line (a) we used the fact that log(1 + x ) ≤ x for x > -1 and observed that -4 δ/ (1 + 2 δ ) > -1 as δ ∈ (0 , 1 / 4) . In line (b) we used δ ∈ (0 , 1 / 4) .

Since P and Q are product of the marginals due to independence we have that

where we used the choices δ = ∆ V T / (2 T ) and ∆ = ( T/V T ) 2 / 3 .

Suppose at the beginning of batch, we reveal the entire observations within that batch y 1:∆ to the algorithm. Note that doing so can only make the problem easier than the sequential unsupervised setting. Let ˆ q t be any measurable function of y 1:∆ . Define the function ϕ t := I { ˆ q t ≥ 1 / 2 } . Then by Proposition 17, we have that

where β is as defined in Eq.(5).

Notice that if q t = 1 / 2 -δ , then L t (ˆ q t ) ≥ 1 / 2 for any ˆ q t ≥ 1 / 2 . Similarly if q t = 1 / 2 + δ , we have that L t (ˆ q t ) ≥ 1 / 2 for any ˆ q t < 1 / 2 .

Further note that L t ( q t ) = 1 / 2 -δ by construction.

For notational clarity define L p t ( x ) := x and L q t ( x ) := 1 -x . We can lower-bound the instantaneous regret as:

where in line (a) we used the fact the success probability for a batch is selected uniformly at random from Θ . In line (b) we used the fact that L p t (ˆ q t ) -L p t (1 / 2 -δ ) ≥ 0 since ˆ q t ∈ D = [1 / 2 -δ, 1 / 2+ δ ] . Similarly term involving L q t is also handled. In line (c) we applied ( E P [ L p t (ˆ q t ) | ˆ q t ≥ 1 / 2] -L p t (1 / 2 -δ )) ≥ δ since E P [ L p t (ˆ q t ) | ˆ q t ≥ 1 / 2] ≥ 1 / 2 and L p t (1 / 2 -δ ) = 1 / 2 -δ . Similar bounding is done for the term involving E Q as well. In line (d) we used Eq.(6).

Algorithm 5 LPA : a black-box reduction to produce a low-switching online regression algorithm input Online regression oracle ALG, failure probability δ , maximum standard deviation σ (see Definition 1).

1: Initialize prev = 0 ∈ R K , b = 1

2: Get estimate ˜ θ t from ALG ( z 1: t -1 )

3: Output ˆ θ t = prev

4: Receive an observation z t // test to detect non-staionarity

5: if ∑ t j = b +1 ∥ prev -˜ θ j ∥ 2 2 > 5 Kσ 2 log(2 T/δ ) then

6: Set b = t +1 , prev = z t

7: Restart ALG

8: else if t -b +1 is a power of 2 then

9: Set prev = ∑ t j = b z j / t -b +1

10: end if

11: Update ALG with z t

Thus we get the total expected regret within batch 1 as

The total regret within any batch i ∈ [ M ] can be lower bounded using exactly the same arguments as above. Hence summing the total regret across all batches yields

The Ω( √ T ) part of the lowerbound follows directly from Theorem 3.2.1 in Hazan [37] by choosing D with diameter bounded by Ω(1) .

D Design of low switching online regression algorithms

Even-though Algorithm 4 has attractive performance guarantees, it requires retraining with weighted ERMat every round. This is not satisfactory since the retraining can be computationally expensive. In this section, we aim to design a version of Algorithm 4 with few retraining steps while not sacrificing the statistical efficiency (up to constants). To better understand why this goal is attainable, consider a time window [1 , n ] ⊆ [ T ] where the true label marginals remain constant or drift very slowly. Due to the slow drift, one reasonable strategy is to re-train the model (with weighted ERM) using the past data only at time points within [1 , n ] that are powers of 2 (i.e via a doubling epoch schedule). For rounds t ∈ [1 , n ] that are not powers of 2, we make predictions with a previous model h prev computed at t prev := 2 ⌊ log 2 t ⌋ which is trained using data seen upto the time t prev. Observe that this constitutes at least half of the data seen until round t . This observation when combined with the slow drift of label marginals implies that the performance of the model h prev at round t will be comparable to the performance of a model obtained by retraining using entire data collected until round t .

To formalize this idea, we need an efficient online change-point-detection strategy that can detect intervals where the TV of the true label marginals is low and retrain only (modulo at most log T times within a low TV window) when there is enough evidence for sufficient change in the TV of the true marginals. We address this problem via a two-step approach. In the first step, we construct a generic black-box reduction that takes an online regression oracle as input and converts it into

another algorithm with the property that the number of switches in its predictions is controlled without sacrificing the statistical performance. Recall that the purpose of the online regression oracles is to track the true label marginals. The output of our low-switching online algorithm remains the same as long as the TV of the true label marginals (TV computed from the time point of the last switch) is sufficiently small. Then we use this low-switching online regression algorithm to re-train the classifier when a switch is detected.

We next provide the L ow switching through P hased A veraging (LPA) (Algorithm 5), our black-box reduction to produce low switching regression oracles. We remark that this algorithm is applicable to the much broader context of online regression or change point detection and can be of independent interest.

We now describe the intuition behind Algorithm 5. The purpose of Algorithm 5 is to denoise the observations z t and track the underlying ground truth θ t in a statistically efficient manner while incurring low switching cost. Hence it is applicable to the broader context of online non-parametric regression [3, 56, 7] and offline non-parametric regression [66, 72].

Algorithm 5 operates by adaptively detecting low TV intervals. Within each time window it performs a phased averaging in a doubling epoch schedule. i.e consider a low TV window [ b, n ] . For a round t ∈ [ b, n ] let t prev := 2 ⌊ log 2 ( t -b +1) ⌋ . In round t , the algorithm plays the average of the observations z b : t prev . So we see that in any low TV window, the algorithm changes its output only at-most O (log T ) times.

For the above scheme to not sacrifice statistical efficiency, it is important to efficiently detect windows with low TV of the true label marginals. Observe that the quantity prev computes the average of at-least half of the observations within a time window that start at time b . So when the TV of the ground truth within a time window [ b, t ] is small, we can expect the average to be a good enough representation of the entire ground truth sequence within that time window. Consider the quantity R t := ∑ t j = b +1 ∥ prev -θ j ∥ 2 2 which is the total squared error (TSE) incurred by the fixed decision prev within the current time window. Whenever the TV of the ground truth sequence θ b : t is large, there will be a large bias introduced by prev due to averaging. Hence in such a scenario the TSE will also be large indicating non-stationarity. However, we can't compute R t due to the unavailability of θ j . So we approximate R t by replacing θ j with the estimates ˜ θ j coming from the input online regression algorithm that is not constrained by switching cost restrictions. This is the rationale behind the non-stationarity detection test at Step 5. Whenever a non-staionarity is detected we restart the input online regression algorithm as well as the start position for computing averages (in Step 6).

We have the following guarantee for Algorithm 5.

Theorem 18. Suppose the input black box ALG given to Algorithm 5 is adaptively minimax optimal (see Definition 1). Then the number of times Algorithm 5 switches its decision is at most ˜ O ( T 1 / 3 V 2 / 3 T ) with probability at least 1 -δ . Further, Algorithm 5 satisfies ∑ T t =1 ∥ ˆ θ t -θ t ∥ 2 2 = ˜ O ( T 1 / 3 V 2 / 3 T ) with probability at least 1 -δ , where V T = ∑ T t =2 ∥ θ t -θ t -1 ∥ 1 .

Remark 19. Since Algorithm 5 is a black-box reduction, there are a number of possible candidates for the input policy ALG that are adaptively minimax. Examples include FLH with online averages as base learners [38] or Aligator algorithm [7].

Armed with a low switching online regression oracle LPA , one can now tweak Algorithm 4 to have sparse number of retraining steps while not sacrificing the statistical efficiency (up to multiplicative constants). The resulting procedure is described in Algorithm 6 (in App. E) which enjoys similar rates as in Theorem 6 (see Theorem 22).

E Omitted proofs from Section 4

First we recall a result from Baby et al. [7].

Proposition 20 (Theorem 5 of Baby et al. [7]) . Consider the online regression protocol defined in Definition 1. Let ˆ θ t be the estimate of the ground truth produced by the Aligator algorithm from Baby et al. [7]. Then with probability at-least 1 -δ , the total squared error (TSE) of Aligator satisfies

where V T = ∑ T t =2 ∥ θ t -θ t -1 ∥ 1 . This bound is attained without any prior knowledge of the variation V T .

The high probability guarantee also implies that

where the expectation is taken with respect to randomness in the observations.

By following the arguments in Baby and Wang [4], a similar statement can be derived also for the FLH-FTL algorithm of Hazan and Seshadhri [38] (Algorithm 7).

Next, we verify that the noise condition in Definition 1 is satisfied for the empirical label marginals computed at Step 5 of Algorithm 4.

Lemma 21. Let s t be as in Step 5 of Algorithm 4. Then it holds that s t = q t + ϵ t with ϵ t being independent across t and Var( ϵ t ) ≤ 1 /N .

Proof. Since s t is simply the empirical label proportions, it holds that E [ s t ] = q t . Further Var( s t ) ≤ 1 as the indicator function is bounded by 1 /N . This concludes the proof.

Theorem 6. Suppose the true label marginal satisfies min t,k q t ( k ) ≥ µ > 0 . Choose the online regression oracle in Algorithm 4 as FLH-FTL (App. F) or Aligator from Baby et al. [7] with its predictions clipped such that ˆ q t [ k ] ≥ µ . Then with probability at least 1 -δ , Algorithm 4 produces hypotheses with R H dynamic = ˜ O ( T 2 / 3 V 1 / 3 T + √ T log( |H| /δ ) ) , where V T = ∑ T t =2 ∥ q t -q t -1 ∥ 1 . Further, this result is attained without any prior knowledge of the variation budget V T .

Proof. In the proof we first proceed to bound the instantaneous regret at round t . Re-write the population loss as:

where the expectation is taken with respect to randomness in the samples.

We define the following quantities:

We decompose the regret at round t as

and

where in the last line we used Eq.(2). Now we proceed to bound each terms as note above. Note that for any label m ,

where in the last line, we used the assumption that the minimum label marginals (and hence of the online estimates via clipping) is bounded from below by µ . So by applying triangle inequality and using the fact that the losses are bounded by B in magnitude, we get

where line (a) is a consequence of Jensen's inequality. In line (b) we used the following fact: by Lemma 21 and Proposition 16, the expected cumulative error of the online oracle at any step is bounded by ϕt 1 / 3 V 2 / 3 t for some multiplier ϕ which can contain poly-logarithmic factors of the horizon (see Proposition 20).

Proceeding in a similar fashion, the term T 4 can be bounded by Eq.(10).

Next, we proceed to handle T3. Let h ∈ H be any fixed hypothesis. Then each summand in Eq.(7) is an independent random variable assuming values in [0 , B/µ ] (recall that the losses lie within [0 , B ] ). Hence by Hoeffding's inequality we have that

with probability at-least 1 -δ/ (3 T |H| ) . Now taking union bound across all hypotheses in H , we obtain that:

with probability at-least 1 -δ/ (3 T ) .

To bound T2, we notice that it is not possible to directly apply Hoeffding's inequality because the summands in Eq.(8) are correlated through the estimates of the online algorithm. So in the following, we propose a trick to decorrelate them. For any hypothesis h ∈ H , we have that

Now using Eq.(9) and proceeding similar to the bouding steps of Eq.(10), we obtain

with probability at-least 1 -δ/ 3 . In line (a) we used Jensen's inequlaity and in the last line we used the fact the the online oracle attains a high probability bound on the total squared error (TSE) (see Proposition 20).

1 N ( t -1) ∑ t -1 i =1 ∑ N j =1 V ij can be bounded using the same expression as above using similar logic.

To bound 1 N ( t -1) ∑ t -1 i =1 ∑ N j =1 W ij , we note that it is the sum of independent random variables. Hence using the same arguments used to obtain Eq.(11), we have that

with probability at-least 1 -δ/ (3 T |H| ) . Hence taking a union bound across all hypothesis classes and across the high probability event of low TSE for the online algorithm yields that

with probability at-least 1 -2 δ/ (3 T ) .

Combining the bounds developed for T1,T2,T3 and T4 and by taking a union bound across the event that resulted in Eq.(12), we obtain the following bound on instantaneous regret.

with probability at-least 1 -δ/T .

Note that via Jensen's inequality:

where in the last line we used Proposition 20.

Similarly it can be shown that

under the event that resulted in Eq.(13).

Observe that

Finally note that

Hence combining the above bounds and adding Eq.(13) across all time steps, followed by a union bound across all rounds, we obtain that

with probability at-least 1 -δ .

Next, we prove Theorem 18.

Theorem 18. Suppose the input black box ALG given to Algorithm 5 is adaptively minimax optimal (see Definition 1). Then the number of times Algorithm 5 switches its decision is at most ˜ O ( T 1 / 3 V 2 / 3 T ) with probability at least 1 -δ . Further, Algorithm 5 satisfies ∑ T t =1 ∥ ˆ θ t -θ t ∥ 2 2 = ˜ O ( T 1 / 3 V 2 / 3 T ) with probability at least 1 -δ , where V T = ∑ T t =2 ∥ θ t -θ t -1 ∥ 1 .

Proof. First we proceed to bound the number of switches. Observe that between two time points where condition in Line 5 of Algorithm 5 evaluates true, we can have at-most log T switches due to the doubling epoch schedule in Line 8.

We first bound the number of times, condition in Line 5 is satisfied. Suppose for some some time t , we have that ∑ t j = b +1 ∥ prev -˜ θ j ∥ 2 2 > 4 Kσ 2 log( T/δ ) . Suppose throughout the run of the algorithm, this is i th time the previous condition is satisfied. Let n i := t -b + 1 and let C i = TV [ b → t ]

where TV [ p → q ] = ∑ q t = p +1 ∥ θ t -θ t -1 ∥ 1 . Due to the doubling epoch schedule, we have that that prev = 1 ℓ ∑ ℓ j = b y j and E [ prev ] = 1 ℓ ∑ ℓ j = b θ j for some n i ≥ ℓ ≥ ( t -b +1) / 2 = n i / 2 .

So we have

with probability at-least 1 -δ/ ( T ) . In line (a) we used the following facts: i) Due to Hoeffding's inequality, ∥ prev -E [ prev ] ∥ 2 2 ≤ σ 2 K log(4 T/δ )) /ℓ ≤ 2 σ 2 K log(2 T/δ )) /n i with probability at-least 1 -δ/ (2 T ) ; ii) ∥ E [ prev ] -θ j ∥ 2 = ∥ 1 ℓ ∑ ℓ i = b θ i -θ j ∥ 2 ≤ 1 ℓ ∑ ℓ i = b ∥ θ i -θ j ∥ 2 ] ≤ C i ; iii) ∥ ˜ θ j -θ j ∥ 2 2 ≤ ϕn 1 / 3 i C 2 / 3 i with probability at-least 1 -δ/ (2 T ) due to condition in Theorem 18; iv) Union bound over the events in (i) and (iii).

Since the condition in Line 5 is satisfied at round t , Eq.(14) will imply that 5 Kσ 2 log(2 T/δ ) ≤ 4 max { n i C 2 i , ϕn 1 / 3 i C 2 / 3 i } +4 σ 2 K log(2 T/δ )) . Rearranging the above, we find that

where we suppress the dependence on constants and log T .

Let the condition in Line 5 be satisfied M number of times. By union bound, we have that with probability at-least 1 -δ

where in Line (a) we used Jensen's inequality. Rearranging we get that

with probability at-least 1 -δ .

Now we proceed to bound the total squared error (TSE) incurred by Algorithm 5. Let ˆ θ j be the output of Algorithm 5 at round j . Suppose at times b -1 and c +1 , the condition in Line (5) is satisfied. Observe that the condition in Line 5 is not satisfied for any times in [ b, c ] . Then we can conclude that within the interval [ b, c ] we have that ∑ c j = b ∥ ˆ θ j -˜ θ j ∥ 2 2 ≤ 5 Kσ 2 log(4 T/δ ) log( T ) , since there are only at-most log T times within [ b, c ] where condition in Line 9 is satisfied. So we have that

with probability at-least 1 -δ/T . Here n i := b -c +1 and C i := TV [ b → c ] . Further we have that ∥ ˆ θ c +1 -θ c +1 ∥ 2 2 ≤ 2 B 2 due to the boundedness condition in Definition 1.

Thus overall we have that ∑ c +1 j = b = ˜ O ( K + n 1 / 3 i C 2 / 3 i ) , with probability at-least 1 -δ for any interval [b,c+1] such that condition in Line 5 is satisfied at times b -1 and c +1 . Thus we have that

with probability at-least 1 -δ . In line (a) we used Eq.(15). In line (b) we used Holder's inequality with the dual norm pair (3 , 3 / 2) . This concludes the proof.

We now present the tweak of Algorithm 4 by instantiating ALG with Algorithm 5 and prove its regret guarantees. The resulting algorithm is described in Algorithm 6.

Algorithm 6 Lazy-TrainByWeights : handling label shift with sparse ERM calls

Input : Instance ALG of Algorithm 5, A hypothesis Class H

1: At round t ∈ [ T ] , get estimated label marginal ˆ q t ∈ R K from ALG ( s 1: t -1 ) .

2: if ˆ q t == ˆ q t -1 then

3: h t = h t -1

4: else

5: Update the hypothesis by calling a weighted-ERM oracle:

6: end if

7: Get N co-variates x t, 1: N and make predictions according to h t

8: Get labels y t, 1: N

9: Compute s t [ i ] = 1 N ∑ N j =1 I { y t,j = i } for all i ∈ [ K ] .

10: Update ALG with the empirical label marginals s t .

Theorem 22. Assume the same notations as in Theorem 6. Suppose we run Algorithm 6 (see Appendix E) with ALG instantiated using Algorithm 5 with σ 2 = 1 /N and predictions clipped as in Theorem 6. Further let the online regression oracle given to Algorithm 5 be chosen as one of the candidates mentioned in Remark 19. Then with probability at-least 1 -δ , we have that

Further, the number of number of calls to ERM oracle (via Step 5) is at-most ˜ O ( T 1 / 3 V 2 / 3 T ) with probability at-least 1 -δ .

Sketch. The proof of this theorem closely follows the steps fused for proving Theorem 6. So we only highlight the changes that need to be incorporated to the proof of Theorem 6.

Replace the use of Proposition 20 in the proof of Theorem 6 with Theorem 18.

For any round t , where Step 5 of Algorithm 6 is triggered, we can use the same arguments as in the Proof of Theorem 22 to bound the instantaneous regret by Eq.(13). i.e:

with probability at-least 1 -δ/T .

For a round t , where Step 5 is not triggered, we proceed as follows:

Let t ′ be the most recent time step prior to t when Step 5 is executed. Notice that the population loss can be equivalently represented as

where the expectation is taken with respect to randomness in the samples.

We define the following quantities:

and

We decompose the regret at round t as

where in the last line we used Eq.(2). Now we proceed to bound each terms as note above.

By using the same arguments as in Proof of Theorem 6 and replacing the use of Proposition 20 with Theorem 18, we can bound T1-4. This will result in an instantaneous regret bound at round t (which doesn't trigger step 5) as:

with probability at-least 1 -δ/T . In the last line we used the fact that t ′ -1 ≥ ( t/ 2) -1 ≥ ( t -1) / 4 for all t ≥ 3 .

Now adding Eq.(16) and (17) across all rounds and proceeding similar to the proof of Theorem 6 (and replacing the use of Proposition 20 with Theorem 18) completes the argument.

We next prove the matching (up to factors of log T ) lower bound.

Theorem 8. Let V T ≤ T/ 8 . There exists a choice of hypothesis class, loss function, and adversarial strategy of generating the data such that R H dynamic = Ω ( T 2 / 3 V 1 / 3 T + √ T log( |H| ) ) , where the expectation is taken with respect to randomness in the algorithm and adversary.

Proof. First we fix the hypothesis class and the data generation strategy. In the problem instance we consider, there are no co-variates. The hypothesis class is defined as

Further we design the hypothesis class such that both h 0 , h 1 ∈ H . Next we fix the data generation strategy:

Divide the time horizon into batches of length ∆ .

At the beginning of a batch i , the adversary picks ˚ q i uniformly at random from { 1 / 2 -δ, 1 / 2 + δ } .

For all rounds t that falls within batch i , the label y t ∼ Ber ( q t ) is sampled with q t := ˚ q i .

Learner predicts a label ˆ y t ∈ { 0 , 1 } and then the actual label y t is revealed (hence N = 1 in the protocol of Fig.3).

̸

Learner suffers a loss given by ℓ t (ˆ y t ) = I { ˆ y t = y t } .

It is easy to see that the losses are bounded in [0 , 1] . Now let's examine the two possibilities of generating labels within a batch. Let's upper bound the variation incurred by the label marginals:

where the last line is obtained by choosing δ = V T / (2 M ) = V T ∆ / (2 T ) .

Since at the beginning of each batch, the sampling probability of true labels is independently renewed, the historical data till the beginning of a batch is immaterial in minimising the loss within the batch. So we can lower bound the regret within each batch separately and add them up. Below, we focus on lower bounding the regret in batch 1 and the computations are similar for any other batch.

Suppose that the probability that an algorithm predict label y t = 1 is ˆ q t , where ˆ q t is a measurable function of the past data y 1: t -1 . Then we have that the population loss L t (ˆ q t ) := E [ ℓ t (ˆ y t )] = (1 -ˆ q t ) q t +ˆ q t (1 -q t ) . Here we abuse the notation L ( q t ) := L ( h q t ) . We see that the population loss L t (ˆ q t ) are convex and its gradient obeys ∇ L t (ˆ q t ) = 1 -2 q t = E [1 -2 y t ] since by our construction y t ∼ Ber ( q t ) . Thus the population losses are convex and its gradients can be estimated in an unbiased manner from the data.

We use the following Proposition due to Besbes et al. [10].

Proposition 23 (due to Lemma A-1 in Besbes et al. [10]) . Let ˜ P denote the joint probability of the label sequence y 1:∆ within a batch when they are generated using Ber (1 / 2 -δ ) . So

Similarly define ˜ Q as

According to the above distribution the data are independently sampled from Bernoulli trials with success probability 1 / 2 + δ . Let ˆ q t be the decision of the online algorithm qt round t so that the algorithm predicts label 1 with probability ˆ q t .

Let P denote the joint probability distribution across the decisions ˆ q 1:∆ of any online algorithm under the sampling model ˜ P . Similarly define Q . Note that any online algorithm can make decisions at round t only based on the past observed data y 1: t -1 . Further after making the decision ˆ q t at round t , an unbiased estimate of the population loss can be constructed due to the fact that ∇ L t (ˆ q t ) = E [1 -2 y t ] . Under the availability of unbiased gradient estimates of the losses, it holds that

By choosing δ = V T / (2 M ) = V T ∆ / (2 T ) and ∆ = ( T/V T ) 2 / 3 , we get that KL ( P || Q ) ≤ 1 .

Since V T ≤ T/ 8 , the above choice implies that δ ∈ (0 , 1 / 4) .

For notational clarity, define L P ( q ) = (1 -q )(1 / 2 -δ ) + q (1 / 2 + δ ) and L Q ( q ) = (1 -q )(1 / 2 + δ ) + q (1 / 2 -δ ) . These corresponds to the population losses according to the sampling models P and Q respectively. Observe that min q L P ( q ) = min q L Q ( q ) = 1 / 2 -δ . The minimum of L P and L Q are achieved at 0 and 1 respectively. Note that both h 0 , h 1 ∈ H . So there is always a hypothesis in H that corresponds the minimiser of the loss.

Further whenever ˆ q ≥ 1 / 2 we have that

Similarly whenever q < 1 / 2 we have L Q ( q ) ≥ 1 / 2 . So we define the selector function as ϕ t := I { ˆ q t ≥ 1 / 2 } . Let q ∗ t ∈ { 0 , 1 } be the minimiser of the loss at round t . Now we can lower bound the instantaneous regret similar as

where the last line is obtained by Propositions 23 and 17.

Thus we get a total lower bound on the instantanoeus regret as

where the last line is obtained by using our choices of δV T ∆ / (2 T ) and ∆ = ( T/V T ) 2 / 3 .

The second term of of Ω( √ T log |H| ) can be obtained from the existing results on statistical learning theory without distribution shifts. (see for example Theorem 3.23 in Mohri et al. [53]).

F More details on experiments

In Algorithm 7, we describe the FLH-FTL algorithm from Hazan and Seshadhri [38] when specialised to squared error losses. When specialized to squared error losses, this algorithm runs FLH with online averages as the base experts.

Algorithm 7 An instance of FLH-FTL from Hazan and Seshadhri [38] with squared error losses

1: Parameter α is defined to be a learning rate

// initializations and definitions

2: For FLH-FTL instantiations within UOLS algorithms (as in Algorithm 2), we set α ← σ 2 min ( C ) / (8 K ) , where C is the confusion matrix as in Assumption 1. For instantiations within SOLS algorithms (as in Algorithm 4) we set α ← 1 / (8 K )

3: For each round t ∈ [ T ] , v t := ( v (1) t , . . . , v ( t ) t ) is a probability vector in R t . Initialize v (1) 1 ← 1

4: For each j ∈ [ T ] , define a base learner E j . For each t > j , the base expert outputs E j ( t ) := 1 t -j ∑ t -1 i = j z j , where z j to be specified as below. Further E j ( j ) := 0 ∈ R K // execution steps

5: In round t ∈ [ T ] , set ∀ j ≤ t , x j t ← E j ( t ) (the prediction of the j th base learner at time t ). Play x t = ∑ t j =1 v ( j ) t x ( j ) t .

6: Receive feedback z t , set ˆ v ( t +1) t +1 ← 0 and perform update for 1 ≤ i ≤ t :

̸

7: Addition step - Set v ( t +1) t +1 to 1 / ( t +1) and for i = t +1 :

Rationale behind the learning rate setting at Line 2 of Algorithm 7 The loss that is incurred by Algorithm 7 and any of its base learners at round t is defined to be the squared error loss ℓ t ( x ) = ∥ z t -x ∥ 2 2 . Whenever ∥ z t ∥ 2 2 ≤ B 2 and ∥ x ∥ 2 2 ≤ B 2 , the losses ℓ t ( x ) are 1 / (8 B 2 ) exp-concave (see for eg. Chapter 3 of [12]). The notion of exp-concavity is crucial for FLH-FTL algorithm since the learning rate is set to be equal to the exp-concavity factor of the loss functions (see Theorem 3.1 in Hazan and Seshadhri [38]).

For the UOLS problem, from Algorithm 2, we have ∥ z t ∥ 2 = ∥ C -1 f 0 ( x t ) ∥ 2 ≤ √ K/σ min ( C ) . Since the decisions of the algorithm is a convex combination of the previously seen z t , we conclude that the losses ℓ t ( x ) are σ 2 min ( C ) / (8 K ) exp-concave.

For the SOLS problem, let z t = s t where s t is as defined in Algorithm 4. We have that ∥ z t ∥ 2 ≤ √ K . Hence arguing in a similar fashion as above, we conclude that the losses ℓ t ( x ) are 1 / (8 K ) expconcave for the SOLS problem.

This is the motivation behind Line 2 in Algorithm 7, where the learning rates are set according to the problem setting.

Dataset and model details.

Synthetic: For the synthetic data, we generated 72 k samples as described in Bai et al. [8]. There are three classes each with 24 k samples generated from three Gaussian distributions in R 12 . Each Gaussian distribution is defined by a randomly generated unit-norm centre v and covariance matrix 0 . 215 · I . 60 k samples are used as source data, and 12 k samples are used as target data to be sampled from during online learning. We used logistic regression to train a linear model. It is trained for a single epoch with learning rate 0 . 1 , momentum 0 . 9 , batch size 200 , and l 2 regularization 1 × 10 -4 .

MNIST [50]: An image dataset of 10 types of handwritten digits. 60 k samples are used as source data and 10 k as target data. We used an MLP for prediction with three consecutive

hidden layers of sizes 100 , 100 , and 20 . It is trained for a single epoch with a learning rate 0 . 1 , momentum 0 . 9 , batch size 200 , and l 2 regularization 1 × 10 -4 .

CIFAR-10 [49]: A dataset of colored images of 10 items: airplane, automobile, bird, cat, deer, dog, frog, horse, ship, and truck. 50 k samples are used as source data and 10 k as target data. We train a ResNet18 model (He et al. [39]) from scratch. It is finetuned for 70 epochs with learning rate 0 . 1 , momentum 0 . 9 , batch size 200 , and l 2 regularization 1 × 10 -4 . The learning rate decayed by 90% at the 25 th and 40 th epochs.

Fashion [77]: An image dataset of 10 types of fashion items: T-shirt, trouser, pullover, dress, coat, sandals, shirt, sneaker, bag, and ankle boots. 60 k samples are used as source data and 10 k as target data. We trained an MLP for prediction. It is trained for 50 epochs with learning rate 0 . 1 , momentum 0 . 9 , batch size 200 , and l 2 regularization 1 × 10 -4 .

EuroSAT [40]: An image dataset of 10 types of land uses: industrial buildings, residential buildings, annual crop, permanent crop, river, sea & lake, herbaceous vegetation, highway, pasture, and forest. 60 k samples are used as source data and 10 k as target data. We cropped the images to the size (3 , 64 , 64) . We train a ResNet18 model for 50 epochs with learning rate 0 . 1 , momentum 0 . 9 , batch size 200 , and l 2 regularization 1 × 10 -4 .

Arxiv [15]: A natural language dataset of 23 classes over different publication subjects. 198 k samples are used as source data and 22 k as target data. We trained a DistilBERT model (Sanh et al. [62]) for 50 epochs with learning rate 2 × 10 -5 , batch size 64 , and l 2 regularization 1 × 10 -2 .

SHL [31, 71]: A tabular locomotion dataset of 6 classes of human motion: still, walking, run, bike, car and bus. 30 k samples are used as source data and 70 k as target data. We trained an MLP for prediction for 50 epochs with learning rate 0 . 1 , momentum 0 . 9 , batch size 200 , and l 2 regularization 1 × 10 -4 .

For all the datasets above, the initial offline data are further split by 80 : 20 into training and holdout data, where the former is used for offline training of the base model and the latter for computing the confusion matrix and retraining (e.g. updating the linear head parameters with UOGD or updating the softmax prediction with our FLT-FTL) during online learning. To examine how well the algorithms adapt when holdout data is limited, we use 10% of the holdout data (i.e., 2% of the initial offline data) in the main paper unless stated otherwise. In App. G.2, we ablate with full hold-out data.

Types of Simulated Shifts. We simulate four kinds of label shifts to capture different non-stationary environments. These shifts are similar to the ones used in Bai et al. [8]. For each shift, the label marginals are a mixture of two different constant marginals µ 1 , µ 2 ∈ ∆ K with a time-varying coefficient α t : µ y t = (1 -α t ) µ 1 + α t µ 2 , where µ y t denotes the label distribution at round t and α t controls the shift non-stationarity and patterns. In particular, we have: Sinusoidal Shift (Sin) : α t = sin iπ L , where i = t mod L and L is a given periodic length. In the experiments, we set L = √ T . Bernoulli Shift (Ber) : at every iteration, we keep the α t = α t -1 with probability p ∈ [0 , 1] and otherwise set α t = 1 -α t -1 . In the experiments, the parameter is set as p = 1 / √ T , which implies V t = Θ( √ T ) . Square Shift (Squ) : at every L rounds we set α t = 1 -α t -1 . Monotone Shift (Mon) : we set α t = t/T . Square, sinusoidal, and Bernoulli shifts simulate fast-changing environments with periodic patterns.

Methods for UOLS Adaptation.

Base: the base classifier without any online modifications.

OFC: the optimal fixed classifier predicts with an optimal fixed re-weighting in hindsight as in Wu et al. [76].

Oracle: re-weight the base model's predictions with the true label marginal of the unlabeled data at each time step.

FTH: proposed by Wu et al. [76], follow-the-history classifier re-weights the base model's predictions with a simple online average of all marginal estimates seen thus far.

FTFWH: proposed by Wu et al. [76], follow-the-fixed-window-history classifier is a version of FTH that tracks only the k most recent marginal estimates. We choose k = 100 throughout the experiments in this work.

ROGD: proposed by Wu et al. [76], ROGD uses online gradient descent to update its reweighting vector based on current marginal estimate.

UOGD: proposed by Bai et al. [8], retrains the last linear layer of the model based on current marginal estimate.

ATLAS: proposed by Bai et al. [8] is a meta-learning algorithm that has UOGD as its base learners.

The learning rates of ROGD, UOGD, and ATLAS are set according to suggestions in the original works. The learning rate of FLH-FTL is set to 1 /K . This corresponds to a faster rate than the theoretically optimal learning rate given in Line 2 of Algorithm 7. It has been observed in prior works such as Baby et al. [7] that the theoretical learning rate is often too conservative and faster rates lead to a better performance.

F.1 Supervised Online Label Shift Experiment Details

For each dataset, we first fix the number of time steps and then simulate the label marginal shift. To train the learner with all the methods, we store all the online data observed giving the storage complexity of O ( T ) . We observe N = 50 examples at every iteration and we split the observed labeled examples into 80:20 split for training and validation. The validation examples are used to decide the number of gradient steps at every time step, in particular, we take gradient steps till the validation error continues to decrease.

Dataset and model details.

MNIST [50]: An image dataset of 10 types of handwritten digits. At each step, we sample 50 samples with the label marginal that step without replacement and reveal the examples to the learner. We used an MLP for prediction with three consecutive hidden layers of sizes 100 , 100 , and 20 . It is trained for a single epoch with a learning rate 0 . 1 , momentum 0 . 9 , batch size 200 , and l 2 regularization 1 × 10 -4 .

CIFAR-10 [49]: A dataset of colored images of 10 items: airplane, automobile, bird, cat, deer, dog, frog, horse, ship, and truck. At each step, we sample 50 samples with the label marginal that step without replacement and reveal the examples to the learner. It is finetuned for 70 epochs with learning rate 0 . 1 , momentum 0 . 9 , batch size 200 , and l 2 regularization 1 × 10 -4 .

We simulate Bernoulli label shifts to capture different non-stationary environments.

Connection of CT-RS to weighted ERM Before making the connection, we first re-visit the CTRS algorithm. Step 1 : Maintain a pool of all the labeled data received till that time step, and at every

iteration, we randomly sample a batch with uniform label marginal to update the model. Step 2 : Reweight the softmax outputs of the updated model with estimated label marginal. Below we show that it is equivalent to wERM:

where ˆ µ t,k = ˆ q t ( k ) / (1 /K ) is the importance ratio at time i with respect to uniform label marginal. Similarly, we define ˆ µ i,k = ˆ q i ( k ) / (1 /K ) . Here, L t -1 ,k is the aggregate loss at t -th time step for k -th class such that at each step the sampling probability of that class is uniform. By continually training a classifier with CT-RS, Step 1 approximates the classifier ˜ f t trained to minimize the average of L t -1 ,k over all classes with uniform proportion for each class. To update the classifier ˜ f t according to label proportions at time t , we update the softmax output of ˜ f t according to ˆ µ t .

The primary benefit of CT-RS over wERM is to avoid re-training from scratch at every iteration. Instead, we can leverage the model trained in the previous iteration to warm-start training in the next iteration.

G Additional Unsupervised Online Label Shift Experiments

G.1 Additional results with Monotone and Square Shifts and Low Amount of Holdout Data

Synthetic

Mon

Squ

11

0

±

.

±

0

.

0

.

05

0

0

00

.

0

±

0

18

±

00

01

.

0

.

21

±

0

23

.

.

0

0

0

0

.

0

±

29

.

.

.

0

±

11

.

0

01

01

01

.

MNIST

Mon

14

0

±

.

±

0

.

0

.

07

0

0

00

0

±

.

0

28

±

00

.

0

05

.

Squ

27

±

0

30

.

.

0

0

0

0

.

0

±

41

.

.

.

0

±

15

.

0

00

00

04

.

0

.

0

.

CIFAR

Mon

15

0

±

.

±

07

0

.

0

0

0

22

±

.

01

.

.

0

00

04

.

01

Squ

28

±

0

30

.

.

0

0

0

0

.

0

±

37

.

.

.

0

±

17

.

0

00

00

03

.

±

±

±

±

00

00

00

00

00

FTH

FTFWH

ROGD

FLH-FTL (ours)

.

.

.

.

09

05

07

0

.

0

.

EuroSAT

Mon

Squ

14

±

0

.

±

07

0

.

0

27

±

.

08

0

0

±

01

.

.

0

00

03

.

01

0

0

0

0

.

.

.

.

28

0

±

30

±

41

±

17

.

0

.

0

±

.

0

00

00

02

.

01

0

.

0

.

Fashion

Mon

Squ

16

±

0

.

±

08

0

.

0

21

±

.

09

0

0

±

02

.

.

0

01

01

.

01

0

0

0

0

.

.

.

.

28

±

0

±

31

±

37

18

.

0

.

0

±

.

0

01

01

01

.

02

0

.

0

.

Mon

18

±

0

.

±

09

0

.

0

21

±

.

11

ArXiv

00

0

.

0

0

±

.

.

0

00

01

.

00

0

0

0

.

.

.

Squ

±

0

30

±

32

±

36

24

.

0

.

0

±

.

0

00

00

01

.

00

G.2 Additional results with All of Holdout Data

Synthetic

Ber

Sin

20

±

0

19

0

.

0

.

0

0

CIFAR

Sin

.

Ber

28

0

±

24

.

0

0

.

0

0

.

14

0

.

00

0

±

31

±

0

±

00

.

±

0

13

0

.

0

.

0

00

00

±

21

±

13

00

0

.

0

.

0

0

.

.

EuroSAT

Ber

Sin

27

0

±

26

±

41

±

0

0

±

15

.

.

.

0

00

00

00

.

00

0

.

.

0

14

±

0

13

0

0

.

±

.

.

34

±

13

0

0

00

.

.

±

0

00

00

.

00

0

.

0

.

0

0

.

.

Ber

27

0

±

24

±

45

±

Fashion

Sin

.

0

.

00

0

0

±

15

.

.

0

00

08

.

00

0

.

0

14

±

0

13

.

0

.

.

±

±

31

13

0

0

00

.

.

±

0

00

00

.

00

0

.

0

.

0

0

.

.

Ber

29

0

±

28

±

34

±

0

0

±

22

ArXiv

.

0

.

00

.

.

0

00

00

.

00

0

.

0

0

Sin

±

0

15

15

.

±

.

.

28

±

15

0

0

00

.

.

±

0

00

00

.

00

FTH

FTFWH

ROGD

FLH-FTL (ours)

.

.

±

0

29

±

0

±

09

.

.

.

0

00

00

00

.

00

0

0

0

0

.

.

.

.

10

±

0

09

±

23

±

08

0

0

±

.

.

.

0

00

00

00

.

00

0

.

0

.

0

0

.

.

Ber

25

±

0

24

MNIST

Sin

.

0

.

00

0

±

43

±

0

±

13

.

.

0

00

00

.

00

0

0

0

.

.

.

14

0

±

13

±

33

±

12

0

0

±

.

.

.

0

00

00

00

.

00

.

.

15

.

.

0

00

00

.

00

.

0

0

.

±

0

.

G.3 Ablation over Number of Online Samples

Here we examine how different algorithms perform as the number of online samples varies. We introduce an additional baseline BBSE , which simply uses the label marginal estimate provided by black box shift estimator to reweight the predictions of classifiers. Figure 3 shows an interesting trend

1

2

4

8

16

32

64

128

Number of Online Samples

0.0

0.1

0.2

0.3

Classification Error

Oracle

BBSE

UOGD

FLH-FTL (ours)

Error

MSE

0.0

0.2

0.4

0.6

0.8

Mean Square Error

that as number of online samples increases, the simple baseline BBSE becomes more competitive and eventually outperforms UOGD, whereas our algorithm remains competitive.

G.4 Ablation over Types of Marginal Estimates

All the algorithms examined in this work use black box shift estimate (BBSE) [51] to obtain an unbiased estimate of the target label marginal. However, two alternative methods exist: Maximum Likelihood Label Shift (MLLS) [61] and Regularized Learning under Label Shift (RLLS) [2]. Table 9 presents additional results using these two estimates. The results shows using the alternative estimates do not substantially change the performances of the algorithms considered in this work.

G.5 Additional results and details on the SHL dataset

0

2000

4000

6000

Time

0.0

0.2

0.4

0.6

0.8

Classification Error

UOGD

ROGD

FTH

FLH-FTL (ours)

(a)

0

2000

4000

6000

Time

0.2

0.4

0.6

0.8

1.0

Mean Square Error

ROGD

FTH

FLH-FTL (ours)

(b)

0

20

40

60

80

100

Time

0.0

0.2

0.4

0.6

0.8

1.0

Distribution

(c)

G.6 Reweighting Versus Retraining Linear Layer

Here we compare the efficacies of re-weighting (RW-FLH-FTL) and retraining (RT-FLH-FTL) given the marginal estimate provided by FLH-FTL; the latter retrains the last linear layer on the loss of the holdout data re-weighted by the marginal estimate. Note that RW-FLH-FTL corresponds to FLHFTL in the main text. We retrain RT-FLH-FTL for 50 epochs at each time step. To compare against the best possible retrained classifiers, we used all the holdout data for retraining. Table 10 shows that retraining is often worse and at best similar to re-weighting in performance, despite greater computational cost and need for holdout data.

Datasets

Synthetic

CIFAR

Shift

Mon

Sin

Ber

Squ

Mon

Sin

Ber

Squ

Base

8

.

7

±

0

.

1

8

.

2

±

0

.

3

8

.

6

±

0

.

3

8

.

5

±

0

.

2

17

±

0

16

±

0

17

±

0

17

±

0

OFC

6

.

8

±

0

.

1

5

.

5

±

0

.

2

6

.

7

±

0

.

2

6

.

7

±

0

.

3

14

±

0

11

±

0

13

±

0

13

±

1

FTFWH

7

.

0

±

0

.

0

5

.

5

±

0

.

3

6

.

8

±

0

.

2

6

.

8

±

0

.

3

13

±

0

10

±

0

12

±

0

13

±

0

UOGD

7

.

4

±

0

.

1

7

.

0

±

0

.

0

7

.

6

±

0

.

4

7

.

6

±

0

.

2

12

±

0

10

±

0

11

±

0

13

±

0

RW-FLH-FTL (ours)

6

.

3

±

0

.

2

5

.

3

±

0

.

2

5

.

4

±

0

.

3

5

.

5

±

0

.

2

12

±

0

10

±

0

11

±

0

12

±

0

RT-FLH-FTL (ours)

6

.

7

±

0

.

1

6

.

2

±

0

.

2

6

.

0

±

0

.

0

6

.

3

±

0

.

4

12

±

0

10

±

0

11

±

0

12

±

0

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe general scope is online classifier learning under online label shift where the key assumption is that the class conditionals $Q(x|y)$ are invariant. The paper focuses on both unsupervised and supervised problems, which they coined as Unsupervised Online Label Shift (UOLS) and Supervised Online Label Shift (SOLS).\n\nKey previous works in the UOLS problem [76, 8] setup a regret function and adapt the initial classifier via minimising the regret function over time.  [76] optimises the classifier via reweighting the predicted distribution $\\hat{q}_t$. [8] retrains the classifier at every step. However, both suffer from convexity issues of the losses. This paper integrates the best of both [76] and [8] in a single framework. On top of that they avoid any previous convexity issue by reducing the online label shift problem to the problem of online regression of the class marginal distribution function. Tight theoretical bounds under the usual Lipschitz condition are provided.\n\nThe second part of the paper is an extension of the approach from the UOLS problem to the SOLS problem. The main contribution here is that theoretical bounds in (weighted) Empirical Risk Minimization are generalised to the SOLS context. Other contributions involve in making online model training faster without compromising the theoretical bounds too much.\n\nFinally, there is an extensive empirical study based on well known datasets. However, in many of the datasets used for the experiments, the label shift dynamics are simulated.\n\nSTRENGTHS:\nThe paper is very well written and very detailed. The authors were very clear of their contributions and limitations of the proposed approaches.\n\nIn my view, the key novelty of the paper is at reformulating the online label shift problem as the problem of online regression of the marginal distribution function, which previous works have overlooked. Everything else in the paper develops from that idea in naturally. This key novelty is clever because the new problem can be solved efficiently with existing tools.\n\nThe provided bounds appear to be quite tight in terms of complexity, although I am not sure whether in practice the coefficient of the highest order term plays a significant role. Proofs in the appendices are simple applications of probabilistic tools, concentration theories and a few key results in online learning theories. I was able to follow without any major trouble.\n\nThe finding in appendix G.6 is interesting, which provides an evidence that retraining does not work better than reweighting, despite an additional computational cost for retraining.\n\nWEAKNESSES:\nAlthough well acknowledged in the paper, the fact that the approach, like [8], requires memory that scales linearly with time is quite a concern from a practical point of view. It would be good to have some quantification results regarding memory usage in the experiments.\n\nAlthough the paper addresses online label shift, among 6 datasets used in the experiments for UOLS and SOLS, 5 of them use simulated label shift data. Only SHL contains real-world label shift data.\n\nI understand that the scope of the paper is online label shift where $Q_t(x|y)$ is unchanged over time. Still I find this scope is rather restrictive compared to real-world data. In the data collection business that I have been involved, data are usually collected from many sources, with large differences in collection periods and frequencies of data being collected per source. With external factors like that it is hard to assume that $Q_t(x|y)$ is unchanged over time. The SHL dataset used in section 5 seems to be the only dataset in which the label shift part of the data is real. However, given that it was collected by only 3 participants, it is hard to justify whether the dataset reflects what happens in the real world.\n\nPerhaps the authors could try with some recent consumer-related datasets (e.g. clothes, food) that contain both pre-Covid and post-Covid periods? During the Covid-19 pandemic, so many shops closed, leading to a large-scale gradual decrement of data that can be collected. Since 2022, many shops have reopened, leading to a gradual increment of collected data.\n\nMaybe a typo. However, on line 112, why does $\\delta$ not play a role in the big $\\tilde{O}$ function?\n\nQUESTIONS:\nI do not have any major question. The paper is very good to me, except for a few practical concerns raised in the Weaknesses section.\n\nLIMITATIONS:\nI do not see any potential negative societal impact of the work.", "TITLE": "Official Review", "RECOMMENDATION": 8, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-06-28"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors study the important problem of learning in an online setting.\nIn particular, they focus on two important problems:\n- Online Unsupervised learning with Label Shift (UOLS)\n- Online Supervised Learning with Label shift (SOLS)\nIn both cases, they assume, as in previous work, that the distribution of the samples can change at each step, however, the marginal distribution of the covariate X  given the label Y is the same. Their algorithm is adaptive and does not require knowledge of the distribution drift. They show that their results on dynamic regret are (almost) optimal in a minimax sense. Unlike previous work, they do not require a convexity assumption on the family of the losses, which is an important technical contribution. They corroborate their theoretical findings with experiments.\n\nSTRENGTHS:\nMain strengths:\n\n- The paper is very well written, easy to follow, and the results are sound. The assumptions used are discussed and well-motivated.\n\n- They provide a major technical contribution in dropping the convexity assumption of the previous work. \n\n- They show the (almost) tightness of their results in a minimax sense.\n\n- They show an interesting algorithm to reduce the number of training steps in the SOLS setting so that they do not need to train a new model at each step. This is a very interesting result, as this is a common problem of many previous works of learning with distribution drift.\n\n- Their theoretical findings are corroborated by extensive experiments. Moreover, I appreciate the effort of the authors to address many problems related to how to obtain the optimal minimax regret efficiently in practice. (Appendix D).\n\nWEAKNESSES:\nThe results are limited to changes in the probability of obtaining an element of a certain class (label shift). This is only a specific setting of learning with distribution drift. Their technique looks limited to this specific setting, as it revolves around estimating (regression) the class probabilities. It does not seem straightforward to extend this work or use the results of this work for the more general problem of learning with generic distribution drift. \n\nThe experiments are only run with respect to data where synthetic drift is introduced.\n\nQUESTIONS:\n-  With respect to Remark 10: The black box algorithm of appendix D can be used to improve the running time, as it detects intervals in which there is little change and we do not need to train again.  \nBy using a similar argument, is it possible to say that if there is a big change in the label probabilities, then the older data is significantly less useful, and we can stop storing it?\n\nLIMITATIONS:\nThe authors discuss limitations, assumptions, and possible future work.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-07-04"}

comments: SUMMARY:
this paper applies techniques from online regression oracles to tackle the problems of unsupervised and supervised label shift, where the marginal probabilities of the labels can be changed over time by an adversary.

STRENGTHS:
the main contributions include:
C1 an algorithm that adapts to the changing labels without knowing the parameter V_T
C2 an algorithm whose online learning assumptions are verified
C3 an algorithm compatible with black box access to the learner, to enable deep neural networks and decision trees to be used as learners
experiments illustrate the proposed algorithm is promising

the paper is well written and extensive experiments are performed with many baselines

WEAKNESSES:
some minor issues:

1 figure 2 does not explain what the uncertainty bands are. I assume these are std's or are they standard errors? similairly for the tables. figure 2c misses erroprbars or are they not visible? 

2 the appendix is extremely long, I did not have time to go over all the details, and I am inclined to suggest a full paper would be better reviewed at a journal. I thought its a shame that fig. 3 (appendix) which shows interesting trends is not included in the main body.

QUESTIONS:
3 the lowerbounds do seem to be of a slightly different form than the upperbounds. does this matter? why (not)? why is it essential to assume V_T is smaller than T*c for some c for the lowerbounds?

LIMITATIONS:
none
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-05

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper delves into the supervised and unsupervised online label shift problem, which focuses on adapting to changing class marginals using online data. The authors propose new algorithms that convert the adaptation problem into an online regression task, guaranteeing the minimax optimal dynamic regret. These methods leverage online regression oracles to track the shifting proportions. Evaluations of various online label shift scenarios show improved performances of the proposed methods.\n\nSTRENGTHS:\n1. The manuscript is neatly structured and clearly drafted.\n\n2. The authors have comprehensively explained both the problem setup and their proposed solutions.\n\n3. The theoretical foundations elaborated in the manuscript appear to be solid. Importantly, the authors extend the concept of non-stationary online learning to scenarios encompassing non-convex loss functions, albeit under a Lipschitzness assumption. This extension significantly broadens the applicability of non-stationary online learning, notably in the context of neural networks or other sophisticated prediction models.\n\nWEAKNESSES:\nWhile I am mainly positive about the paper. However, there remain some issues that may need to be further addressed for a more compelling submission:\n\n1. The usage of the term \"sample efficient\" in both the abstract and experimental sections appears confusing to me. Sample efficiency is typically understood in a theoretical context. However, the proposed method does not theoretically demonstrate sample efficiency (consistent with previous methods). Moreover, the experiments merely evaluate the size of the hold-out offline labeled data, hardly serving as evidence for \"sample efficiency.\" Consequently, the paper does not sufficiently convince me of this.\n2. It is commendable that, unlike its predecessors, the proposed method does not hinge on the convexity of the functions (line 37). Nevertheless, given the importance of the Lipschitz constant in this paper, questions arise regarding the control of the Lipschitz constant G in complex models such as Deep Neural Networks (DNNs) and decision trees. For instance, decision trees, lacking continuity, presumably possess an infinite G.\n3. In Theorem 6 (line 834), a simple union bound is utilized to derive the lower bound of the online label shift problem, resulting in a term $|\\mathcal{H}|$ in the lower bound. Yet, can this bound accommodate complex scenarios where $\\mathcal{H}$ is the DNNs or decision trees, given that $|\\mathcal{H}|$ would be infinite in such instances?\n4. Based on the points raised in Weaknesses 1, 2, and 3, the paper's title, \"Online Label Shift: Optimal Dynamic Regret with Non-convex Implementation\", appears somewhat overclaimed to me:\n    * In Remark 10, the authors acknowledge that Algorithm 4 necessitates storing and reprocessing all past data in each iteration, which contradicts the fundamental principles of online learning. Consequently, the term \"Practical Algorithms\" in the title seems overclaimed to me.\n    * A more fitting title, such as \"Online Label Shift: Optimal Dynamic Regret with Non-convex Implementation,\" could be considered, given the paper's principal contribution lies in addressing non-convexity. Such a title could potentially draw more attention from readers.\n5. The authors assert (line 55) that their approach achieves the best-of-both worlds of previous works. However, the meaning of \"worlds\" is unclear. Could the authors elucidate this further?\n6. The theoretical exposition is presented in a complicated manner and is rather daunting to read. It would potentially be more friendly to a broader audience if the authors simplified and condensed the theoretical content.\n7. Within the methodology section, the discrepancy between estimated and actual label marginals is converted into a 2-norm gap, which subsequent methods seek to optimize directly. However, in the context of label shift problems, is this direct optimization of the 2-norm (square loss) between distributions appropriate? Is the 2-norm excessively susceptible to out-of-distribution (OOD) data? Perhaps, alternative measures like KL-divergence or cross-entropy could be more suitable?\n8. Some other relevant works related to online distribution shift should be considered to be explored and discussed, such as:\n\n    * Kumar et al., Understanding Self-Training for Gradual Domain Adaptation;\n    * Zhang et al., Adapting to Continuous Covariate Shift via Online Density Ratio Estimation;\n    * Zhou et al., Online Continual Adaptation with Active Self-Training.\n9. I wonder what are the \"NA\"s mean in the experimental tables?\n10. I am unsure about the FLH and Alligator algorithms (line 193). Could the authors briefly introduce these algorithms and explain their function?\n11. line 196: The averaging within \"intervals\" xxx, can the author detail explain what the \"intervals\" is? and how to divide these intervals.\n12. The paper can be further refined to enhance readability, and there are several typographical errors:\n     * line 31: Wu et al. [76] only controls -> Wu et al. [76] only control\n     * line 35: However their approach based on -> However, their approach is based on\n     * line 82: the extend of distribution drift -> the extent of distribution drift.\n     * line 94: i.e. -> i.e.,\n     * line 96: respectively -> , respectively\n     * line 314: i.e. -> i.e.,\n     * line 1055: $\\alpha_t = 1-\\alpha_{t-1}$.\n     * line 1079: to better performance -> to a better performance\n\nQUESTIONS:\nThe paper is mainly sound and well-organized, but there remain some concerns for me, see the Weaknesses part above.\n\nLIMITATIONS:\nAs delineated in the Weakness section, the authors could provide a more detailed elucidation of the methodologies employed, and potentially consider revising the title to garner a broader impact. I am open to adjusting the evaluation score if the authors address my concerns.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-06"}