SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Dheeraj Baby*
dheeraj@ucsb.edu
sgarg2@andrew.cmu.edu
Tzu-Ching Yen*
tzuchiny@andrew.cmu.edu
sbalakri@andrew.cmu.edu
zlipton@andrew.cmu.edu
This paper focuses on supervised and unsupervised online label shift, where the class marginals  $Q(y)$  varies but the class-conditionals  $Q(x|y)$  remain invariant. In the unsupervised setting, our goal is to adapt a learner, trained on some offline labeled data, to changing label distributions given unlabeled online data. In the supervised setting, we must both learn a classifier and adapt to the dynamically evolving class marginals given only labeled online data. We develop novel algorithms that reduce the adaptation problem to online regression and guarantee optimal dynamic regret without any prior knowledge of the extent of drift in the label distribution. Our solution is based on bootstrapping the estimates of online regression oracles that track the drifting proportions. Experiments across numerous simulated and real-world online label shift scenarios demonstrate the superior performance of our proposed approaches, often achieving 1-3% improvement in accuracy while being sample and computationally efficient. Code is publicly available at this url.
Supervised machine learning algorithms are typically developed assuming independent and identically distributed (iid) data. However, real-world environments evolve dynamically [55, 67, 46, 29]. Absent further assumptions on the nature of the shift, such problems are intractable. One line of research has explored causal structures such as covariate shift [64], label shift [61, 51], and missingness shift [86], for which the optimal target predictor is identified from labeled source and unlabeled target data. Let's denote the feature-label pair of an example by  $(x,y)$ . Label shift addresses the setting where the label marginal distribution  $Q(y)$  may change but the conditional distribution  $Q(x|y)$  remains fixed. Most prior work addresses the batch setting for unsupervised adaptation, where a single shift occurs between a source and target population [61, 51, 1, 2, 1, 26]. However, in the real world, shifts are more likely to occur continually and unpredictably, with data arriving in an online fashion. A nascent line of research tackles online distribution shift, typically in settings where labeled data is available in real time [4], seeking to minimize the dynamic regret.
Researchers have only begun to explore the role that structures like label shift might play in such online settings. Initial attempts to learn under unsupervised online label shifts were made by Wu et al. [76] and Bai et al. [8], both of which rely on reductions to Online Convex Optimization (OCO) [37, 54]. This line of research aims in updating a classification model based on online data so that the overall regret is controlled. However, Wu et al. [76] only control for static regret against a fixed
Figure 1: UOLS and SOLs setup. Dashed (double) arrows are exclusive to UOLS (SOLS) settings. Other objects are common to both setups. Central question: how to adapt the model in real-time to drifting label margins based on all the available data so far?
classifier (or model) in hindsight and makes the assumption of the convexity (of losses), which is often violated in practice. In the face of online label shift, where the class marginals can vary across rounds, a more fitting notion is to control the dynamic regret against a sequence of models in hindsight. Motivated by this observation, Bai et al. [8] control for the dynamic regret. However, their approach is based on updating model parameters (of the classifier) with online gradient descent and relying on convex losses limits the applicability of their methods (e.g. algorithms in Bai et al. [8] can not be employed with decision tree classifiers).
In this paper, we study the problem of learning classifiers under Online Label Shift (OLS) in both supervised and unsupervised settings (Fig.1). In both these settings, the distribution shifts are an online process that respects the label shift assumption. Our primary goal is to develop algorithms that sidestep convexity assumptions and at the same time optimally adapt to the non-stationarity in the label drift. In the Unsupervised Online Label Shift (UOLS) problem, the learner is provided with a pool of labeled offline data sampled iid from the distribution  $Q_{0}(x,y)$  to train an initial model  $f_{0}$ . Afterwards, at every online round  $t$ , few unlabeled data points sampled from  $Q_{t}(x)$  are presented. The goal is to adapt  $f_{0}$  to the non-stationary target distributions  $Q_{t}(x,y)$  so that we can accurately classify the unlabelled data. By contrast, in Supervised Online Label Shift (SOLS), our goal is to learn classifiers from only the (labeled) samples that arrive in an online fashion from  $Q_{t}(x,y)$  at each time step, while simultaneously adapting to the non-stationarity induced due to changing label proportions. While SOLS is similar to online learning under non-stationarity, UOLS differs from classical online learning as the test label is not seen during online adaptation. Below are the list of contributions of this paper.
- Unsupervised adaptation. For the UOLS problem, we provide a reduction to online regression (see Defn. 1), and develop algorithms for adapting the initial classifier  $f_{0}$  in a computationally efficient way leading to minimax optimal dynamic regret. Our approach achieves the best-of both worlds of Wu et al. [76], Bai et al. [8] by controlling the dynamic regret while allowing us to use expressive black-box models for classification (Sec. 3).
- Supervised adaptation. We develop algorithms for SOLS problem that lead to minimax optimal dynamic regret without assuming convexity of losses (Sec. 4). Our theoretically optimal solution is based on weighted Empirical Risk Minimization (wERM) with weights tracked by online regression. Motivated by our theory, we also propose a simple continual learning baseline which achieves empirical performance competitive to the wERM from scratch at each time step across several semi-synthetic SOLs problems while being  $15 \times$  more efficient in computation cost.
- Low switching regressors. We propose a black-box reduction method to convert an optimal online regression algorithm into another algorithm that switches decisions sparingly while maintaining minimax optimality. This method is relevant for online change point detection. We demonstrate its application in developing SOLs algorithms to train models only when significant distribution drift is detected, while maintaining statistical optimality (App. D and Algorithm 6).
- Extensive empirical study. We corroborate our theoretical findings with experiments across numerous simulated and real-world OLS scenarios spanning vision and language datasets (Sec. 5). Our proposed algorithms often improve over the best alternatives in terms of both final accuracy and label marginal estimation. This advantage is particularly prominent with limited initial holdout data (in the UOLS problem) highlighting the sample efficiency of our approach.
Notes on technical novelties. Even though online regression is a well studied technique, to the best of our knowledge, it is not used before to address the problem of online label shift. It is precisely the usage of regression which lead to tractable adaptation algorithms while side-stepping convexity assumptions thereby allowing us to use very flexible models for classification. This is in stark contrast
to OCO based reductions in [76] and [8]. We propose new theoretical frameworks and identify the right set of assumptions for materializing the reduction to online regression. It was not evident initially that this link would lead to minimax optimal dynamic regret rates as well as consistent empirical improvement over prior works. Proof of the lower bounds requires adapting the ideas from non-stationary stochastic optimization [10] in a non-trivial manner. Further, none of the proposed methods require the prior knowledge of the extent of distribution drift.
Let  $\mathcal{X} \subseteq \mathbb{R}^d$  be the input space and  $\mathcal{V} = [K] \coloneqq \{1, 2, \dots, K\}$  be the output space. Let  $Q$  be a distribution over  $\mathcal{X} \times \mathcal{V}$  and let  $q(\cdot)$  denotes the corresponding label marginal.  $\Delta_K$  is the  $K$ -dimensional simplex. For a vector  $v \in \mathbb{R}^K$ ,  $v[i]$  is its  $i^{th}$  coordinate. We assume that we have a hypothesis class  $\mathcal{H}$ . For a function  $f \in \mathcal{H}: \mathcal{X} \to \Delta_K$ , we also use  $f(i|x)$  to indicate  $f(x)[i]$ . With  $\ell(f(x), y)$ , we denote the loss of making a prediction with the classifier  $f$  on  $(x, y)$ .  $L$  denotes the expected loss, i.e.,  $L = \mathbb{E}_{(x, y) \sim Q}[\ell(f(x), y)]$ .  $\tilde{O}(\cdot)$  hides dependencies in absolute constants and poly-logarithmic factors of horizon and failure probabilities.
In this work, we study online learning under distribution shift, where the distribution  $Q_{t}(x,y)$  may continuously change with time. Throughout the paper, we focus on the label shift assumption where the distribution over label proportions  $q_{t}(y)$  can change arbitrarily but the distribution of the covariate conditioned on a label value (i.e.,  $Q_{t}(x|y)$ ) is assumed to be invariant across all time steps. We refer to this setting as Online Label Shift (OLS). Here, we consider settings of unsupervised and supervised OLS settings captured in Frameworks 1 and 3 respectively. In both settings, at round  $t$  a sample  $(x_{t},y_{t})$  is drawn from a distribution with density  $Q_{t}(x_{t},y_{t})$ . In the UOLS setting, the label is not revealed to the learner. However, we assume access to offline labeled data sampled iid from  $Q_{0}$  which we use to train an initial classifier  $f_{0}$ . The goal is to adapt the initial classifier  $f_{0}$  to drifting label distributions. In contrast, for the SOLS setting, the label is revealed to the learner after making a prediction and the goal is to learn a classifier  $f_{t}\in \mathcal{H}$  for each time step.
Next, we formally define the concept of online regression which will be central to our discussions. Simply put, an online regression algorithm tracks a ground truth sequence from noisy observations.
Definition 1 (online regression). Fix any  $T > 0$ . The following interaction scheme is defined to be the online regression protocol.
- At round  $t \in [T]$ , an algorithm predicts  $\hat{\theta}_t \in \mathbb{R}^K$ .
- A noisy version of ground truth  $z_{t} = \theta_{t} + \epsilon_{t}$  is revealed where  $\theta_t, \epsilon_t \in \mathbb{R}^K$ , and  $\| \epsilon_t \|_2, \| \theta_t \|_2 \leq B$ . Further the noise  $\epsilon_t$  are independent across time with  $E[\epsilon_t] = 0$  and  $\mathrm{Var}(\epsilon_t[i]) \leq \sigma^2 \forall i \in [K]$ .
An online regression algorithm aims to control  $\sum_{t=1}^{T}\|\hat{\theta}_t - \theta_t\|_2^2$ . Moreover, the regression algorithm is defined to be adaptively minimax optimal if with probability at least  $1 - \delta$ ,  $\sum_{t=1}^{n}\|\hat{\theta}_t - \theta_t\|_2^2 = \tilde{O}(T^{1/3}V_T^{2/3})$  without knowing  $V_T$  ahead of time. Here  $V_T := \sum_{t=2}^{T}\|\theta_t - \theta_{t-1}\|_1$  is termed as the Total Variation (TV) of the sequence  $\theta_{1:T}$ .
3 Unsupervised Online Label Shift
In this section, we develop a framework for handling the UOLS problem. We summarize the setup in Framework 1. Since in practice, we may need to work with classifiers such as deep neural networks or decision trees, we do not impose convexity assumptions on the (population) loss of the classifier as a function of the model parameters. Despite the absence of such simplifying assumptions, we provide performance guarantees for our label shift adaption techniques so that they are certified to be fail-safe.
Under the label shift assumption, we have  $Q_{t}(y|x)$  as a re-weighted version of  $Q_{0}(y|x)$ :
$$
Q _ {t} (y | x) = \frac {Q _ {t} (y)}{Q _ {t} (x)} Q _ {t} (x | y) = \frac {Q _ {t} (y)}{Q _ {t} (x)} Q _ {0} (x | y) = \frac {Q _ {t} (y) Q _ {0} (x)}{Q _ {t} (x) Q _ {0} (y)} Q _ {0} (y | x) \propto \frac {Q _ {t} (y)}{Q _ {0} (y)} Q _ {0} (y | x),
$$
where the second equality is due to the label shift assumption. Hence, a reasonable strategy is to reweight the initial classifier  $f_{0}$  with label proportions (estimate) at the current step, since we only have to correct the label distribution shift. This re-weighting technique is widely used for offline label shift correction [51, 2, 1] and for learning under label imbalance [41, 74, 18].
Framework 1 Unsupervised Online Label Shift (UOLS) protocol
Input: Initial classifier  $f_0:\mathcal{X}\to \Delta_K$  trained on offline labeled dataset  $\{(x_i,y_i)\}_{i = 1}^N$  sampled iid from  $Q_{0}$
1:  $f_{1} = f_{0}$
2: for each round  $t \in [T]$  do
3: Nature samples  $x_{t} \in \mathcal{X}$  and  $y_{t} \in \mathcal{Y}$ , with  $(x_{t},y_{t}) \sim Q_{t}$ ; Only  $x_{t}$  is revealed to the learner.
4: Learner predicts a label  $i \sim f_t(x_t) \in \Delta_K$ .
5:  $f_{t + 1} = \mathcal{A}(f_0,x_{1:t})$  , where  $\mathcal{A}$  is strategy to adapt the classifier based on past data.
6: end for
Algorithm 2 RegressAndReweight to handle UOLS
Input: i) Online regression oracle ALG; ii) Initial classifier  $f_{0}$ ; iii) The confusion matrix  $C$ ; iv) The label marginal  $q_{0} \in \mathcal{D}$  of the training distribution;
1: At round  $t$ , get the classifier covariate  $x_{t}$ .
2: Let  $\widehat{q}_t = \Pi_{\mathcal{D}}(\mathrm{ALG}(s_{1:t - 1}))$  where  $\Pi_{\mathcal{D}}(x) = \operatorname{argmin}_{y \in \mathcal{D}} \| y - x \|_2$ .
3: Sample a label  $i$  with probability  $\propto \frac{\widehat{q}_t(i)}{q_0(i)} f_0(i|x_t)$ .
4: Let  $s_t = C^{-1} f_0(x_t)$ .
5: Update the online regression oracle with the estimate  $s_t$ .
Our starting point in developing a framework is inspired by Wu et al. [76], Bai et al. [8]. For self-containedness, we briefly recap their arguments next. We refer interested readers to their papers for more details. Wu et al. [76] considers a hypothesis class of re-weighted initial classifier  $f_{0}$ . The loss of a hypothesis is parameterised by the re-weighting vector. They use tools from OCO to optimise the loss and converge to a best fixed classifier. However as noted in Wu et al. [76], the losses are not convex with respect to the re-weight vector in practice. Hence usage of OCO techniques is not fully satisfactory in their problem formulation.
In a complementary direction, Bai et al. [8] abandons the idea of re-weighting. Instead, they update the parameters of a model at each round using online gradient descent and a loss function whose expected value is assumed to be convex with respect to model parameters. They provide dynamic regret guarantees against a sequence of changing model parameters in hindsight, and connects it to the variation of the true label margins. More precisely, they provide algorithms with  $\sum_{t=1}^{T} L_t(w_t) - L_t(w_t^*)$  to be well controlled where  $w_t^*$  is the best model parameter to be used at round  $t$  and  $L_t$  is a (population level) loss function. However, there are some scopes for improvement in this direction as well. For example, the convexity assumption can be easily violated when working with interpretable models based on decision trees, or if we want to retrain few final layers of a deep classifier based on new data. Further as noted in the experiments (Sec. 5), their methods based on retraining the classifier require more data than re-weighting based methods. Our experiments also indicate that re-weighting can be computationally cheaper than re-training without sacrificing the classifier accuracy.
Thus, on the one hand, the work of Wu et al. [76] allows us to use the power of expressive initial classifiers while only controlling the static regret against a fixed hypothesis. On the other hand, the work of Bai et al. [8] allows controlling the dynamic regret while limiting the flexibility of deployed models. We next provide our framework for handling label shifts that achieves the best of both worlds by controlling the dynamic regret while allowing the use of expressive blackbox models.
In summary, we estimate the sequence of online label marginals and leverage the idea of re-weighting an initial classifier as in Wu et al. [76]. In particular, given an estimate  $\widehat{q}_t(y)$  of the true label marginal at round  $t$ , we compute the output of the re-weighted classifier  $f_t$  as  $\frac{\widehat{q}_t(y)}{q_0(y)} f_0(y|x)/Z$  where  $Z = \sum_y \frac{\widehat{q}_t(y)}{q_0(y)} f_0(y|x)$ . However, to get around the issue of non-convexity, we separate out the process of estimating the re-weighting vectors via a reduction to online regression which is a well-defined and convex problem with computationally efficient off-the-shelf algorithms readily available. Second, and more importantly, Wu et al. [76] competes with the best fixed re-weighted hypothesis. However, in the problem setting of label shift, the true label marginals are in fact changing. Hence, we control the dynamic regret against a sequence of re-weighted hypotheses in hindsight. All proofs for the next sub-section are deferred to App. C.
3.1 Proposed algorithm and performance guarantees
We start by presenting our assumptions. This is followed by the main algorithm for UOLS and its performance guarantees. Similar to the treatment in Bai et al. [8], we assume the following.
Assumption 1. Assume access to the true label marginals  $q_0 \in \Delta_K$  of the offline training data and the true confusion matrix  $C \in \mathbb{R}^{K \times K}$  with  $C_{ij} = E_{x \sim Q_0(\cdot | y = j)}, f_0(i|x)$ . Further the minimum singular value  $\sigma_{\min}(C) = \Omega(1)$  is bounded away from zero.
As noted in prior work [51, 26], the invertibility of the confusion matrix holds whenever the classifier  $f_{0}$  has good accuracy and the true label marginal  $q_{0}$  assigns a non-zero probability to each label. Though we assume perfect knowledge of the label margins of the training data and the associated confusion matrix, this restriction can be easily relaxed to their empirical counterparts computable from the training data. The finite sample error between the empirical and population quantities can be bounded by  $O(1 / \sqrt{N})$  where  $N$  is the number of initial training data samples. To this end, we operate in the regime where the time horizon obeys  $T = O(\sqrt{N})$ . However, similar to Bai et al. [8], we make this assumption mainly to simplify presentation without trivializing any aspect of the OLS problem.
Next, we present our assumptions on the loss function. Let  $p \in \Delta_K$ . Consider a classifier that predicts a label  $\widehat{y}(x)$ , by sampling  $\widehat{y}(x)$  according to the distribution that assigns a weight  $\frac{p(i)}{q_0(i)} f_0(i|x)$  to the label  $i$ . Define  $L_t(p)$  to be any non-negative loss that ascertains the quality of the marginal  $p$ . For example,  $L_t(p) = E[\ell(\widehat{y}(x), y)]$  where the expectation is taken wrt the randomness in the draw  $(x,y) \sim Q_t$  and in sampling  $\widehat{y}(x)$ . Here  $\ell$  is any classification loss (e.g. 0-1, cross-entropy).
Assumption 2 (Lipschitzness of loss functions). Let  $\mathcal{D}$  be a compact and convex domain. Assume that  $L_{t}(p)$  is  $G$  Lipschitz with  $p\in \mathcal{D}\subseteq \Delta_K$ , i.e.,  $L_{t}(p_{1}) - L_{t}(p_{2})\leq G\| p_{1} - p_{2}\|_{2}$  for any  $p_1,p_2\in \mathcal{D}$ . The constant  $G$  need not be known ahead of time.
We show in Lemmas 11 and 12 that the above assumption is satisfied under mild regularity conditions. Furthermore, the prior works such as Wu et al. [76] and Bai et al. [8] also require that losses are Lipschitz with a known Lipschitz constant apriori to set the step sizes for their OGD based methods.
The main goal here is to design appropriate re-weighting estimates such that the dynamic regret:
$$
R _ {\text {d y n a m i c}} (T) = \sum_ {t = 1} ^ {T} L _ {t} \left(\hat {q} _ {t}\right) - L _ {t} \left(q _ {t}\right) \leq \sum_ {t = 1} ^ {T} G \| \hat {q} _ {t} - q _ {t} \| _ {2} \tag {1}
$$
is controlled where  $\hat{q}_t \in \Delta_K$  is the estimate of the true label marginal  $q_t$ . Thus we have reduced the problem of handling OLS to the problem of online estimation of the true label marginals.
Under label shift, we can get an unbiased estimate of the true marginals at any round via the techniques in Lipton et al. [51], Azizzadenesheli et al. [2], Alexandari et al. [1]. More precisely,  $s_t = C^{-1}f_0(x_t)$  has the property that  $E[s_t] = q_t$  (see Lemma 15). Further, the variance of the estimate  $s_t$  is bounded by  $1 / \sigma_{min}^2(C)$ . Unfortunately, these unbiased estimates can not be directly used to track the moving marginals  $q_t$ . This is because the total squared error  $\sum_{t=1}^{T}E[\|s_t - q_t\|_2^2]$  grows linearly in  $T$  as the sum of the variance of the point-wise estimates accumulates unfavorably over time.
To get around these issues, one can use online regression algorithms such as FLH [38] with online averaging base learners or the Aligator algorithm [7]. These algorithms use ensemble methods to (roughly) output running averages of  $s_t$  where the variation in the true label margins is small enough. The averaging within intervals where the true marginals change slowly helps to reduce the overall variance while injecting only a small bias. We use such online regression oracles to track the moving marginals and re-calibrate the initial classifier. Overall, Algorithm 2 summarizes our method which has the following performance guarantee.
Theorem 2. Suppose we run Algorithm 2 with the online regression oracle ALG as FLH-FTL (App. F) or Aligator [7]. Then under Assumptions 1 and 2, we have
$$
E [ R _ {d y n a m i c} (T) ] = \tilde {O} \left(\frac {K ^ {1 / 6} T ^ {2 / 3} V _ {T} ^ {1 / 3}}{\sigma_ {m i n} ^ {2 / 3} (C)} + \frac {\sqrt {K T}}{\sigma_ {m i n} (C)}\right),
$$
where  $V_{T} \coloneqq \sum_{t=2}^{T} \| q_{t} - q_{t-1} \|_{1}$  and the expectation is taken with respect to randomness in the revealed co-variates. Further, this result is attained without prior knowledge of  $V_{T}$ .
Remark 3. We emphasize that any valid online regression oracle ALG can be plugged into Algorithm 2. This implies that one can even use transformer-based time series models to track the moving marginals  $q_{t}$ . Further, we have the flexibility of choosing the initial classifier to be any black-box model that outputs a distribution over the labels.
Framework 3 Supervised Online Label Shift (SOLS) protocol
input A hypothesis class  $\mathcal{H}$
1: for each round  $t \in [T]$  do
2: Nature samples  $N$  iid data points  $x_{t,1:N} \in \mathcal{X}$  and  $y_{t,1:N} \in \mathcal{Y}$ , with each  $(x_{t,i}, y_{t,i}) \sim Q_t$ ;  $x_{t,1:N}$  is revealed to the learner.
3: For each  $i \in [N]$ , learner predicts a label  $f_{t}(x_{t,i})$ .
4: The label  $y_{t,i}\in \mathcal{V}$  for each  $i\in [N]$  is revealed.
5:  $f_{t + 1} = \mathcal{A}(f_t,\{x_{1:t,1:N},y_{1:t,1:N}\})$  where algorithm  $\mathcal{A}$  updates the classifier with past data.
6: end for
Algorithm 4 TrainByWeights to handle SOLS
input Online regression oracle ALG, hypothesis class  $\mathcal{H}$
1: At round  $t\in [T]$  get estimated label marginal  $\hat{q}_t$  from  $\mathrm{ALG}(s_{1:t - 1})$
2: Update the hypothesis with weighted ERM:
$$
f _ {t} = \underset {f \in \mathcal {H}} {\operatorname {a r g m i n}} \sum_ {i = 1} ^ {t - 1} \sum_ {j = 1} ^ {N} \frac {\hat {q} _ {t} \left(y _ {i , j}\right)}{\hat {q} _ {i} \left(y _ {i , j}\right)} \ell \left(f \left(x _ {i, j}\right), y _ {i, j}\right) \tag {2}
$$
3: Get co-variates  $x_{t,1:N}$  and make predictions with  $f_t$
4: Get labels  $y_{t,1:N}$
5: Compute  $s_t[i] = \frac{1}{N} \sum_{j=1}^{N} \mathbb{I}\{y_{t,j} = i\}$  for all  $i \in [K]$ .
6: Update ALG with the empirical label marginals  $s_t$ .