SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
\begin{array}{l} \sum_ {j = b} ^ {c} \| \hat {\theta} _ {j} - \theta_ {j} \| _ {2} ^ {2} \leq \sum_ {j = b} ^ {c} \| \hat {\theta} _ {j} - \tilde {\theta} _ {j} \| _ {2} ^ {2} + \| \theta_ {j} - \tilde {\theta} _ {j} \| _ {2} ^ {2} \\ \leq 5 K \sigma^ {2} \log (2 T / \delta) \log (T) + \phi \cdot n _ {i} ^ {1 / 3} C _ {i} ^ {2 / 3}, \\ \end{array}
$$
with probability at-least  $1 - \delta /T$ . Here  $n_i\coloneqq b - c + 1$  and  $C_i\coloneqq \mathrm{TV}[b\to c]$ . Further we have that  $\| \hat{\theta}_{c + 1} - \theta_{c + 1}\| _2^2\leq 2B^2$  due to the boundedness condition in Definition 1.
Thus overall we have that  $\sum_{j = b}^{c + 1} = \tilde{O}\big(K + n_i^{1 / 3}C_i^{2 / 3}\big)$ , with probability at-least  $1 - \delta$  for any interval  $[b,c + 1]$  such that condition in Line 5 is satisfied at times  $b - 1$  and  $c + 1$ . Thus we have that
$$
\begin{array}{l} \sum_ {t = 1} ^ {T} \| \hat {\theta} _ {j} - \theta_ {j} \| _ {2} ^ {2} \stackrel {<  } {_ \sim} \sum_ {i = 1} ^ {M} K + n _ {i} ^ {1 / 3} C _ {i} ^ {2 / 3} \\ \prec_ {(a)} T ^ {1 / 3} V _ {T} ^ {2 / 3} K ^ {1 / 3} + \sum_ {i = 1} ^ {M} n _ {i} ^ {1 / 3} C _ {i} ^ {2 / 3} \\ \prec_ {(b)} T ^ {1 / 3} V _ {T} ^ {2 / 3} K ^ {1 / 3} + \left(\sum_ {i = 1} ^ {M} n _ {i}\right) ^ {1 / 3} \left(\sum_ {i = 1} ^ {M} C _ {i}\right) ^ {2 / 3} \\ \precsim T ^ {1 / 3} V _ {T} ^ {2 / 3} K ^ {1 / 3}, \\ \end{array}
$$
with probability at-least  $1 - \delta$ . In line (a) we used Eq.(15). In line (b) we used Holder's inequality with the dual norm pair  $(3,3 / 2)$ . This concludes the proof.
We now present the tweak of Algorithm 4 by instantiating ALG with Algorithm 5 and prove its regret guarantees. The resulting algorithm is described in Algorithm 6.
Algorithm 6 Lazy-TrainByWeights: handling label shift with sparse ERM calls
Input: Instance ALG of Algorithm 5, A hypothesis Class  $\mathcal{H}$
1: At round  $t \in [T]$ , get estimated label marginal  $\hat{q}_t \in \mathbb{R}^K$  from  $\mathrm{ALG}(s_{1:t-1})$ .
2: if  $\hat{q}_t = = \hat{q}_{t - 1}$  then
3:  $h_t = h_{t-1}$
4: else
5: Update the hypothesis by calling a weighted-ERM oracle:
$$
h _ {t} = \operatorname * {a r g m i n} _ {h \in \mathcal {H}} \sum_ {i = 1} ^ {t - 1} \sum_ {j = 1} ^ {N} \frac {\hat {q} _ {t} \left(y _ {i , j}\right)}{\hat {q} _ {i} \left(y _ {i , j}\right)} \ell \left(h \left(x _ {i, j}\right), y _ {i, j}\right)
$$
6: end if
7: Get  $N$  co-variates  $x_{t,1:N}$  and make predictions according to  $h_t$
8: Get labels  $y_{t,1:N}$
9: Compute  $s_t[i] = \frac{1}{N} \sum_{j=1}^{N} \mathbb{I}\{y_{t,j} = i\}$  for all  $i \in [K]$ .
10: Update ALG with the empirical label marginals  $s_t$ .
Theorem 22. Assume the same notations as in Theorem 6. Suppose we run Algorithm 6 (see Appendix E) with ALG instantiated using Algorithm 5 with  $\sigma^2 = 1 / N$  and predictions clipped as in Theorem 6. Further let the online regression oracle given to Algorithm 5 be chosen as one of the candidates mentioned in Remark 19. Then with probability at-least  $1 - \delta$ , we have that
$$
R _ {d y n a m i c} ^ {\mathcal {H}} = \tilde {O} \left(T ^ {2 / 3} V _ {T} ^ {1 / 3} + \sqrt {T \log (| \mathcal {H} | / \delta)}\right).
$$
Further, the number of number of calls to ERM oracle (via Step 5) is at-most  $\tilde{O}\big(T^{1 / 3}V_T^{2 / 3}\big)$  with probability at-least  $1 - \delta$
Sketch. The proof of this theorem closely follows the steps fused for proving Theorem 6. So we only highlight the changes that need to be incorporated to the proof of Theorem 6.
Replace the use of Proposition 20 in the proof of Theorem 6 with Theorem 18.
For any round  $t$ , where Step 5 of Algorithm 6 is triggered, we can use the same arguments as in the Proof of Theorem 22 to bound the instantaneous regret by Eq.(13). i.e:
$$
L _ {t} \left(h _ {t}\right) - L _ {t} \left(h _ {t} ^ {*}\right) \leq \frac {2 B \sqrt {K}}{\mu^ {2}} \left(\| \hat {q} _ {t} - q _ {t} \| _ {2} + E \left[ \| \hat {q} _ {t} - q _ {t} \| _ {2} \right] + \phi \cdot \frac {V _ {T} ^ {1 / 3}}{(t - 1) ^ {1 / 3}} + \sqrt {\frac {\log (3 T | \mathcal {H} | / \delta)}{(t - 1)}}\right), \tag {16}
$$
For a round  $t$ , where Step 5 is not triggered, we proceed as follows:
Let  $t'$  be the most recent time step prior to  $t$  when Step 5 is executed. Notice that the population loss can be equivalently represented as
$$
L _ {t} (h) = \frac {1}{N \left(t ^ {\prime} - 1\right)} \sum_ {i = 1} ^ {t ^ {\prime} - 1} \sum_ {j = 1} ^ {N} E \left[ \frac {q _ {t} \left(y _ {i j}\right)}{q _ {i} \left(y _ {i j}\right)} \ell \left(h \left(x _ {i j}\right), y _ {i j}\right) \right],
$$
$$
L _ {t} ^ {\mathrm {e m p}} (h) := \frac {1}{N \left(t ^ {\prime} - 1\right)} \sum_ {i = 1} ^ {t ^ {\prime} - 1} \sum_ {j = 1} ^ {N} \frac {q _ {t} \left(y _ {i j}\right)}{q _ {i} \left(y _ {i j}\right)} \ell \left(h \left(x _ {i j}\right), y _ {i j}\right),
$$
$$
\tilde {L} _ {t} (h) := \frac {1}{N \left(t ^ {\prime} - 1\right)} \sum_ {i = 1} ^ {t ^ {\prime} - 1} \sum_ {j = 1} ^ {N} E \left[ \frac {\hat {q} _ {t} \left(y _ {i j}\right)}{\hat {q} _ {i} \left(y _ {i j}\right)} \ell \left(h \left(x _ {i j}\right), y _ {i j}\right) \right],
$$
$$
\tilde {L} _ {t} ^ {\mathrm {e m p}} (h) := \frac {1}{N \left(t ^ {\prime} - 1\right)} \sum_ {i = 1} ^ {t ^ {\prime} - 1} \sum_ {j = 1} ^ {N} \frac {\hat {q} _ {t} \left(y _ {i j}\right)}{\hat {q} _ {i} \left(y _ {i j}\right)} \ell \left(h \left(x _ {i j}\right), y _ {i j}\right).
$$
$$
\begin{array}{l} L _ {t} (h _ {t}) - L _ {t} (h _ {t} ^ {*}) = L _ {t} (h _ {t}) - \tilde {L} _ {t} (h _ {t}) + \tilde {L} _ {t} (h _ {t}) - \tilde {L} _ {t} ^ {\mathrm {e m p}} (h _ {t}) + L _ {t} ^ {\mathrm {e m p}} (h _ {t} ^ {*}) - L _ {t} (h _ {t} ^ {*}) + \tilde {L} _ {t} ^ {\mathrm {e m p}} (h _ {t}) - L _ {t} ^ {\mathrm {e m p}} (h _ {t} ^ {*}) \\ \leq \underbrace {L _ {t} (h _ {t}) - \tilde {L} _ {t} (h _ {t})} _ {\text {T 1}} + \underbrace {\tilde {L} _ {t} (h _ {t}) - \tilde {L} _ {t} ^ {\text {e m p}} (h _ {t})} _ {\text {T 2}} + \underbrace {L _ {t} ^ {\text {e m p}} (h _ {t} ^ {*}) - L _ {t} (h _ {t} ^ {*})} _ {\text {T 3}} + \underbrace {\tilde {L} _ {t} ^ {\text {e m p}} (h _ {t} ^ {*}) - L _ {t} ^ {\text {e m p}} (h _ {t} ^ {*})} _ {\text {T 4}}, \\ \end{array}
$$
By using the same arguments as in Proof of Theorem 6 and replacing the use of Proposition 20 with Theorem 18, we can bound T1-4. This will result in an instantaneous regret bound at round  $t$  (which doesn't trigger step 5) as:
$$
\begin{array}{l} L _ {t} \left(h _ {t}\right) - L _ {t} \left(h _ {t} ^ {*}\right) \leq \frac {2 B \sqrt {K}}{\mu^ {2}} \left(\| \hat {q} _ {t} - q _ {t} \| _ {2} + E [ \| \hat {q} _ {t} - q _ {t} \| _ {2} ] + \phi \cdot \frac {V _ {T} ^ {1 / 3}}{\left(t ^ {\prime} - 1\right) ^ {1 / 3}} + \sqrt {\frac {\log (3 T | \mathcal {H} | / \delta)}{\left(t ^ {\prime} - 1\right)}}\right), \\ \leq \frac {2 B \sqrt {K}}{\mu^ {2}} \left(\| \hat {q} _ {t} - q _ {t} \| _ {2} + E [ \| \hat {q} _ {t} - q _ {t} \| _ {2} ] + \phi \cdot 4 ^ {1 / 3} \cdot \frac {V _ {T} ^ {1 / 3}}{(t - 1) ^ {1 / 3}} + \sqrt {\frac {4 \log (3 T | \mathcal {H} | / \delta)}{(t - 1)}}\right), \\ \end{array}
$$
with probability at-least  $1 - \delta /T$ . In the last line we used the fact that  $t^\prime -1\geq (t / 2) - 1\geq (t - 1) / 4$  for all  $t\geq 3$ .
Now adding Eq.(16) and (17) across all rounds and proceeding similar to the proof of Theorem 6 (and replacing the use of Proposition 20 with Theorem 18) completes the argument.
We next prove the matching (up to factors of  $\log T$ ) lower bound.
Proof. First we fix the hypothesis class and the data generation strategy. In the problem instance we consider, there are no co-variates. The hypothesis class is defined as
$$
\mathcal {H} := \left\{h _ {p}: h _ {p} \text {p r e d i c t s a l a b e l} y \sim \operatorname {B e r} (p); p \in [ | \mathcal {H} | ] \right\}.
$$
Further we design the hypothesis class such that both  $h_0, h_1 \in \mathcal{H}$ . Next we fix the data generation strategy:
- Divide the time horizon into batches of length  $\Delta$ .
- At the beginning of a batch  $i$ , the adversary picks  $\mathring{q}_i$  uniformly at random from  $\{1/2 - \delta, 1/2 + \delta\}$ .
- For all rounds  $t$  that falls within batch  $i$ , the label  $y_{t} \sim \mathrm{Ber}(q_{t})$  is sampled with  $q_{t} \coloneqq \mathring{q}_{i}$ .
- Learner predicts a label  $\hat{y}_t \in \{0,1\}$  and then the actual label  $y_t$  is revealed (hence  $N = 1$  in the protocol of Fig.3).
- Learner suffers a loss given by  $\ell_t(\hat{y}_t) = \mathbb{I}\{\hat{y}_t \neq y_t\}$ .
It is easy to see that the losses are bounded in  $[0,1]$ . Now let's examine the two possibilities of generating labels within a batch. Let's upper bound the variation incurred by the label margins:
$$
\begin{array}{l} \sum_ {t = 2} ^ {T} \left| q _ {t} - q _ {t - 1} \right| = \sum_ {i = 2} ^ {M} \left| \mathring {q} _ {i} - \mathring {q} _ {i - 1} \right| \\ \leq 2 \delta M \\ \leq V _ {T}, \\ \end{array}
$$
where the last line is obtained by choosing  $\delta = V_{T} / (2M) = V_{T}\Delta /(2T)$ .
Since at the beginning of each batch, the sampling probability of true labels is independently renewed, the historical data till the beginning of a batch is immaterial in minimising the loss within the batch. So we can lower bound the regret within each batch separately and add them up. Below, we focus on lower bounding the regret in batch 1 and the computations are similar for any other batch.
Suppose that the probability that an algorithm predict label  $y_{t} = 1$  is  $\hat{q}_t$ , where  $\hat{q}_t$  is a measurable function of the past data  $y_{1:t-1}$ . Then we have that the population loss  $L_t(\hat{q}_t) := E[\ell_t(\hat{y}_t)] = (1 - \hat{q}_t)q_t + \hat{q}_t(1 - q_t)$ . Here we abuse the notation  $L(q_t) := L(h_{q_t})$ . We see that the population loss  $L_t(\hat{q}_t)$  are convex and its gradient obeys  $\nabla L_t(\hat{q}_t) = 1 - 2q_t = E[1 - 2y_t]$  since by our construction  $y_t \sim \mathrm{Ber}(q_t)$ . Thus the population losses are convex and its gradients can be estimated in an unbiased manner from the data.
We use the following Proposition due to Besbes et al. [10].
Proposition 23 (due to Lemma A-1 in Besbes et al. [10]). Let  $\tilde{\mathbb{P}}$  denote the joint probability of the label sequence  $y_{1:\Delta}$  within a batch when they are generated using  $\operatorname{Ber}(1/2 - \delta)$ . So
$$
\tilde {\mathbb {P}} \left(y _ {1}, \dots , y _ {\Delta}\right) = \Pi_ {i = 1} ^ {\Delta} \left(1 / 2 - \delta\right) ^ {y _ {i}} \left(1 / 2 + \delta\right) ^ {1 - y _ {i}}.
$$
Similarly define  $\tilde{\mathbb{Q}}$  as
$$
\tilde {\mathbb {Q}} (y _ {1}, \dots , y _ {\Delta}) = \Pi_ {i = 1} ^ {\Delta} (1 / 2 + \delta) ^ {y _ {i}} (1 / 2 - \delta) ^ {1 - y _ {i}}.
$$
According to the above distribution the data are independently sampled from Bernoulli trials with success probability  $1 / 2 + \delta$ . Let  $\hat{q}_t$  be the decision of the online algorithm  $qt$  round  $t$  so that the algorithm predicts label  $1$  with probability  $\hat{q}_t$ .
Let  $\mathbb{P}$  denote the joint probability distribution across the decisions  $\hat{q}_{1:\Delta}$  of any online algorithm under the sampling model  $\tilde{\mathbb{P}}$ . Similarly define  $\mathbb{Q}$ . Note that any online algorithm can make decisions at round  $t$  only based on the past observed data  $y_{1:t-1}$ . Further after making the decision  $\hat{q}_t$  at round  $t$ , an unbiased estimate of the population loss can be constructed due to the fact that  $\nabla L_t(\hat{q}_t) = E[1 - 2y_t]$ . Under the availability of unbiased gradient estimates of the losses, it holds that
$$
K L (\mathbb {P} | | \mathbb {Q}) \leq 4 \Delta \delta^ {2}.
$$
By choosing  $\delta = V_T / (2M) = V_T\Delta /(2T)$  and  $\Delta = (T / V_T)^{2 / 3}$ , we get that  $KL(\mathbb{P}||\mathbb{Q})\leq 1$
Since  $V_{T}\leq T / 8$ , the above choice implies that  $\delta \in (0,1 / 4)$ .
For notational clarity, define  $L^{\mathbb{P}}(q) = (1 - q)(1/2 - \delta) + q(1/2 + \delta)$  and  $L^{\mathbb{Q}}(q) = (1 - q)(1/2 + \delta) + q(1/2 - \delta)$ . These corresponds to the population losses according to the sampling models  $\mathbb{P}$  and  $\mathbb{Q}$  respectively. Observe that  $\min_q L^{\mathbb{P}}(q) = \min_q L^{\mathbb{Q}}(q) = 1/2 - \delta$ . The minimum of  $L^{\mathbb{P}}$  and  $L^{\mathbb{Q}}$  are achieved at 0 and 1 respectively. Note that both  $h_0, h_1 \in \mathcal{H}$ . So there is always a hypothesis in  $\mathcal{H}$  that corresponds the minimiser of the loss.
Further whenever  $\hat{q} \geq 1/2$  we have that
$$
\begin{array}{l} L ^ {\mathbb {P}} (q) = (1 / 2 - \delta) + q (2 \delta) \\ \geq 1 / 2. \\ \end{array}
$$
Similarly whenever  $q < 1/2$  we have  $L^{\mathbb{Q}}(q) \geq 1/2$ . So we define the selector function as  $\phi_t := \mathbb{I}\{\hat{q}_t \geq 1/2\}$ . Let  $q_t^* \in \{0,1\}$  be the minimiser of the loss at round  $t$ . Now we can lower bound the instantaneous regret similar as
$$
\begin{array}{l} E \left[ L _ {t} \left(\hat {q} _ {t}\right) - L _ {t} \left(q _ {t} ^ {*}\right) \right] = \frac {1}{2} \left(E _ {\mathbb {P}} \left[ L _ {t} ^ {\mathbb {P}} \left(\hat {q} _ {t}\right) - L _ {t} ^ {\mathbb {P}} (0) \right] + \frac {1}{2} \left(E _ {\mathbb {Q}} \left[ L _ {t} ^ {\mathbb {Q}} \left(\hat {q} _ {t}\right) - L _ {t} ^ {\mathbb {Q}} (1) \right] \right. \right. \\ \geq \frac {1}{2} (E _ {\mathbb {P}} [ L _ {t} ^ {\mathbb {P}} (\hat {q} _ {t}) - L _ {t} ^ {\mathbb {P}} (0) | \phi_ {t} = 1 ] \mathbb {P} (\phi_ {t} = 1) + \frac {1}{2} (E _ {\mathbb {Q}} [ L _ {t} ^ {\mathbb {Q}} (\hat {q} _ {t}) - L _ {t} ^ {\mathbb {Q}} (1) | \phi_ {t} = 0 ] \mathbb {Q} (\phi_ {t} = 0) \\ \geq \delta / 2 \max  \left\{\mathbb {P} \left(\phi_ {t} = 1\right), \mathbb {Q} \left(\phi_ {t} = 0\right) \right\} \\ \geq (\delta / 8) e ^ {- 1}, \\ \end{array}
$$
where the last line is obtained by Propositions 23 and 17.
Thus we get a total lower bound on the instantanoeous regret as
$$
\begin{array}{l} \sum_ {t = 1} ^ {T} E \left[ L _ {t} \left(\hat {q} _ {t}\right) - L _ {t} \left(q _ {t} ^ {*}\right) \right] \geq T \delta / (8 e) \\ = \Delta V _ {T} / (1 6 e) \\ = T ^ {2 / 3} V _ {T} ^ {1 / 3} / (1 6 e), \\ \end{array}
$$
where the last line is obtained by using our choices of  $\delta V_{T}\Delta /(2T)$  and  $\Delta = (T / V_T)^{2 / 3}$ .
The second term of  $\Omega (\sqrt{T\log|\mathcal{H}|})$  can be obtained from the existing results on statistical learning theory without distribution shifts. (see for example Theorem 3.23 in Mohri et al. [53]).
F More details on experiments
In Algorithm 7, we describe the FLH-FTL algorithm from Hazan and Seshadhri [38] when specialised to squared error losses. When specialized to squared error losses, this algorithm runs FLH with online averages as the base experts.
Algorithm 7 An instance of FLH-FTL from Hazan and Seshadhri [38] with squared error losses
1: Parameter  $\alpha$  is defined to be a learning rate // initializations and definitions
2: For FLH-FTL instantiations within UOLS algorithms (as in Algorithm 2), we set  $\alpha \gets \sigma_{min}^2(C) / (8K)$ , where  $C$  is the confusion matrix as in Assumption 1. For instantiations within SOLS algorithms (as in Algorithm 4) we set  $\alpha \gets 1 / (8K)$
3: For each round  $t \in [T]$ ,  $v_t \coloneqq (v_t^{(1)}, \ldots, v_t^{(t)})$  is a probability vector in  $\mathbb{R}^t$ . Initialize  $v_1^{(1)} \gets 1$
4: For each  $j \in [T]$ , define a base learner  $E^j$ . For each  $t > j$ , the base expert outputs  $\bar{E}^j(t) := \frac{1}{t - j} \sum_{i = j}^{t - 1} z_j$ , where  $z_j$  to be specified as below. Further  $E^j(j) := 0 \in \mathbb{R}^K$  // execution steps
5: In round  $t \in [T]$ , set  $\forall j \leq t, x_t^j \gets E^j(t)$  (the prediction of the  $j^{th}$  base learner at time  $t$ ). Play  $x_t = \sum_{j=1}^{t} v_t^{(j)} x_t^{(j)}$ .
6:Receive feedback  $z_{t}$  ,set  $\hat{v}_{t + 1}^{(t + 1)}\gets 0$  and perform update for  $1\leq i\leq t$  ..
$$
\hat {v} _ {t + 1} ^ {(i)} \gets \frac {v _ {t} ^ {(i)} e ^ {- \alpha \| x _ {t} ^ {(i)} - z _ {t} \| _ {2} ^ {2}}}{\sum_ {j = 1} ^ {t} v _ {t} ^ {(j)} e ^ {- \alpha \| x _ {t} ^ {(j)} - z _ {t} \| _ {2} ^ {2}}}
$$
7: Addition step - Set  $v_{t+1}^{(t+1)}$  to  $1/(t+1)$  and for  $i \neq t+1$ :
$$
v _ {t + 1} ^ {(i)} \gets (1 - (t + 1) ^ {- 1}) \hat {v} _ {t + 1} ^ {(i)}
$$
Rationale behind the learning rate setting at Line 2 of Algorithm 7 The loss that is incurred by Algorithm 7 and any of its base learners at round  $t$  is defined to be the squared error loss  $\ell_t(x) = \| z_t - x\| _2^2$ . Whenever  $\| z_t\| _2^2\leq B^2$  and  $\| x\| _2^2\leq B^2$ , the losses  $\ell_t(x)$  are  $1 / (8B^{2})$  exp-concave (see for eg. Chapter 3 of [12]). The notion of exp-concavity is crucial for FLH-FTL algorithm since the learning rate is set to be equal to the exp-concavity factor of the loss functions (see Theorem 3.1 in Hazan and Seshadhri [38]).
For the UOLS problem, from Algorithm 2, we have  $\| z_t\| _2 = \| C^{-1}f_0(x_t)\| _2\leq \sqrt{K} /\sigma_{min}(C)$ . Since the decisions of the algorithm is a convex combination of the previously seen  $z_{t}$ , we conclude that the losses  $\ell_t(x)$  are  $\sigma_{min}^2 (C) / (8K)$  exp-concave.
For the SOLS problem, let  $z_{t} = s_{t}$  where  $s_t$  is as defined in Algorithm 4. We have that  $\| z_{t}\|_{2}\leq \sqrt{K}$ . Hence arguing in a similar fashion as above, we conclude that the losses  $\ell_t(x)$  are  $1 / (8K)$  exp-concave for the SOLS problem.
This is the motivation behind Line 2 in Algorithm 7, where the learning rates are set according to the problem setting.
Dataset and model details.
- Synthetic: For the synthetic data, we generated 72k samples as described in Bai et al. [8]. There are three classes each with 24k samples generated from three Gaussian distributions in  $\mathbb{R}^{12}$ . Each Gaussian distribution is defined by a randomly generated unit-norm centre  $v$  and covariance matrix  $0.215 \cdot I$ . 60k samples are used as source data, and 12k samples are used as target data to be sampled from during online learning. We used logistic regression to train a linear model. It is trained for a single epoch with learning rate 0.1, momentum 0.9, batch size 200, and  $l_{2}$  regularization  $1 \times 10^{-4}$ .
- MNIST [50]: An image dataset of 10 types of handwritten digits. 60k samples are used as source data and 10k as target data. We used an MLP for prediction with three consecutive
hidden layers of sizes 100, 100, and 20. It is trained for a single epoch with a learning rate 0.1, momentum 0.9, batch size 200, and  $l_{2}$  regularization  $1 \times 10^{-4}$ .
- CIFAR-10 [49]: A dataset of colored images of 10 items: airplane, automobile, bird, cat, deer, dog, frog, horse, ship, and truck. 50k samples are used as source data and 10k as target data. We train a ResNet18 model (He et al. [39]) from scratch. It is finetuned for 70 epochs with learning rate 0.1, momentum 0.9, batch size 200, and  $l_{2}$  regularization  $1 \times 10^{-4}$ . The learning rate decayed by  $90\%$  at the 25th and 40th epochs.
- Fashion [77]: An image dataset of 10 types of fashion items: T-shirt, trouser, pullover, dress, coat, sandals, shirt, sneaker, bag, and ankle boots. 60k samples are used as source data and 10k as target data. We trained an MLP for prediction. It is trained for 50 epochs with learning rate 0.1, momentum 0.9, batch size 200, and  $l_{2}$  regularization  $1 \times 10^{-4}$ .
- EuroSAT [40]: An image dataset of 10 types of land uses: industrial buildings, residential buildings, annual crop, permanent crop, river, sea & lake, herbaceous vegetation, highway, pasture, and forest. 60k samples are used as source data and 10k as target data. We cropped the images to the size (3, 64, 64). We train a ResNet18 model for 50 epochs with learning rate 0.1, momentum 0.9, batch size 200, and  $l_{2}$  regularization  $1 \times 10^{-4}$ .
- Arxiv [15]: A natural language dataset of 23 classes over different publication subjects. 198k samples are used as source data and 22k as target data. We trained a DistilBERT model (Sanh et al. [62]) for 50 epochs with learning rate  $2 \times 10^{-5}$ , batch size 64, and  $l_{2}$  regularization  $1 \times 10^{-2}$ .
- SHL [31, 71]: A tabular locomotion dataset of 6 classes of human motion: still, walking, run, bike, car and bus. 30k samples are used as source data and 70k as target data. We trained an MLP for prediction for 50 epochs with learning rate 0.1, momentum 0.9, batch size 200, and  $l_{2}$  regularization  $1 \times 10^{-4}$ .
For all the datasets above, the initial offline data are further split by  $80:20$  into training and holdout data, where the former is used for offline training of the base model and the latter for computing the confusion matrix and retraining (e.g. updating the linear head parameters with UOGD or updating the softmax prediction with our FLT-FTL) during online learning. To examine how well the algorithms adapt when holdout data is limited, we use  $10\%$  of the holdout data (i.e.,  $2\%$  of the initial offline data) in the main paper unless stated otherwise. In App. G.2, we ablate with full hold-out data.
Types of Simulated Shifts. We simulate four kinds of label shifts to capture different non-stationary environments. These shifts are similar to the ones used in Bai et al. [8]. For each shift, the label margins are a mixture of two different constant marginals  $\mu_1, \mu_2 \in \Delta_K$  with a time-varying coefficient  $\alpha_t$ :  $\mu_{y_t} = (1 - \alpha_t)\mu_1 + \alpha_t\mu_2$ , where  $\mu_{y_t}$  denotes the label distribution at round  $t$  and  $\alpha_t$  controls the shift non-stationarity and patterns. In particular, we have: Sinusoidal Shift (Sin):  $\alpha_t = \sin \frac{i\pi}{L}$ , where  $i = t \mod L$  and  $L$  is a given periodic length. In the experiments, we set  $L = \sqrt{T}$ . Bernoulli Shift (Ber): at every iteration, we keep the  $\alpha_t = \alpha_{t-1}$  with probability  $p \in [0,1]$  and otherwise set  $\alpha_t = 1 - \alpha_{t-1}$ . In the experiments, the parameter is set as  $p = 1/\sqrt{T}$ , which implies  $V_t = \Theta(\sqrt{T})$ . Square Shift (Squ): at every  $L$  rounds we set  $\alpha_t = 1 - \alpha_{t-1}$ . Monotone Shift (Mon): we set  $\alpha_t = t/T$ . Square, sinusoidal, and Bernoulli shifts simulate fast-changing environments with periodic patterns.
Methods for UOLS Adaptation.
- Base: the base classifier without any online modifications.
- OFC: the optimal fixed classifier predicts with an optimal fixed re-weighting in hindsight as in Wu et al. [76].
- Oracle: re-weight the base model's predictions with the true label marginal of the unlabeled data at each time step.
- FTH: proposed by Wu et al. [76], follow-the-history classifier re-weights the base model's predictions with a simple online average of all marginal estimates seen thus far.
- FTFWH: proposed by Wu et al. [76], follow-the-fixed-window-history classifier is a version of FTH that tracks only the  $k$  most recent marginal estimates. We choose  $k = 100$  throughout the experiments in this work.
CT (base)	CT-RS (ours) w FLH	CT-RS (ours) w FLT-FTL	w-ERM (oracle)
MNIST	CI Err	5.0±0.5	4.71±0.2	4.53±0.1	3.2±0.4
MSE	NA	0.12±0.01	0.08±0.01	NA
Table 5: Results on SOLs setup. We report results with MNIST SOLs setup runs for  $T = 200$  steps. We observe that continual training with re-sampling improves over the base model which continually trains on the online data and achieves competitive performance with respect to weighted ERM oracle.
CT-RS (ours)	w-ERM (oracle)
CIFAR	145±3.7	1882±14
MNIST	20±2.7	107±3.6
Table 6: Comparison on computation time (in minutes). We report results with MNIST and CIFAR SOLS setup runs for  $T = 200$  steps. We observe that continual training with re-sampling is approximately  $5 - 15 \times$  more efficient than weighted ERM oracle.
- ROGD: proposed by Wu et al. [76], ROGD uses online gradient descent to update its reweighting vector based on current marginal estimate.
- UOGD: proposed by Bai et al. [8], retransits the last linear layer of the model based on current marginal estimate.
- ATLAS: proposed by Bai et al. [8] is a meta-learning algorithm that has UOGD as its base learners.