Title: Test-time Adaptation in Non-stationary Environments via Adaptive Representation Alignment
Venue: NeurIPS 2024
Accepted: yes
Keywords: non-stationary environments

continuous distribution shift

test-time adaptation

unlabeled data
Abstract: Adapting to distribution shifts is a critical challenge in modern machine learning, especially as data in many real-world applications accumulate continuously in the form of streams. We investigate the problem of sequentially adapting a model to non-stationary environments, where the data distribution is continuously shifting and only a small amount of unlabeled data are available each time. Continual test-time adaptation methods have shown promising results by using reliable pseudo-labels, but they still fall short in exploring representation alignment with the source domain in non-stationary environments. In this paper, we propose to leverage non-stationary representation learning to adaptively align the unlabeled data stream, with its changing distributions, to the source data representation using a sketch of the source data. To alleviate the data scarcity in non-stationary representation learning, we propose a novel adaptive representation alignment algorithm called Ada-ReAlign. This approach employs a group of base learners to explore different lengths of the unlabeled data stream, which are adaptively combined by a meta learner to handle unknown and continuously evolving data distributions. The proposed method comes with nice theoretical guarantees under convexity assumptions. Experiments on both benchmark datasets and a real-world application validate the effectiveness and adaptability of our proposed algorithm.

Test-time Adaptation in Non-stationary Environments via Adaptive Representation Alignment

Abstract

Adapting to distribution shifts is a critical challenge in modern machine learning, especially as data in many real-world applications accumulate continuously in the form of streams. We investigate the problem of sequentially adapting a model to non-stationary environments, where the data distribution is continuously shifting and only a small amount of unlabeled data are available each time . Continual testtime adaptation methods have shown promising results by using reliable pseudolabels, but they still fall short in exploring representation alignment with the source domain in non-stationary environments. In this paper, we propose to leverage non-stationary representation learning to adaptively align the unlabeled data stream, with its changing distributions, to the source data representation using a sketch of the source data. To alleviate the data scarcity in non-stationary representation learning, we propose a novel adaptive representation alignment algorithm called Ada-ReAlign. This approach employs a group of base learners to explore different lengths of the unlabeled data stream, which are adaptively combined by a meta learner to handle unknown and continuously evolving data distributions. The proposed method comes with nice theoretical guarantees under convexity assumptions. Experiments on both benchmark datasets and a real-world application validate the effectiveness and adaptability of our proposed algorithm.

1 Introduction

Machine learning algorithms, particularly deep learning models, have achieved remarkable success when the test data share the same distribution of the training data. However, in many real-world applications, the learning environment is changing over time, resulting in the test data inevitably encountering natural variations. For example, once an autonomous driving model is deployed, external factors such as weather changes (e.g., snow, frost, or fog) and internal factors like sensor degradation (e.g., causing Gaussian noise, defocus, or blur) can negatively impact its performance. Unfortunately, many well-trained models are highly sensitive to such distribution shifts and may suffer significant performance degradation, even when the shifts are minor [12]. Moreover, once deployed, models often lack access to the original training data, making it essential to equip the learning system with the ability to adapt to non-stationary environments in test time [29, 45].

A promising line of research is known as test-time adaptation (TTA), which focuses on adapting models to new environments using only unlabeled data. Pioneering approaches consider adaptation to a fixed distribution, including introducing auxiliary self-supervised learning tasks [30, 28] or employing

entropy minimization to update the model [26, 32]. In addition, aligning the representation of unlabeled data back to the source representation has also been proposed to improve performance [28, 10]. More recently, several seminal works have explored continual test-time adaptation with unlabeled data streams, where data arrive continuously, and the distribution may change over time. These algorithms often rely on generating and selecting high-confidence pseudo-labels to update the model, with techniques such as reliable entropy minimization [34, 25]. Non-parametric approaches have also be proposed, leveraging both source labels and pseudo-labeled data for adaptation [40]. Despite these advances, the exploration of effective representation alignment with the source data representation in non-stationary environments remains underexplored.

In this paper, we propose a non-stationary representation learning approach to adaptively align the unlabeled data stream with changing distributions to the source data representation, leveraging the marginal information from the source distribution. Our framework follows a common configuration used in prior work, which assumes the model comprises a representation learning module and a linear classifier [30, 32]. The core idea is to continually update the representation learning module so that it projects non-stationary unlabeled data at each time step to a distribution aligned with the source data representation, relying solely on the source marginal distribution. Different from prior representation alignment methods for fixed distributions [23, 28], the key challenge in non-stationary representation learning lies in the limited amount of data available at each time. Updating the model with only a single batch of data can introduce high variance due to the small sample size, while relying on the entire data stream risks high bias as the distribution evolves over time.

To handle the challenges posed by the unknown and ever changing distributions and limited data in each round, we propose a novel algorithm named adaptive representation alignment (Ada-ReAlign). Drawing inspiration from recent advances in non-stationary online learning [38, 43, 41], the representation learning module employs a group of base models learning with varying window sizes to explore different part of the data stream, along with a meta learner that adaptively combines their output to learn a well-aligned representation. We then further refine the linear classifier by applying entropy minimization, with regularization to the initial one, based on the aligned the data representation. Our algorithm enjoys nice theoretical guarantees with convex models and loss functions, providing a solid foundation for its empirical success. Through benchmark experiments and a real-world application, we demonstrate the effectiveness and efficiency of our method in improving performance under non-stationary environments. We summarize our contributions as follow:

We propose non-stationary representation learning for continual test-time adaptation, which adaptively aligns non-stationary data with the source representation using a sketch of source data.

We propose a novel online learning algorithm that adaptively aligns representations to the source by exploring variations across different periods of the data. Theoretical analysis demonstrates that the proposed method approximates the optimal model sequence for convex losses and models.

We demonstrate the effectiveness and efficiency of our method for test-time adaptation in nonstationary environments through benchmark experiments and a real-world application.

Organization. We first introduce related work in Section 2. In Section 3, we present our main results, including the proposed algorithm with corresponding theoretical analysis. Section 4 reports the experiment results, followed by the conclusion in Section 5. All proofs and omitted details are provided in the Appendix.

2 Related Work

In this section, we present the most relevant literature related to our learning setting and the techniques used in our approach. A more detailed description is provided in Appendix B .

Continual Test-time Adaptation in Non-stationary Environments. Recently, continual test-time adaptation in non-stationary environments has gained significant attention. Due to the evolving nature of streaming data, the data distribution naturally changes over time. A simple idea involved recovering the model weights from the initial model after each adaptation step of a mini-batch, such as MEMO [39], episodic Tent [32], or DDA [10]. However, these sequential one-step adaptation methods could be insufficient because they only explore the limited number of data at each round, and thus cannot explore the knowledge of the accumulated historical data.

To tackle this challenge, CoTTA [34] was introduced, generating robust pseudo-labels through a weighted average of historical models while preserving the initial model's parameters. It also stochastically replaces the model's parameters with the initial model's parameters at each round to resist distribution change. Building on this, EcoTTA [27] and BcoTTA [20] improved parameter and memory efficiency during continual adaptation. Similarly, SAR [25] updated the model based on reliable entropy and reset it to its initial state whenever the entropy exceeds a predefined threshold. AdaNPC [40] resisted non-stationarity by constructing a memory buffer to store historical distributions. Although these methods have demonstrated empirical success in various real-world applications, they rely on prior knowledge to estimate pseudo-labels or require access to source-labeled data during adaptation.

Adapting to Non-stationary Environments with Offline Labeled Data. This line of research focuses on specific types of distribution shifts, employing adaptive learning with weighted sourcelabeled data to handle continuous distribution changes. Previous studies have tackled the challenge of non-stationarity in the context of online label shift, where only the class priors change. In such scenarios, an unbiased loss estimator is used to estimate the loss at each round, enabling dynamic regret minimization in non-stationary environments [2]. Another work investigated the problem of continuous covariate shift, where only the input distribution changes [41]. They reframed this problem as an online density ratio estimation task and proposed a generic reduction of the density ratio estimation problem to dynamic regret optimization. However, these approaches assume access to the offline training data for adaptation, which may not always be feasible in practice.

3 Algorithm and Theory

We start by introducing the notations and problem formulation, followed by a detailed explanation of the proposed Ada-ReAlign algorithm and its theoretical analysis.

3.1 Problem Formulation

In this part, we first formulate the learning problem. We assume access to a well-trained model on source data, along with a sketch of them. During adaptation, unlabeled data arrive sequentially from non-stationary environments, where the underlying data distribution could change over time. Following previous work [30, 32], we assume the model consists of a representation learning module and a linear function as classification module. Let ϕ t ( · ) : X ↦→ R d be the representation learning module, and let the linear function be represented by a matrix w t ∈ R k × d where k = |Y| . Thus, the prediction model is defined as f t ( · ) = ⟨ w t , ϕ t ( · ) ⟩ . We denote the representation module of the well-trained initial model on the source data by ϕ 0 ( · ) and its corresponding linear function by w 0 .

As streaming data are collected in non-stationary environments, the underlying data distribution remains both unknown and ever-changing . The learning task is framed as a sequential prediction problem over T rounds, with T > 0 . At each round t ∈ [ T ] := { 1 , . . . , T } , the learner receives a batch of unlabeled data S t = { x t,i } N t i =1 , sampled independently and identically from the distribution D t , where N t ≥ 1 and D t could change over time. Our objective is to learn a sequence of models { f t } T t =1 that perform well across the evolving distributions {D t } T t =1 .

We define µ 0 ∈ R d and Σ 0 ∈ R d × d as the mean and covariance of the representation distribution of the source data with the initial model, where µ 0 = E x ∼S 0 [ ϕ 0 ( x )] , Σ 0 = E x ∼S 0 [( ϕ 0 ( x ) -µ 0 ) T ( ϕ 0 ( x ) -µ 0 )] , and d denotes the dimensionality of the feature embedding. This approach of sketching the source data does not rely on label information from the source data, which is particularly advantageous in tasks involving privacy concerns. Moreover, marginal information can be generated using coreset techniques [31, 16], eliminating the need for direct access to the original source data.

Following the online learning literature [11], we use dynamic regret as the performance measure. The performance of the model sequence { f t } T t =1 is evaluated through the average excess risk, defined as:

where R t ( f ) = E ( x ,y ) ∼D t [ ℓ ( f ( x ) , y )] is the expected loss at time t with loss function ℓ , f ∗ t ∈ arg min f ∈F R t ( f ) represents the corresponding optimal model in the hypothesis space at each round.

When the distribution of unlabeled data stream is fixed, e.g., D t = D 1 , ∀ t ∈ [ T ] , this formulation recover back to the previous setting of TTA to a stationary environment [30, 32]

Remark 1 (Dynamic Regret and Catastrophic Forgetting) . The dynamic regret defined in Eqn. (1) quantifies the difference between the performance of the learned model and the optimal model at each time step. In our formulation, the data distribution at any given time step can correspond to any previously encountered distribution, and the model has no prior knowledge of this distribution during prediction. Therefore, if the performance of the model remains comparable to that of the optimal model that minimizes the expected loss at that time step, this indicates that the learned model successfully retains previously acquired knowledge and mitigates the catastrophic forgetting issue.

3.2 Representation Alignment with Source Sketch

To adapt the model to a new target domain, a natural intuition is that if the data in the target domain can be accurately projected back to their representation in the source domain, the well-trained source model can be reused for predictions. Based on this idea, we keep the classification module fixed by setting w t = w 0 and adapt the representation learning module { ̂ ϕ t } T t =1 to ensure that the representation of the non-stationary unlabeled data stream aligns closely with that of the source data.

We employ dynamic regret as the performance measure for representation learning in non-stationary environments and define the objective as:

where L t ( · , · ) represents the representation discrepancy between ϕ t and ϕ 0 in each round, where ϕ t is the model learned at round t . Here ϕ ∗ t denotes the optimal representation learning function, defined as ϕ ∗ t = arg min ϕ L t ( ϕ, ϕ 0 ) .

We now define the loss function L t ( · , · ) . Given the challenges in accurately estimating representation discrepancy, we propose to use a surrogate loss function to approximate it. Inspired by prior work on representation learning with deep neural networks [37, 28], we model the representation distribution using a Gaussian approximation. Specifically, we measure the discrepancy as the gap in mean and covariance between the projected unlabeled data and the source data representation, defined as follows:

where λ is the hyperparameter, µ t = E x ∼D t [ ϕ t ( x t )] , Σ t = E x ∼D t [( ϕ t ( x ) -µ t ) T ( ϕ t ( x ) -µ t )] , ∥ · ∥ 2 denotes the Euclidean norm, and ∥ · ∥ F denotes the Frobenius norm. To improve numerical stability, we also add the identity matrix multiplied with a small constant to each covariance matrices Σ t to reduce their condition number.

Since we only have empirical data at each round, we use the empirical loss to approximate the discrepancy. We define ̂ µ t = ∑ n t i =1 ϕ t ( x i ) /n t and Σ t = ∑ n t i =1 [( ϕ t ( x ) -̂ µ t ) T ( ϕ t ( x ) -̂ µ t )] /n t , thus we have the empirical estimation of the divergence,

At first glance, it may seem straightforward to minimize the loss defined in Eqn. (3) to update the representation learning module. However, the limited amount of data available in each round poses a challenge to obtain a well-generalized model. Therefore, we leverage online learning techniques to reuse a suitable number of historical data, ensuring effective learning of the representation model.

Remark 2 (Comparison with Distribution Alignment Approaches) . Aligning the representations of unlabeled data with those of source data has been explored in works such as [23, 28]. These studies primarily focus on adapting the model to a fixed domain with a large amount of unlabeled data. Our problem involves adaptation to non-stationary environments, where the data distribution can change continuously with a limited number of data available in each round. This non-stationary setting requires the development of novel methods to adaptively leverage historical data for adaptation.

The proposed discrepancy measure aligns the global representation distribution between the source data and the new unlabeled data. Additionally, exploring class-specific prototypes can further enhance performance [17, 28], which we leave for future investigation.

…

Base

t2t3

1t4

t6t7

it8

67

it10

Time

Learners

Po, μo, Eo

Po, μo, Eo

B1

Lt

restart

B2

L2

一

restart

L#

1

B3

Meta

restart

K

KL

BK

Change

Change

K

Change

3.3 Adaptive Representation Alignment

In this part, we present the proposed algorithm. Inspired from the online ensemble framework [43] developed in recent research on non-stationary online learning, we propose a two-layer adaptive learning algorithm. This approach is designed to handle the unknown change in data distribution and the limited data available in each round.

Base Learner. We construct a set of base learners { ϕ i } K i =1 , each with a different learning window size. These base learners initialize their parameters as ϕ 0 . As shown in Figure 1, at each round, they perform online gradient descent using the loss defined in Eqn. (3), i.e,

In addition to performing gradient descent, each base learner is assigned a learning window of varying size, determined by its index. As an example, the i -th base learner trains on data segments of length 2 i . At each round t = 2 i , its representation function ϕ i t is re-initialized to the initial state ϕ 0 and a new learning process is initiated using online gradient descent - a procedure we refer to as a 'restart'.

Intuitively, when the data distribution shifts gradually or stabilizes in a new environment, the base learner that leverages the entire historical dataset tends to perform well. In contrast, when the distribution undergoes abrupt changes, a base learner that frequently restarts and relies only on recent data can achieve competitive performance [18]. The flexibility of exploring a set of base learners allows us to design a meta learner that strategically combines these base learners, optimizing the overall performance of the ensemble algorithm.

Meta Learner. As shown in Figure 1, we employ a meta learner to combine the base learners that learn on different time length. To implement the meta learner, we employ the AdaNormalHedge algorithm with the geometric covering scheme [24]. At each round, the meta learner receives a set of loss (in Eqn. (3)) from the base learners and combine them to generate the output for round t . Let p i t be the weight assigned to the i -th base learner at round t . The meta leaner outputs

The weight p i t for each base-learner f i t at round t is updated by first calculating

where Φ( R,C ) = exp([ R ] 2 + / 3 C ) , L t = ∑ i p i t · L i t , and

We set R i t and C i t to 0 when t = 2 i for each restarted base learner.

Algorithm 1 Adaptive Representation Alignment

After updating the representation learning model, we follow previous studies on TTA [32, 39, 25] by employing entropy minimization with a regularization term. This regularization ensures that the updated linear classification model remains close to the initial one, enhancing performance, which is defined as

where [ · ] y is taking the y -th entry of the vector [ · ] . This regularization term encourages the model to generate confident predictions on unlabeled data by assigning higher probabilities to the most likely classes and lower probabilities to less likely ones. As a result, it helps prevent the model from becoming overly uncertain and making unreliable predictions. We then minimize the entropy of the predictions using the updated representation learning model with regularization, that is,

We summarize the main procedures of the proposed algorithm in Algorithm 1.

Remark 3 (Computational Efficiency) . Since the i -th base learner is restarted every 2 i -th round, within a time interval of size T , we only need to maintain at most log( T ) base learners. For example, with T = 100 , 000 , only 11 base learners are required. In the next section, we will demonstrate that this is sufficient to achieve the optimal dynamic regret for convex losses and models.

Note that the ensemble structure requires maintaining multiple base learners, to further enhance the computational efficiency, we follow the spirit of [42] to reduce the base learners' update complexity, which involves updating only the affine parameters of the normalization layers similar to those used in previous studies [21, 32, 28, 25]. The affine parameters typically comprise less than 1% of the total model parameters [32], making them particularly efficient to update.

3.4 Theoretical Analysis

In this part, we provide the theoretical analysis of our proposed Ada-ReAlign algorithm for convex losses and models. For convex representation learning models, such as input convex neural networks [1], and convex loss functions, we show that the proposed algorithm achieves a dynamic regret guarantee, using the optimal representation sequence { ϕ ∗ t } T t =1 as the comparator.

Theorem 1. Assuming ϕ t is convex, L t ( ϕ ) is a convex with respect to ϕ , and the input x t , the value of the loss function, and its gradient are all bounded. The Ada-ReAlign algorithm satisfies

where function variation V T = ∑ T t =2 sup ϕ | L t ( ϕ ) -L t -1 ( ϕ ) | . Detailed proofs are in Appendix C.1 .

Theorem 1 demonstrates that the average regret decreases at a rate of O ( T -1 / 3 ) . In this theorem, the function variation V T captures the cumulative change in the optimal representation function sequence,

serving as a measure of the underlying distribution shift in non-stationary environments and reflecting the inherent difficulty of the learning task. When the unlabeled test data stream is generated from a relatively stable environment, indicated by a small V T , the accumulative loss decreases nearly at a rate of O ( T 2 / 3 ) . We notice that directly optimizing Eqn. (3) in each round would result in O ( T ) regret, as a generalization error would accumulate in every round. Thus, Theorem 1 provides the theoretical foundation for the empirical success of the Ada-ReAlign algorithm in effectively adapting to unknown and continuous distribution shifts.

4 Experiments

We evaluate the proposed Ada-ReAlign algorithm on two large-scale benchmark datasets: CIFAR10C and ImageNetC. Our empirical studies aim to answer the following three questions: Q1 : Does the Ada-ReAlign algorithm outperform competing methods? Q2 : Are the mechanisms in the proposed algorithm effective in handling non-stationary data? Q3 : Can the proposed method handle real-world data streams with unknown distribution shifts?

4.1 Experimental Setups

Data Stream Generation. The CIFAR10C and ImageNetC datasets provide both original clean data and corrupted data with varying types and levels of severity. We train the offline model on the clean data and use the corrupted data to generate unlabeled data streams, allowing us to simulate diverse distribution shifts within the data stream.

We assume a small batch of data is obtained at each round t , where t ∈ [ T ] := { 1 , ..., T } . In each round, this batch is generated from a fixed data distribution with a specific corruption type and severity level. By continuously varying the corruption types or severity levels, we simulate unlabeled data streams across different non-stationary environments. Let N t denote the number of data points in round t and M represent the duration for which the data distribution remains unchanged, spanning M rounds between distribution shifts.

In our empirical studies, we simulate two common types of distribution shifts:

Gradual Shift : To simulate an unlabeled data stream with a gradual shift, we keep the type of data corruption constant throughout the stream while varying the severity level every M rounds. For example, in an unlabeled stream representing the 'Snow' condition, the severity levels change sequentially as follows: [1] ∗ M → [2] ∗ M → [3] ∗ M → ... . Here, [1] ∗ M denotes M consecutive rounds of data under the 'Snow' condition with a severity level of 1 .

Sequential Shift : To simulate an unlabeled data stream with a sequential shift, we keep the corruption severity level constant throughout the stream while changing the type of data corruption every M rounds. For instance, with a fixed severity level of 5, the corruption types evolve sequentially as follows: [ gaussian ] ∗ M → [ shot ] ∗ M → [ impulse ] ∗ M → ... . Here, [ shot ] ∗ M represents M consecutive rounds of data with 'shot' at severity level 5 . In this paper, we set the severity level of the sequentially shifting data stream to 5 .

Contenders. We compare the proposed algorithm with six competing methods. First, we use the performance of the initial Non-adapt model as a baseline. Next, we include methods that restart the model at each round and adapt based on the current round's data: TENT-RE [32], which minimizes entropy at each round, and MEMO [39], which enhances robustness through data augmentation. We also evaluate methods that leverage all the data, such as TENT-CT , which updates the model using the results from the previous round via the TENT [32] and TTAC [28], which focuses on aligning the representation with the initial model using the entire data stream. Additionally, we include two state-of-the-art TTA methods designed for non-stationary environments: CoTTA [34] and SAR [25], both of which incorporate a reset mechanism to mitigate long-term forgetting during adaptation.

Implementation Details. We conduct experiments using a deep neural network with a ResNet50 architecture from the torchvision library. The initial model is trained on the original CIFAR-10 and ImageNet datasets. For our proposed algorithm, we use SGD as the update rule, with a momentum of 0 . 9 and a learning rate of 0 . 0005 . Following prior studies [21, 32, 28, 25], we adapt the affine parameters of the normalization layers in ResNet50 during the adaptation process. Further details are provided in Appendix A.1 .

4.2 Performance Comparison

We present the comparison results for the CIFAR10-to-CIFAR10C datasets in Table 1 and Table 2. Results for the ImageNet-to-ImageNetC datasets are provided in Appendix A.2 , where we observe similar trends. Our proposed Ada-ReAlign algorithm consistently achieves top performance under both gradual and sequential shifts, particularly in sequential shift scenarios, highlighting its adaptability in diverse non-stationary environments.

In Table 1, we report the average classification error rate, along with the standard deviation, for gradual severity shifts across 15 types of distribution shifts. These results are based on batches of N t = 10 data points per round, with a distribution duration of M = 10 rounds, averaged over five runs with different initial models. Overall, our algorithm demonstrates competitive performance compared to other methods. We also observe performance improvements from TENT-CT and TTAC, which reuse all historical data, underscoring the importance of effectively leveraging past data. Meanwhile, Ada-ReAlign outperforms CoTTA and SAR, both of which incorporate reset mechanisms to enhance adaptability to severity shifts.

In Table 2, we evaluate the algorithm across the entire data stream and report its average accuracy for each type of data corruption. By effectively reusing historical data, our proposed Ada-ReAlign algorithm achieves a lower average classification error across all types of distribution shifts compared to the competitors. Notably, MEMO and TENT-RE, which reuse only the current batch of data, outperform TENT-CT, which utilizes all historical data. These findings align with our theoretical analysis, confirming that a fixed adaptation model is not well-suited for non-stationary environments.

Additionally, CoTTA, which stochastically resets parts of the model parameters to their initial values, also underperforms in non-stationary settings. This further highlights the superiority of our adaptive representation alignment algorithm. These results answer the question Q1 .

0

1

2

3

4

5

6

7

8

9 10 11 12 13 14

Interval length (|

I

i

| = 2

i

)

16

32

64

128

256

512

Distribution Change Interval (

M

)

8.2

8.2

9.5 10.719.0 8.2

5.7

3.3

3.1

4.7

4.9

4.6

3.4

2.8

4.0

3.0

7.3

7.8

8.9 16.323.510.1 5.7

3.1

4.0

2.9

0.9

0.8

3.8

1.8

2.7

7.2

7.6

8.2

4.6 17.622.1 1.9

0.6

0.1

6.3

8.1

2.6

1.9

8.7

1.4

6.6

6.7

4.9

3.2

8.4 15.421.5 6.4

4.5

4.5

4.5

4.2

4.3

3.5

3.1

5.2

4.3

4.5

5.7

6.2 10.115.222.0 4.3

4.2

4.3

3.1

4.7

3.2

2.1

4.9

5.2

5.2

5.4

5.9

9.7

9.7 14.321.9 5.0

2.4

3.1

2.9

2.3

5

10

15

20

(a) Sequential shift

0

1

2

3

4

5

6

7

8

9 10 11 12 13 14

Interval length (|

I

i

| = 2

i

)

10

50

100

200

400

800

Distribution Change Interval (

M

)

4.3

6.8 11.824.514.0 7.9

4.5

1.4

7.4

4.2

2.1

3.5

4.5

1.2

1.6

6.2

5.0

7.1

8.7 15.820.114.6 5.4

5.4

2.7

1.3

2.6

1.3

3.3

0.4

4.0

1.9

9.0

3.7

2.0 15.918.019.3 9.0

2.7

1.4

6.0

2.8

1.2

3.2

1.6

4.9

4.1

5.2

7.4

7.8 10.622.716.0 3.1

3.2

2.8

2.3

3.3

4.9

4.5

2.7

3.4

4.2

4.5

7.8

7.6 13.117.711.9 3.3

5.2

4.3

5.1

4.7

2.9

2.6

0.7

2.9

6.7

8.8

8.6

8.8 11.415.6 9.0

4.7

4.9

6.9

5.6

5

10

15

20

4.3 Ablation Study

Next, we evaluate the adaptability of the Ada-ReAlign algorithm to changing distributions, along with its computational efficiency and the effectiveness of each component. Additional ablation studies on the impact of the number of data each round N t and duration M are provided in Appendix A.3 .

Detailed Analysis of the Meta-Base Structure in Ada-ReAlign. We now take a closer look at our adaptive representation alignment algorithm, which integrates a meta-base structure. Specifically, we conduct experiments to evaluate the algorithm's ability to handle sequential shifts, where the distribution of online data alternates between two distributions every M rounds. To explore this, we vary the duration M and report the weight assignment (expressed as percentages) for base learners with different interval lengths in Figure 2. For instance, when M = 8 , the data distribution shifts every 8 rounds. The weights assigned to each base learner are averaged over its active period.

Our results show that the meta learner effectively assigns the highest weight to the base learner whose interval length aligns with the switch period M . This result shows that the right amount of historical data is reused, leading to strong performance in non-stationary environments.

In addition, in Figure 3 (a) compares the average error of the Ada-ReAlign algorithm with that of the TTAC algorithm over 200 iterations. During the experiment, the data distribution shifts three times, transitioning from Noise 'Gaussian' to Blur 'defocus', then to Weather 'Snow', and finally to Digital 'contrast'. Each distribution is maintained for 50 iterations. TTAC is chosen for comparison since it also aims to align the model's representation with the initial one.

The results show that the Ada-ReAlign algorithm adapts quickly to new distributions as soon as a shift occurs. In contrast, the TTAC algorithm struggles, as it relies on all historical data, including samples from different distributions, which limits its adaptability.

Efficiency and Performance Evaluation. We conduct additional experiments to evaluate the algorithm's efficiency and performance gain. Specifically, we measure the processing time per round and the average accuracy, comparing the proposed algorithm with SAR (single model with forward and backward procedures) and AdaNPC (single forward procedure only) [40]. Since AdaNPC requires access to the labeled source data during adaptation, we exclude it from the main comparison.

As shown in Figure 3 (b), the Ada-ReAlign algorithm demonstrates superior performance in both gradual and sequential shifts on the CIFAR10-to-CIFAR10C dataset. In terms of computational time, Ada-ReAlign (with 11 base learners) is approximately five times slower than SAR. However, SAR involves solving a bi-level optimization problem during adaptation, which incurs additional computational overhead. AdaNPC is more efficient than both gradient-based algorithms, as it employs a KNN-based non-parametric classifier. These findings answer the question Q2 .

Component Analysis of the Proposed Algorithm. We evaluate the impact of each component within the proposed algorithm in the sequential shift environment. The proposed algorithm consists of two elements: alignment of the representation divergence with the initial model and minimization of prediction entropy at each round. To investigate their individual contributions, we run the algorithm across the entire data stream using different loss configurations. Specifically, 'Ada-ReAlign w/o DA' refers to the version where only prediction entropy is minimized at each step (without Distribution Alignment), while 'Ada-ReAlign w/o EM' denotes the version where only the representation is aligned at each round (without Entropy Minimization).

50

100

150

200

Time

0.1

0.2

0.3

0.4

0.5

0.6

Average Error (%)

0

0.5

1

1.5

2

Processing Time per Round (sec)

10

12

14

16

18

20

22

24

26

28

SAR

Ada-ReAlign

SAR

Ada-ReAlign

(b) Algorithm efficiency

Gauss

Shot

Impul.

Defoc.

Motio

Zoom

Snow

Frost

Fog

Brigh.

Contr.

Elast.

0

10

20

30

40

50

60

Classification Error

Non-Adapt

Ada-ReAlign w/o DA

Ada-ReAlign w/o EM

Ada-ReAlign (ours)

Gauss

Shot

Impul.

Defoc.

Motio

Zoom

Snow

Frost

Fog

Brigh.

Contr.

Elast.

0

10

20

30

40

50

Classification Error

Ada-ReAlign RE

Ada-ReAlign CT

Ada-ReAlign TS

Ada-ReAlign (ours)

We report the average accuracy for each type of data corruption. As shown in Figure 4, both representation alignment and entropy minimization play crucial roles in the performance of the Ada-ReAlign algorithm. Moreover, the results highlight that, in most cases of distribution shifts, representation alignment offers greater performance gains, underscoring the importance of adaptive representation alignment in non-stationary environments.

Comparison with Restart Mechanisms. We compare the proposed algorithm in a sequential shift environment with three different restart mechanisms: Ada-RE , Ada-CT , and Ada-TS . In Ada-RE , the model is reset to the initial offline model at the beginning of each round and undergoes 'one-step' TTA using the surrogate loss defined in Eqn (3). This approach is similar to the MEMO and TENT-RE algorithms, which restart the model at each round.

The second method, Ada-CT , updates a single model continuously, where the adapted model from the previous round serves as the initial model for the next. The third approach, Ada-TS , also employs a single model but incorporates a restart mechanism based on a threshold for model entropy. Following the previous study [25], we restart the model whenever the entropy falls below a threshold of 0 . 4 .

The average accuracy of our proposed method and the three competing approaches is presented in Figure 5. The Ada-ReAlign algorithm consistently outperforms the competitors across nearly all datasets. These results highlight the critical role of the meta learner in the online ensemble structure, which enables the adaptive combination of base learners to resist the non-stationarity.

4.4 Real-World Evaluation on Wildlife Species Classification

We further evaluate the proposed algorithm on a real-world wildlife species classification task using the iWildCam dataset [3], where the distribution of images naturally varies with the time and location of capture. The earliest 10% of the data is used as labeled offline data to train the initial model, while the remaining data serves as the unlabeled data stream. We compare the performance of our method against competing approaches, and the results demonstrate that our algorithm achieves the best performance. These results answer question Q3 .

5 Conclusion

In this paper, we explored non-stationary representation learning for continual test-time adaptation. Beyond entropy minimization with regularization, we proposed adaptively aligning the unlabeled data stream, with its evolving distributions, to the source data representation by leveraging a sketch of the source data. To exploit this marginal information, we introduced a novel two-layer algorithm, Ada-ReAlign, designed to track and to approximate the optimal representation learning model at each round. Our theoretical analysis showed that the learned model is comparable to the optimal model sequence under convexity assumptions. Experiments on various benchmark datasets and a real-world application demonstrated the superiority of our approach over competing methods, confirming the effectiveness of the adaptive representation alignment mechanism.

Acknowledgments

MS was supported by the Institute for AI and Beyond, UTokyo.

References

Brandon Amos, Lei Xu, and J Zico Kolter. Input convex neural networks. In Proceedings of the 34th International Conference on Machine Learning (ICML) , pages 146-155, 2017.

Yong Bai, Yu-Jie Zhang, Peng Zhao, Masashi Sugiyama, and Zhi-Hua Zhou. Adapting to online label shift with provable guarantees. Advances in Neural Information Processing Systems (NeurIPS) , 35:29960-29974, 2022.

Sara Beery, Arushi Agarwal, Elijah Cole, and Vighnesh Birodkar. The iwildcam 2021 competition dataset. arXiv preprint arXiv:2105.03494 , 2021.

Omar Besbes, Yonatan Gur, and Assaf Zeevi. Non-stationary stochastic optimization. Operations Research , 63(5):1227-1244, 2015.

Dian Chen, Dequan Wang, Trevor Darrell, and Sayna Ebrahimi. Contrastive test-time adaptation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , pages 295-305, 2022.

Yining Chen, Haipeng Luo, Tengyu Ma, and Chicheng Zhang. Active online learning with hidden shifting domains. In Proceedings of the 24th International Conference on Artificial Intelligence and Statistics (AISTATS) , pages 2053-2061, 2021.

Chao-Kai Chiang, Tianbao Yang, Chia-Jung Lee, Mehrdad Mahdavi, Chi-Jen Lu, Rong Jin, and Shenghuo Zhu. Online optimization with gradual variations. In Proceedings of the 25th Conference on Learning Theory (COLT) , pages 1-20, 2012.

Francesco Croce, Maksym Andriushchenko, Vikash Sehwag, Edoardo Debenedetti, Nicolas Flammarion, Mung Chiang, Prateek Mittal, and Matthias Hein. Robustbench: a standardized adversarial robustness benchmark. Advances in Neural Information Processing Systems (NeurIPS) Datasets and Benchmarks Track , 34, 2021.

Yulu Gan, Yan Bai, Yihang Lou, Xianzheng Ma, Renrui Zhang, Nian Shi, and Lin Luo. Decorate the newcomers: Visual domain prompt for continual test time adaptation. In Proceedings of the AAAI Conference on Artificial Intelligence (AAAI) , volume 37, pages 7595-7603, 2023.

Jin Gao, Jialing Zhang, Xihui Liu, Trevor Darrell, Evan Shelhamer, and Dequan Wang. Back to the source: Diffusion-driven adaptation to test-time corruption. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , pages 11786-11796, 2023.

Elad Hazan. Introduction to Online Convex Optimization. Foundations and Trends in Optimization , 2(3-4):157-325, 2016.

Dan Hendrycks and Thomas Dietterich. Benchmarking neural network robustness to common corruptions and perturbations. In Proceedings of the 7th International Conference on Learning Representations (ICLR) , 2019.

Dan Hendrycks, Norman Mu, Ekin Dogus Cubuk, Barret Zoph, Justin Gilmer, and Balaji Lakshminarayanan. Augmix: A simple data processing method to improve robustness and uncertainty. In Proceedings of the 8th International Conference on Learning Representations (ICLR) , 2020.

Judy Hoffman, Trevor Darrell, and Kate Saenko. Continuous manifold based adaptation for evolving visual domains. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , pages 867-874, 2014.

Xuefeng Hu, Gokhan Uzunbas, Sirius Chen, Rui Wang, Ashish Shah, Ram Nevatia, and SerNam Lim. Mixnorm: Test-time adaptation through online normalization estimation. arXiv preprint arXiv:2110.11478 , 2021.

Jonathan Huggins, Trevor Campbell, and Tamara Broderick. Coresets for scalable bayesian logistic regression. Advances in Neural Information Processing Systems (NIPS) , 29:4080-4088, 2016.

Yusuke Iwasawa and Yutaka Matsuo. Test-time classifier adjustment module for model-agnostic domain generalization. Advances in Neural Information Processing Systems (NeurIPS) , 34:24272440, 2021.

Bartosz Krawczyk, Leandro L Minku, Joao Gama, Jerzy Stefanowski, and Michal Wozniak. Ensemble learning for data stream analysis: A survey. Information Fusion , 37:132-156, 2017.

Ananya Kumar, Tengyu Ma, and Percy Liang. Understanding self-training for gradual domain adaptation. In Proceedings of the 37th International Conference on Machine Learning (ICML) , pages 5468-5479, 2020.

Daeun Lee, Jaehong Yoon, and Sung Ju Hwang. Becotta: Input-dependent online blending of experts for continual test-time adaptation. In Proceedings of the 41st International Conference on Machine Learning (ICML) , pages 27072-27093, 2024.

Jian Liang, Dapeng Hu, and Jiashi Feng. Do we really need to access the source data? source hypothesis transfer for unsupervised domain adaptation. In Proceedings of the 37th International Conference on Machine Learning (ICML) , pages 6028-6039, 2020.

Hyesu Lim, Byeonggeun Kim, Jaegul Choo, and Sungha Choi. TTN: A domain-shift aware batch normalization in test-time adaptation. In Proceedings of the 11th International Conference on Learning Representations (ICLR) , 2023.

Yuejiang Liu, Parth Kothari, Bastien Van Delft, Baptiste Bellot-Gurlet, Taylor Mordan, and Alexandre Alahi. TTT++: When does self-supervised test-time training fail or thrive? Advances in Neural Information Processing Systems (NeurIPS) , 34:21808-21820, 2021.

Haipeng Luo and Robert E Schapire. Achieving all with no parameters: Adanormalhedge. In Proceedings of the 28th Conference on Learning Theory (COLT) , pages 1286-1304, 2015.

Shuaicheng Niu, Jiaxiang Wu, Yifan Zhang, Zhiquan Wen, Yaofo Chen, Peilin Zhao, and Mingkui Tan. Towards stable test-time adaptation in dynamic wild world. In Proceedings of the 11th International Conference on Learning Representations (ICLR) , 2023.

Steffen Schneider, Evgenia Rusak, Luisa Eck, Oliver Bringmann, Wieland Brendel, and Matthias Bethge. Improving robustness against common corruptions by covariate shift adaptation. Advances in Neural Information Processing Systems (NeurIPS) , 33:11539-11551, 2020.

Junha Song, Jungsoo Lee, In So Kweon, and Sungha Choi. Ecotta: Memory-efficient continual test-time adaptation via self-distilled regularization. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , pages 11920-11929, 2023.

Yongyi Su, Xun Xu, and Kui Jia. Revisiting realistic test-time training: Sequential inference and adaptation by anchored clustering. Advances in Neural Information Processing Systems (NeurIPS) , pages 17543-17555, 2022.

Masashi Sugiyama and Motoaki Kawanabe. Machine Learning in Non-Stationary Environments - Introduction to Covariate Shift Adaptation . MIT Press, 2012.

Yu Sun, Xiaolong Wang, Zhuang Liu, John Miller, Alexei Efros, and Moritz Hardt. Test-time training with self-supervision for generalization under distribution shifts. In Proceedings of the 37th International Conference on Machine Learning (ICML) , pages 9229-9248, 2020.

Ivor W Tsang, James T Kwok, Pak-Ming Cheung, and Nello Cristianini. Core vector machines: Fast svm training on very large data sets. Journal of Machine Learning Research , 6(4):363-392, 2005.

Dequan Wang, Evan Shelhamer, Shaoteng Liu, Bruno Olshausen, and Trevor Darrell. Tent: Fully test-time adaptation by entropy minimization. In Proceedings of the 9th International Conference on Learning Representations (ICLR) , 2021.

Haoxiang Wang, Bo Li, and Han Zhao. Understanding gradual domain adaptation: Improved analysis, optimal path and beyond. In Proceedings of the 39th International Conference on Machine Learning (ICML) , pages 22784-22801, 2022.

Qin Wang, Olga Fink, Luc Van Gool, and Dengxin Dai. Continual test-time domain adaptation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , pages 7201-7211, 2022.

Chen-Yu Wei and Haipeng Luo. Non-stationary reinforcement learning without prior knowledge: An optimal black-box approach. In Proceedings of the 34th Conference on Learning Theory (COLT) , pages 4300-4354, 2021.

Colin Wei, Kendrick Shen, Yining Chen, and Tengyu Ma. Theoretical analysis of self-training with deep networks on unlabeled data. In Proceedings of the 9th International Conference on Learning Representations (ICLR) , 2021.

Werner Zellinger, Thomas Grubinger, Edwin Lughofer, Thomas Natschläger, and Susanne Saminger-Platz. Central moment discrepancy (cmd) for domain-invariant representation learning. In Proceedings of the 5th International Conference on Learning Representations (ICLR) , 2017.

Lijun Zhang, Shiyin Lu, and Tianbao Yang. Minimizing dynamic regret and adaptive regret simultaneously. In Proceedings of the 23rd International Conference on Artificial Intelligence and Statistics (AISTATS) , pages 309-319, 2020.

Marvin Zhang, Sergey Levine, and Chelsea Finn. Memo: Test time robustness via adaptation and augmentation. Advances in Neural Information Processing Systems (NeurIPS) , 35:3862938642, 2022.

Yifan Zhang, Xue Wang, Kexin Jin, Kun Yuan, Zhang Zhang, Liang Wang, Rong Jin, and Tieniu Tan. Adanpc: Exploring non-parametric classifier for test-time adaptation. In Proceedings of the 40th International Conference on Machine Learning (ICML) , pages 41647-41676, 2023.

Yu-Jie Zhang, Zhen-Yu Zhang, Peng Zhao, and Masashi Sugiyama. Adapting to continuous covariate shift via online density ratio estimation. Advances in Neural Information Processing Systems (NeurIPS) , 36:29074-29113, 2023.

Peng Zhao, Yan-Feng Xie, Lijun Zhang, and Zhi-Hua Zhou. Efficient methods for non-stationary online learning. In Advances in Neural Information Processing Systems 35 (NeurIPS) , pages 11573-11585, 2022.

Peng Zhao, Yu-Jie Zhang, Lijun Zhang, and Zhi-Hua Zhou. Adaptivity and non-stationarity: Problem-dependent dynamic regret for online convex optimization. Journal of Machine Learning Research , 25(98):1 - 52, 2024.

Shiji Zhou, Han Zhao, Shanghang Zhang, Lianzhe Wang, Heng Chang, Zhi Wang, and Wenwu Zhu. Online continual adaptation with active self-training. In Proceedings of the 25th International Conference on Artificial Intelligence and Statistics (AISTATS) , pages 8852-8883, 2022.

Zhi-Hua Zhou. Open-environment machine learning. National Science Review , 9(8), 07 2022. nwac123.

Martin Zinkevich. Online convex programming and generalized infinitesimal gradient ascent. In Proceedings of the 20th International Conference on Machine Learning (ICML) , pages 928-936, 2003.

Appendix

A Experiments

In this section, we supplement the omitted details in Section 4. We begin by presenting a comprehensive overview of the experimental setups, followed by showcasing the previously omitted empirical results on the ImageNet dataset. Furthermore, we conduct additional ablation studies to further investigate the proposed Ada-ReAlign algorithm.

A.1 Experimental Setups

Contenders. We compare our proposed approach with six contenders, including a baseline method,

Non-adapt : This method directly uses the offline model to make predictions on the unlabeled data stream without any adaptation.

One modified 'one-round' TTA method is employed, where the model parameters are reset to the offline model at the start of each round and then undergo one-step adaptation:

MEMO [39]: This method applies various data augmentations to each input and adapts all model parameters by minimizing the entropy of the average output distribution across these augmentations. At the beginning of each round, the model parameters are reset to those of the offline model.

Two continuous TTA methods are also evaluated, where the model is updated throughout the entire data stream without resets:

TENT [32]: This method optimizes the channel-wise affine transformations within the normalization layers by minimizing the entropy of the model's predictions.

TTAC [28]: This approach introduces a test-time anchored clustering method to improve the alignment of test-time features with the offline model's learned representation. It also employs pseudo-label filtering to enhance the effectiveness of the clustering process.

Additionally, we include two state-of-the-art continuous TTA methods specifically designed to address distribution shifts in non-stationary environments:

CoTTA [34]: This algorithm continuously adapts by learning pseudo-labels from its own predictions. It leverages weight-averaged and augmentation-averaged predictions to produce more reliable pseudo-labels for the unlabeled data stream.

SAR [25]: SAR is a sharpness-aware algorithm that minimizes entropy while ensuring robustness by filtering out noisy samples with large gradients. It encourages the model weights to converge to a flat minimum, improving resilience to noisy labeled data.

In summary, these contenders cover a wide range of continuous TTA approaches, from methods that rely on individual batches of data to those that incorporate the entire data stream. Our comparison also includes state-of-the-art algorithms specifically designed to handle distribution shifts in non-stationary environments.

Algorithm Implementation Details of Contenders. For the remaining contenders, we provide their detailed hyperparameter settings used in the experiments as follows.

MEMO [39]: We adopted the default hyper-parameter settings used in MEMO. Specifically, we employed AugMix [13] for data augmentations, with an augmentation size of 32 . The optimizer is SGD with a momentum of 0 . 9 , a batch size of 32 , and a learning rate of 0 . 0005 .

TENT-RE [32]: We adopted the default hyper-parameter settings used in TENT. The optimizer is SGD with a momentum of 0 . 9 , a batch size of 32 , and a learning rate of 0 . 0005 . The trainable parameters are the affine parameters within the batch normalization layers. At the start of each round, the model parameters are reset to match those of the offline model.

TENT-CT [32]: We adopted the default hyper-parameter settings used in TENT. The optimizer is SGD with a momentum of 0 . 9 , a batch size of 32 , and a learning rate of 0 . 0005 . The trainable parameters are the affine parameters within the batch normalization layers.

TTAC [28]: We adopted the default hyper-parameter settings used in TTAC. The optimizer is SGD with a momentum of 0 . 9 , a batch size of 32 , and a learning rate of 0 . 0005 . The trainable parameters are the affine parameters within the batch normalization layers.

CoTTA [34]: We adopted the default hyper-parameter settings used in CoTTA. The optimizer is Adam with a learning rate of 0 . 001 . We use the default data augmentation techniques in the CoTTA algorithm. We updated all trainable parameters while incorporating a restoration probability of p = 0 . 01 to mitigate the risk of catastrophic forgetting.

SAR [25]: We adopted the default hyper-parameter settings used in SAR. The optimizer is SGD with a momentum of 0 . 9 , a batch size of 32 , and a learning rate of 0 . 0005 . To clip the large gradients, we applied a threshold set to 0 . 4 × ln 1000 . Additionally, we set ρ to 0 . 05 to encourage the optimization process to converge towards flat minima, following the default settings. For model recovery, we tracked the entropy loss values using a moving average factor of 0 . 9 . The trainable parameters are the affine parameters within the batch normalization layers.

Algorithm Implementation Details. We adopt the standard ResNet-50 model from RobustBench [8] as the model structure. We freeze the top two linear layers and only update the affine parameters of the normalization layers within the remaining shallow layers of ResNet-50. The optimizer employed is SGD, with a momentum of 0 . 9 and a batch size of 32 for rounds where the number of data data exceeds 32 . In cases where the number of data in a round is smaller than 32 , we employ data augmentation techniques [39] to augment the data and ensure its number reaches a minimum of 32 . The learning rate is set to 0 . 0005 . We use the following compute resource configuration: 2 Xeon Gold 6242R with a base frequency of 3.1 GHz, 8 GeForce 3090 with 24GB VRAM, and a total of 768GB RAM. The operating system employed is Ubuntu 20.04.

A.2 More Empirical Results on ImageNet-C for Section 4.2

Table 4 and Table 5 present the omitted results from Section 4.2. The results show that Ada-ReAlign outperforms the competing methods across nearly all tasks in the ImageNet-C dataset. These empirical findings confirm the effectiveness of our approach in adapting to non-stationary environments.

A.3 Additional Ablation Studies

The number of data points in each round is a crucial measure of non-stationarity in the environment. As data accumulate continuously, and assuming that the data received in each round are generated from the same distribution, the number of data points per round serves as an indicator of the speed of distribution shifts. For a fixed duration M , a smaller number of data points per round suggests

rapid distribution shifts, while a larger number indicates slower shifts, with a prolonged period during which data are generated from a consistent distribution.

Table 6 presents the performance of our Ada-ReAlign algorithm with varying numbers of data per round. Notably, even with limited data per round, the algorithm maintains relatively strong performance.

We further evaluate the algorithm's performance across different values of M and N , where M represents the length of time during which the data distribution remains constant, and N is the number of data points available at each time step. Throughout the experiments, we maintain a batch size of 32. When N ≤ 32 , the batch size is adjusted to N . A smaller M indicates more frequent distribution shifts within the unlabeled data stream.

From Tables 7 and 8, we observe that the Ada-ReAlign algorithm consistently outperforms its closest competitor, the SAR algorithm, across various values of M and N , especially in scenarios involving sequential shifts. Both SAR and Ada-ReAlign experience performance degradation as distribution shifts become more frequent and fewer data points are available per time step. However, Ada-ReAlign exhibits more modest degradation compared to SAR, demonstrating greater robustness in the face of rapid distribution shifts and limited data availability at each step.

B Related Work

In this section, we introduce the relevant literature on test-time adaptation and learning algorithms designed for non-stationary environments.

TTA in Stationary Environment. TTA considers the problem in which the source labeled data are no longer accessible during the adaptation phase. In this task, the learner relies solely on the source model and seeks to adapt it to a fixed test dataset with a distinct distribution. TTA methods can be broadly categorized into two groups based on whether they require a specific training process.

Test-Time Training [30] is the first group, involves optimizing the initial model with a combination of supervised and self-supervised losses, followed by self-supervised learning during test time to update the model. Common self-supervised losses include rotation prediction [30], contrastive self-supervised learning [23], and others.

The second group of methods does not require a specific training process and can be directly applied to any neural network. These methods include adapting batch normalization statistics [26, 15, 22], entropy minimization [32, 25, 39], pseudo-labeling [34, 5], and more.

However, these approaches are designed for adaptation to a fixed distribution and are not well-suited for non-stationary environments, where the data distribution evolves over time.

Continual TTA in Non-stationary Environments. In recent years, TTA in non-stationary environments has gained significant attention. As streaming data evolve over time, the underlying data distribution naturally shifts. A straightforward approach involves resetting the model's parameters to those of the initial model after each mini-batch adaptation, as seen in methods like MEMO [39], episodic TENT [32], and DDA [10]. However, these one-step adaptation methods often fall short due to high variance from the limited data available per round and their inability to leverage accumulated historical data for improved performance.

To address these challenges, CoTTA [34] was introduced, employing robust pseudo-labels generated through a weighted average of historical models. CoTTA also preserves the initial model's parameters by stochastically replacing the model's parameters with those of the initial model at each round. Similarly, SAR [25] estimates reliable entropy and resets the model to its initial state when the entropy exceeds a predefined threshold. AdaNPC [40] tackles non-stationary shifts by constructing a memory to store historical distributions. Additionally, continual test-time adaptation techniques have been extended to large language models [9].

While these approaches have demonstrated empirical success in various real-world tasks, the challenge of effective representation alignment in non-stationary environments remains underexplored.

Adapting to Non-stationary Environments with Offline Labeled Data. Online convex optimization [11] provides a powerful paradigm to handle sequential prediction problems. Over the decades, a variety of online learning algorithms have been proposed to handle the changing environments by optimizing the dynamic regret measure [46, 4, 43, 35] under different kinds of feedback information and

non-stationarity measures of environments. When data is weakly labeled, handling non-stationarity becomes extremely challenging as it is hard to estimate the loss at each round.

This line of research then consider certain kinds of distribution shift and adaptive learning with weighted source labeled data or test unlabeled data with pseudo labels to adapt to continuous distribution change. Previous work tackled the challenge of non-stationarity in the context of online label shift, where only the class prior is changing. In such scenarios, an unbiased loss estimator is constructed to estimate the loss at each round, enabling the use of dynamic regret minimization to handle non-stationary environments [2]. Another work investigated the problem of continuous covariate shift where only the input distribution changes [41]. They reframed this problem as an online density ratio estimation task and proposed a generic reduction of the density ratio estimation problem to dynamic regret optimization. However, these methods may not always be feasible to access offline labeled data for online adaptation.

In contrast to label shift or covariate shift, gradual domain adaptation consider the gradual shift of underlying distribution. An early work in this area is continuous manifold adaptation [14], which considers adaptation to evolving domains. More recently, self-training has shown promising results by pseudo labeling the unlabeled data stream and adaptively re-train the model based on the pseudo labeled data [19]. Theoretical analysis demonstrated that when the optimal classifier only undergoes slight shifts between consecutive batches, self-training can provide a well-generalized classifier throughout the entire data stream [36, 33]. However, these approaches assume that the data stream satisfies this assumption, and may not handle outliers that deviate from it. To handle such outliers, several approaches employ active learning [6, 44] to alleviate their negative impact. Nevertheless, these methods are designed for specific types of data distribution change problems.

C Theoretical Analysis

In this section, we present the theoretical analysis of the guarantees for our proposed algorithm. The analysis is based on the convex optimization framework, assuming both the model and the loss function are convex. While our method does not strictly fit within the convex optimization framework, the analysis offers valuable insights that inform the design of an effective algorithm. Furthermore, the theoretically guided approach demonstrates strong empirical performance [44, 41].

We first introduce the following assumptions for our analysis.

Assumption 1. The gradients of all functions are bounded by G , i.e.,

Assumption 2. The domain Φ contains the origin 0 , and its diameter is bounded by D , i.e.,

Assumption 3. The value of each function belongs to [0 , 1] , i.e.

As long as the loss functions are bounded, they can always be scaled and restricted to [0 , 1] .

C.1 Proof of Theorem 1

In this part, we provide the detailed proof for Theorem 1.

Proof. As we use the information each round the update the model, this operation implicitily is an optimistic mirror descent where the optimism is the gradient of the empirical loss at that time. We firstly show that these two framework are equal, so that we can analyze our proposed algorithm with the optimistic mirror descent framework. The optimistic mirror descent works as follows, for t = 1 , ..., T ,

where f t ( · ) is the loss function each time. Therefore, we prove that our method is equal to optimistic mirror descent by choosing M t = ∇ ̂ L t ( ̂ ϕ t ) . In the following, we use the optimistic mirror descent framework to analyze it.

Now we begin to prove the dynamic regret of the representation learning model. We firstly prove the static regret for the base model. Under the Assumption 1, 2 and 3, according to Lemma 2, we have the following inequality for any comparator u in any interval I ,

For the first term in Eqn. (8), according to we have the following inequality

where the second inequality holds due to Theorem 1 in [43].

For the second term in Eqn. (8), considering our update rule ϕ t = Π( ̂ ϕ t -ηM t ) , where M t = ∇ ̂ L t ( ̂ ϕ t ) , according to the Lemma 1, we have

By choosing η = O ( √ | I | ) , we have

Secondly, as we use the AdaNormalHedge as the meta learner, according to Lemma 3, we can obtain the following strongly adaptive regret

Now we use the guarantee of adaptive regret to obtain the dynamic regret. Let V T = ∑ T t =2 sup ϕ | L t ( ϕ ) -L t -1 ( ϕ ) | is the functional variation, we have

where in the first inequality we introduce a split of the whole time horizon, which can be arbitrary and we only use it in the analysis. In the second inequality, the fisrt term is the adaptive regret used in Lemma 3. For the second term can, we can follow the reasoning in [4] and subsequently simplified analysis in [43, Theorem 7] to show that

Therefore, we have

Impact Statements

This research investigates a general machine learning problem of test-time adaptation, where we consider the continuous distribution shift in the unlabeled data stream. The consequences of system failure and bias in the data are not applicable.

where s m = ( m -1) T/M +1 is the first time step at interval I m . In the above, the first inequality is due to the optimality of ϕ ∗ I m over the interval I m . The second inequality holds since ϕ ∗ s m ∈ arg min ϕ R s m ( ϕ ) .

Combining Eqn. (10) and Eqn. (11), we can obtain

The third inequality is due to the AM-GM inequality. Now we complete the proof.

C.2 Technical Lemmas

This section provides several useful technical lemmas used in the proof. The first three lemmas are the concentration on each time-stamp while the final one is a general inequality.

Lemma 1 (Stability lemma [7, Proposition 7]) . Consider the following two updates: (i) x ∗ = arg min x ∈X ⟨ a , x ⟩ + B ψ ( x , c ) , and (ii) x ′ ∗ = arg min x ∈X ⟨ a ′ , x ⟩ + B ψ ( x , c ) . When the regularizer ψ : X → ❘ is a 1-strongly convex function with respect to the norm ∥ · ∥ , we have ∥ x ∗ -x ′ ∗ ∥ ≤ ∥ ( ∇ ψ ( c ) -a ) -( ∇ ψ ( c ) -a ′ ) ∥ ∗ = ∥ a -a ′ ∥ ∗ .

Lemma 2 (Lemma 2 in [44]) . Under the Assumption 1, 2 and 3, we have the following inequality for u t , t = 1 , ...T ,

Lemma 3 (Theorem 3 in [38]) . Under the Assumption 1, 2 and 3, running AdaNormalHedge as the meta learner, we have the following inequality for any interval I that

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: We summarize the main contributions in the abstract and introduction.

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: We analyze the number of base learner required for the proposed algorithm and its effects in computational efficiency in Remark 3 in Section 3.3.

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [Yes]

Justification: We provide the full set of assumptions and results in Section 3.4 and in Appendix C.

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: All the details of the experiments can be found in A.

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [Yes]

Justification: All the details of the data can be found in A.1. Codes can be found in Supplementary Material.

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/ public/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https: //nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: All the details of the experiments implementation can be found in A.1.

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [Yes]

Justification: All the error bars are reported in the experiments.

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [Yes]

Justification: All the details of the experiments compute resources can be found in A.1.

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: We have reviewed the NeurIPS Code of Ethics.

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [Yes]

Justification: We discuss the broader impacts in the end of the Appendix.

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification: This paper poses no such risks.

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [NA]

Justification: This paper does not use existing assets.

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [NA]

Justification: This paper does not release new assets.

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification: This paper does not involve crowdsourcing nor research with human subjects.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification: This paper does not involve crowdsourcing nor research with human subjects.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a test-time adaptation (TTA) method focusing on adapting to a non-stationary environment.\nThe proposed method DART aims at minimizing the cumulative loss over time steps, which is equivalent to minimize the distribution gap between the source and current features.\nTo capture unknown changing periods of the environment, DART employs multiple base learners that are reset at different periods.\nDART incorporates the outputs of the base learners with dynamically learned weights to make final predictions.\nExperimental results show that DART can follow continual environmental changes on image corruption datasets.\n\nSTRENGTHS:\n- S1: Addressing continual adaptation of non-stational environments is practical.\n- S2: Providing the theoretical insights is beneficial.\n- S3: It is interesting that minimizing the distribution gap results in minimizing the regret.\n\nWEAKNESSES:\n- W1: Equations should be presented accurately.   \n  - The argument of $\\hat{L}_t$ in Eq. (4) seems to be $\\hat{L}\\_t(\\phi\\_t,\\phi\\_0)$.  \n  - Softmax seems to be missing in the formulation of $\\ell\\_\\text{entro}^t$.  \n  - Ada-ReAlign minimizes the norms of the feature statistics between the source and current distributions, but Fig. 1 says that it minimizes KL divergence.\n- W2: Providing more detailed explanations of theoretical analyses in Sec. 3.4 would be beneficial. For example, where do $\\mathcal{O}(T^{-1/3})$ in L221 and $\\mathcal{O}(T)$ in L227 come from?\n- W3: Ada-ReAlign requires to keep $\\log T$ models, which is more than the baseline methods. Comparing memory consumption would be fair.\n- W4: Can we estimate the distribution gap in Eq. (3) accurately? When the batch size is smaller than the number of feature dimensions, the covariance matrix degenerates, i.e., the covariances cannot be estimated as we can take arbitrary subspaces that include all samples in the batch. Even if numerical stability is improved by adding a small constant to the diagonals of the covariance matrix (L153), this problem still remains unless the batch size is larger than the feature dimensions. It would be beneficial to evaluate the distribution gap with other metrics, e.g., optimal transport or MMD, at each round.\n\nQUESTIONS:\n- Q1: How do we determine the number of the base learners when $T$ is unknown?\n- Q2: Can we incorporate the idea of the meta-learner and base learners with other TTA methods?\n\nLIMITATIONS:\nDiscussed in Appendix.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-10"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a novel algorithm called Ada-ReAlign for sequentially adapting a model to non-stationary environments. The proposed algorithm uses a set of base learners, each equipped with different learning window sizes, with a meta learner that combines their outputs. This online ensemble allows an adaptive projection of the unlabeled data, which exhibits changing distributions, onto the source distribution. Subsequently, entropy minimization is applied to the unlabeled data, facilitating the refinement of representations according to the updated distributions. This paper provides theoretical analysis and empirical experiments to justify and validate the effectiveness of the proposed approach.\n\nSTRENGTHS:\n+ This paper is well organized and easy to follow, all the problem formulation and details of the proposed algorithm are well discussed, the experimental setup is clear to me and the result is convincing.\n\n+ The transformation of continuous test-time adaptation to dynamic regret minimization for representation alignment is novel to me. Based on this transformation, the proposed approach is versatile and capable of handling non-stationary environments without explicitly defining or identifying distribution shift boundaries. \n\n+ The proposed approach is theoretically sound. The dynamic exploration of different sliding window sizes of the data stream is novel and intriguing to me. The algorithm is sound with nice theoretical guarantees.\n\n+ Empirical results show superiority over competing algorithms on two simulated benchmark datasets and a wildlife species classification task of the iWildCam dataset. Many ablation studies are performed and the results seem convincing.\n\nWEAKNESSES:\n- The proposed algorithm requires the sketch of the source domain feature embeddings during the adaptation, which may limit the use of the proposed algorithm in practice.\n\n- Although the proposed approach uses only a small number of base learners, $n$ base learners means $n$ times slower than a traditional base learner algorithm, which increases the computational burden.\n\nQUESTIONS:\nShould the number of base learners used depend on the downstream problem? How is the number of base models determined and the size of the sliding window chosen? Large windows adapt to gradual changes over time, while small windows adapt to drastic changes. It seems that the window size does not depend on the specific problem.\n\nThe meta-learner seems rather complicated. Why this particular design?\n\nIn the proposed algorithm, the number of base learners seems to be fixed. What is the performance when the number of experts is limited to a fixed number, less than the required number?\n\nLIMITATIONS:\nPlease see Weaknesses.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-11"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper investigates test-time adaptation in a non-stationary environment where unlabeled data batches arrive sequentially with changing data distributions. The authors propose non-stationary representation learning to project the changing distribution back to the original source data distribution and update the classifier with entropy minimization based on the projected representation. Theoretical analysis shows that the proposed method has solid dynamic regret guarantees. Experiments show that the method of representation adaptation is effective in mitigating the distribution shift.\n\nSTRENGTHS:\n1. The continuous test-time adaptation problem studied in the paper is a popular and crucial area of research for real-world applications of domain adaptation.\n\n2. The proposed non-stationary representation learning is novel and offers non-trivial technical contributions. The authors adapt techniques from online ensemble learning to test-time adaptation, addressing the challenges posed by unknown distribution changes and the problems of high bias and high variance due to the limited amount of unlabeled data available in each round in the data streams.\n\n3. The effectiveness of the method is validated through experiments on both benchmark datasets and a real-world application. In addition, its computational efficiency is well discussed.\n\nWEAKNESSES:\n1. The paper lacks a sufficient review of related work, particularly in the online learning literature. For instance, the method uses dynamic regret as a performance measure and uses techniques from online ensembles. However, readers unfamiliar with this area or these techniques may find it difficult to understand the underlying motivation. \n\n2. The theoretical guarantees focus primarily on convex models and losses, making it hard to adapt them to deep learning models.\n\n3. In the sequential shift, the cyclicality period is fixed by $M$. The performance under conditions of sudden changes or non-cyclic changes is not well studied. Ablation studies are mainly conducted on benchmark datasets, leaving the performance in real-world applications unclear.\n\nQUESTIONS:\n1. How does Theorem 1 apply to deep models, considering that the theory was originally formulated for linear models?\n\n2. In the sequential shift, the cyclicality period is fixed by $M$. How does the performance change when the cyclicality period varies?\n\n3. How stable is the performance of the proposed method on the iWildCam dataset with different hyperparameters? are there any ablation studies?\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-11"}