CONTRAST: Continual Multi-source Adaptation to Dynamic Distributions
Authors: Sk Miraj Ahmed
Fahim Faisal Niloy
Xiangyu Chang
Dripta S. Raychaudhuri
Samet Oymak
Amit Roy-Chowdhury
Venue: NeurIPS 2024 poster (2024)
Keywords: Multi-source Adaptation
Online learning
Test Time Adaptation
Primary area: online_learning
TL;DR: We propose the first work to consider dynamically evolving multi-source adaptation at test time.

Abstract: Adapting to dynamic data distributions is a practical yet challenging task. One effective strategy is to use a model ensemble, which leverages the diverse expertise of different models to transfer knowledge to evolving data distributions. However, this approach faces difficulties when the dynamic test distribution is available only in small batches and without access to the original source data. To address the challenge of adapting to dynamic distributions in such practical settings, we propose continual multi-source adaptation to dynamic distributions (CONTRAST), a novel method that optimally combines multiple source models to adapt to the dynamic test data. CONTRAST has two distinguishing features. First, it efficiently computes the optimal combination weights to combine the source models to adapt to the test data distribution continuously as a function of time. Second, it identifies which of the source model parameters to update so that only the model which is most correlated to the target data is adapted, leaving the less correlated ones untouched; this mitigates the issue of ``forgetting" the source model parameters by focusing only on the source model that exhibits the strongest correlation with the test batch distribution. Through theoretical analysis we show that the proposed method is able to optimally combine the source models and prioritize updates to the model least prone to forgetting. Experimental analysis on diverse datasets demonstrates that the combination of multiple source models does at least as well as the best source (with hindsight knowledge), and performance does not degrade as the test data distribution changes over time (robust to forgetting).

CONTRAST: Continual Multi-source Adaptation to Dynamic Distributions
Sk Miraj Ahmed Fahim Faisal Niloy!.", Xiannyu Chang', Dripta Raychaudhuri
Samet Oymak?, Amit K. hoy-Chowdhury
University of Califomia, Riverside, BBrokkhaven National Laboratory, #AWS AI Labs, *University of Michigan, Ann Arbor
(sahme047@,, fnilo001 @, @x drayc001 @ amitre amitre@ece.. JucreJucr,ddu oymak @ umich.edu
Abstract
Adapting to dynamic data distributions is practical yet challenging task. One effective strategy is use model ensemble, which leverages the diverse expertise different models to transfer knowledge to evolving data distributions. However, this approach faces difficulties when the dynamic test distribution is available only in small batches and without access to the original source data. To address the challenge of adapting to dynamic distributions in such practical settings, propose CONtinual mulTi-souRce Adaptation to dynamic diStribuTions (CONTRAST), novel method that optimally combines multiple source models to adapt the dynamic test data. CONTRAST has two distinguishing features. First, efficiently computes the optimal combination weights to combine the source models to adapt to the test data distribution continuously as a function of time. Second, it identifies which of the source model parameters is most correlated to the target data is adapted, leaving the less correlated ones untouched; this mitigates the issue of "forgetting" the source model parameters by focusing only on the source model that exhibits the strongest correlation with the test batch distribution. Through theoretical analysis we show that the proposed method is able to optimally combine the source models and prioritize updates to the model least prone forgetting. Experimental analysis on diverse datasets demonstrates that the combination of multiple source models does at least as well as the best source (with hindsight knowledge), and performance does not degrade as the test data distribution changes over time (robust to forgetting).
update so that only the model which
Introduction
Deep neural networks have shown impressive performance on test inputs that closely resemble the training distribution. However, their performance degrades significantly when they encounter test. inputs from a different data distribution. Unsupervised domain adaptation (UDA) techniques [1, 2] aim mitigate this performance drop. Addressing the distribution shift in case of dynamic data distributions is even more challenging and practically relevant in many real-world applications like autonomous navigation, models often encounter dynamically evolving distributions. Furthermore, test data is often accessed in streaming batches rather than all at once, and source data may not always be available due to privacy and storage concerns.
For domain adaptation to dynamically evolving environments, employing a model ensemble can be beneficial, as it allows leveraging the learned knowledge of differrnt models to more effectively
"quual contribution; Co-first authors listed alphabetically by last name.
1 Currently Brookhaven National Laboratory. Work done while the author UCR. Cuurently at AWS AI Labs. Work done while the author was at UCR.
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

weather condition (%)
(1)
(3)
(4)
Weather condition
Source Model
weights
update
weights
update
weights
update
update weights
update
Sun
x
x
X
x
x
H
L
X
Snow
L
H
Percentage sunny
Percentoge rainy
Percentage snowy
H/M/L High/MMdlummLLow
Figure 1: Problem setup. Consider several source models trained using data from different weather conditions. During the deployment these models, they may encounter varying weather conditions that could be a combination of multiple conditions in varying proportions (represented by the pie charts on top). Our goal is to infer on the test data using the ensemble of models by automatically figuring out proper combination weights and adapting the appropriate models the fly.
mitigate dynamic distribution shifts. Additionally, situations may arise wherein the user has access to a diverse set of pre-trained models across distinct source domains, and no access to source domain data corresponding to each model due to privacy, storage other constraints. Consequently, training a unified model using the combined source data becomes unfeasible. In those scenarios, it both reasonable and effective employ and adapt the entire available array of source models during testing, thereby enhancing performance beyond the scope of single source model adaptation. Moreover, employing a model ensemble provides the flexibility effortlessly incorporate exclude models post-deployment, aligning with the user's preferences and the needs of the given task. This flexibility not achievable with a single comain-generalize model trained on combined source data.
As an example, consider a scenario where a recognition model, initially trained on clear weather conditions, faces data from mixed weather scenarios, like sunshine interspersed with rain (see Figure 1). In such cases, employing multiple models specifically those trained on clear weather and rain with appropriate weighting can potentially reduce the test error opposed to relying single source model. In this context, the models for clear weather and rain would be assigned higher weights, while models for other weather conditions would receive relatively lesser weightage.
The main challenge of developing such a model ensembling method to learn appropriate com- bination weights optimally combine the source model ensemble during the test phase data is streaming in, such that it results in a test error equal or lower than that of the best source model. To solve this, propose CONtinual mulTi-souRce Adaptation to dynamic diStribuTions (CONTRAST) that handles multiple source models and optimally combines them to adapt to the test data.
The efficacy of using multiple source models also extends to preventing catastrophic forgetting that may arise when adapting to dynamic distributions for prolonged time. Consider again the scenario of multiple source models, each trained on different weather condition. During inference, only the parameters of the models most closely related to the weather encountered during test time will get updates, and the unrelated ones will be left untouched. This ensures that the model parameters do not drift too far from the initial state, since only those related to the test data are being updated. This mechanism mitigates forgetting when the test data distribution varies over long time scale, as is likely to happen in most realistic conditions. Even an entirely unrelated distribution appears during testing and there is no one source model to handle it, the presence of multiple sources can significantly reduce the rate at which the forgetting occurs. This is again because only the most closely related models (clear and rainy weather in the example above) updated, while others (e.g.. snow) are left untouched. Our setting is closely related to Test Time Adaptation methods (TTA) [3], and ours is the first to address adaptation of multiple sources for dynamically shifling distributions during test time.
Main Contributions, Our proposed approach, CONTRAST, makes the following contributions.

---

We propose framework for multi-souree adaptation to dynamic distribution shifts from streaming test data and without access the source data. Our approach has the ability to merge the source models using appropriate combination weights during test time, enabling it to perform just as well as the pest-performing source or even surpass it.
Our framework achieves performance on par with the best-performing source and also effectively mitigates catastrophic forgetting when faced with long-term, fluctuating test distributions.
We provide theoretical insights on CONTRAST, illustrating how it addresses domain shift by optimally combining source models and prioritizing updates to the model least prone to forgetring.
To demonstrate the real-worke advantages of our methodology. we perform experiments on a diverse range of benchmark datasets.
2 Related Works
Unsupervised Domain Adaptation. UDA methods have been applied to many machine learning tasks, including image classification [1], semantic segmentation [2], object detection [4] and rein- forcement learning [5], in an effort to address the data distribution shift. Most approaches try to align the source and target data distributions, using techniques such maximum mean discrepancy [6], adversarial learning [7, 1, 5] and image translation [8, 9]. Recently, there has been a growing interest in adaptation using only pre-trained source model due to privacy and memory storage concerns related to the source data [10-16]. These approaches include techniques such as information maximization [17-19], pseudo labeling [20, 21J, and self-superviiion [22].
Table I: Comparison of our setting to the existing adaptation settings. Our proposed setting meets all the criteria that are expected in comprehensive adaptation framework.
Source
Adaptation
Dynamic
Multi
Setting
Free
On the Fly
Target
Source
UDA [1]
Source-free free UDA [18]
TTA [3]
CONTRAST
Multi-Source Domain Adaptation (MSDA). Both UDA and source-free UDA have been extended to multi-source setting by incorporating knowledge from multiple source models [18, 23]. Notable techniques include discrepancy-based MSDA [24], higher-order moments [25], adversarial methods [26], and Wasserstein distance-base methods [27]. However, these methods are specifically tailored to UDA scenarios, where the whole target data is assumed to be available during adaptation. Whereas, in our setting we consider access to a batch of target data at an instance. Another related field is Domain Generalization (DG) [28, 29], which refers training a single model combined set of data from different source domains. Hence, DG requires data from all distinct domains to be available altogether during training, which may not be always feasible. Additionally, Model Soups [30] is a popular approach to ensemble models fine-tuned on same data distribution, where the weights of multiple models are averaged for inference. On the other hand, use weighting approach for model predictions, where models are pre-trained on different source data distributions. In our problem, inspired by MSDA, users are only provided with pre-traine pre-trained source models.
Adaptation to Dynamic Data. Few works [31-33] have addressed the adaptation to dynamic data distributions. However, these works either require source data or the entire target domain data to be available during adaptation. When additional constraints such as streaming target data batches and no access to source data are considered, the setting closely aligns with Test Time Adaptation (TTA). While UDA methods typically require substantial volume of target domain data for model adaptation, which is performed offline and prior to deployment, TTA adjusts a model post-deployment, during inference or testing. One of the early works [34] use test-batch statistics for batch normalization adaptation. Tent [3] updates a are-trained source model by minimizing entropy and updating batch- norm parameters. DUA [35] updates batch-norm stats with incoming test batches. TTA methods have also been applied to segmentation problems [36-39]. When these TTA methods are used to adapt to changing target distribution, they usually suffer from 'forgetting' and 'error accumulation' [40]. In order to solve this, CoTTA [40] restores source knowledge stochastically to avoid drifting of source knowledge, EATA [41] adds a regularization loss to preserve important weights for less forgetting. While motivated by TTA, our method considers multi-source adaptation in dynamic setting and
3

---

has an inherent capability to mitigate forgetting. In Table I, we illustrate comparison between our setting and existing settings.
3 CONTRAST Framework
3.1 Problem Setting
In this problem setting, propose combine multiple pre-trained models during test time through the application of suitable combination weights, determined based on a limited number of test samples. Specifically, will focus the classification task that involves K categories. Consider the scenario where have collection of N source models, denoted as {f' 1 that aim
to deploy during test time. In this situation, we assume that sequence of test data {z/ {2(")__ - .... {x]²)
+ ... are coming batch by batch in an online fashion, where is the index of time-stamp and B is the number of samples in the test batch. We also denote the test ( 1 1i=1
fz: |= 12
distribution at time-stamp as D(d), which implies {x.(*) LB ~ Df Motivated by [18], we model the test distribution in each time-stamp t linear combination of source distributions where the
combination weights are denoted by {w] N j==1 Thus, our inference model on test batch can be
written as e(0 = - Wj (()p)( where g/(() the adapted j-th source time stamp Based on this
setup our objective twofold:
We want to determine the optimal combination weights { (w/() } for the current test batch such that the test error for the optimal inference model is lesser than or equal to the test error of best source model. Mathematically we can write this as follows:
Cesst (ET)) I min (
(E)
I9SN
where (())(() (-) evaluates the test error I-th batch.
2. We also aim for the model to maintain consistent performance on source domains, as it progres- sively adapts to the changing test conditions. This necessary to ensure that the model has not catastrophically forgotten the original training distribution of the source domain and maintains its original performance the source data re-encountered in the future We would ideally want to have:
Eara (P(()) Rs Esre((s)
Vj, 1
(2)
where, sare (1/) denote the error of j-th source on its corresponding test data when using the original source model fg. whereas (fro(fs (f) (t)) represents the test error on the same test data using
the j-th source model adapted up to time step f, denoted as ed(()
3.2 Overall Framework
Our framework undertakes two operations on each test batch. First, we learn the combination weights for the current batch at time step t by freezing the model parameters. Then, update the model corresponding to the largest weight with existing state-of-the-art TTA methods, which allows us to fine-tune the model and improve its performance. This implies that the model parameters of source might get updated up to times at time-step f, where 0 S S (t- (t-
In other words, the states of the source models evolve over time depending the characteristics of the test batches up to the previous time step. To formalize this concept, we define the state of the
source model at time-step as (i(t) In the next section, we will provide detailed explanation of both aspects of our framework: (i) learning the combination weights, and (ii) updating the model
parameters. By doing so, we aim to provide comprehensive understanding of how our approach works in practice.
3.3 Learning the combination weights
For an unlabeled target sample x. that arrives time-stamp we denote its pseudo-label, as predicted by source as g. 2)()) (z!"), where ()(1) is the state of source at time-stamp t. Now

---

Time-stmm
1
Step 1
Step 2
W1
Repeat
(++3)
Inference
Weighted buot butch poeado label
DE000
(4+-3)
Aacepooccareaiiin wiigh
Updats
Gondia 0112200e indiati
paraneser Highos sprina comtoctio weight
Frocer parsiuters
Figure Overall Framework. During test time, we aim to adapt multiple source models in manner such that it optimally blends the sources with suitable weights based on the current test distribution. Additionally, we update the parameters of only one model that exhibits the strongest correlation with the test distribution.
we linearly combine these pseudo- labels by source combination weights [w, W2.. WN] R to get weighted pseudo-label 7!¹ Σj= wyDly (t) Using these weighted pseudo-labell for all the samples in the t-th batch we calculate the expected Shannon entropy as,
K
( (w) -B
Σ
Dic log(0
(3)
Based on this loss we solve the following optimization:
minimize
[(t) (w)
subject to
Wj 2 0, (1,
(4)
Σ Wj wj=1
Suppose we get (+))) to be the optimal combination weight vector by performing the optimization in (4). In such case, the optimal inference model for test batch can expressed as follows:
(t) ((())
(5)
Thus, by learning w this step, we satisfy Eqn. (1).
Model parameter update. After obtaining ((),, next we select the most relevant source model given by = arg max w;((), This indicates that the distribution of the current test batch is ISJEN
most correlated with the source model k. We then adapt model A to the test batch t using any state-of-the-art single source method that adapts to dynamic target distributions. Specifically, we employ three distinct adaptation approaches: (i TENT [3], (ii) CoTTA [40], and (iii) EaTA [41].
Optimization strategy for (4), Solving the optimization problem Eq. is a prerequisite for inferring the current test batch. As inference speed is critical for test-time adaptation, it is desirable to learn the weights quickly. To achieve this, we design two strategies: (i) selecting an appropriate initialization for w, and (ii) determining optimal learning rate.
(i) Initialization: Pre-trained models contain information about expected batch mean and variance in their Batch Norm (BN) layers based the data they were trained on. To leverage this information,

---

we extract these stored values from each source model prior to adaptation. Specifically, we denote the expected batch mean and standard deviation for the I-th layer of the j-th source model as ju| and of respectively.
During testing on the current batch t, we pass the data through each model and extract its mean and standard deviation from each BN layer. We denote these values as H4 (()) and of (() respectively. One useful metric for evaluating the degree of aligmment between the test data and each source is the distance between their respective batch statistics. A smaller distance implies stronger correlation between the test data and the corresponding source. Assuming that the batch-mean statistic per node of the BN layers to be univariate Gaussian, calculate the distance (KL divergence) between the j-th source (approximated as N(H (o/)²) and the t-th test batch (approximated N ((4, ((7))) (o) (())))) as follows (derivation in Appendix Section H):
Σ DEL [N (1, (o, .N (12.
R5
Im T())
im
T(t)
+ (
Mim
T(t) 2
log
2 2ommm)
2
1m 1
Im
where subscript Im denotes the mnth node of I-th layer. After obtaining the distances, we use
softmax function denoted by 8(-) to normalize their negative values. The softmax function is defined
exp(a,) as (a) ΣN rxp(a.)) where IR-V and j E 1,2, ... If of 101, 02.. 02.00.0000 0xx' IR is the
vectorized form of the distances from all the sources, we set
(E) Srit
6( 0(-
(6)
where w Wintt is the initialization for w. As we shall see, this choice leads to substantial performance boost compared to random initialization.
(t)
(ii) Optimal step size: Since we would like to ensure rapid convergence of optimization in Eqn. 4 select the optimal step size for gradient descent in the initial stage. Given initialization w (t) innt
and step size (1), we compute the second-order Taylor series approximation of the function c(t) at
the updated point after one gradient step. Next, determine the best step size Oe by minimizing
(t)
the approximation with respect (1),, This is essentially approximate Newton's 's method (details in Appendix section I) and has closed-form solution given by
best (Fw.. (FwC!t) (Vw Hw (V c())
wints
Here Cw and H.. are the gradient and Hessian of ((t) with respect to w. Together with w inat (t) and Obest optimization of (4) converges very quickly as demonstrated in the experiments (in Table 6 of Appendix). Please note that, we calculate the Hessian for only n scalar parameters, with representing the number of source models. Typically, common application domains, addressing distribution shifts requires only small number of source models, making the computational overhead of calculating hessian negligible.
Please refer to Algorithm for complete overview of CONTRAST.
3.4 Theoretical insights regarding combination weights
Theorem (Convergence of Optimization 4.). The Optimization 4 converges according the rule as follows:
V (1 (w Low (w))
(8)
(k +11 3=0
a (K + 1)
(t)
where, Vx represents the gradient the objective function over the set of n n-simplex & and j represents the iteration number.
Proof. Please refer to the Appendix (Section A) for the proof.

---

Implication of Theorem 1. The theorem tells us that to make the optimization converge faster with fewer iterations (small k), crucial start with a good initialization close the best solution ((C(w())) [(w*)) should be small). By using Eqn. (6), we ensure this condition for quicker convergence, Also, please note that in Theorem 1, j denotes the iteration number in the optimization process, and for simplicity, the batch number has been intentionally omitted from the notation.
Algorithm 1: Overview of CONTRAST
Input: Pre-trained source models (1/) 1 (
streaming sequential unlabeled test data
( { Fi=1 + { fi=1 + { fi=1 +...
Output: Optimal inference model for t-th test batch 1/ Vt Initialization: Assign f5(3)) + fg Vj
while t 2 1 do
Set initial w!o using Eqn. (6)
Set Set (1) using Eqn. (7)
Solve optimization 4 to get (*)))
Infer the test batch using inference model r2") using Equ. (5)
Find source index & such that & = arg max W (t))
I IS/SN
Update source model pk(()) according to Model Parameter Update paragraph of Section 3.3 to get fg" for j I N
p)())
jf t: then
(2)1 Set 1 (1++))
else
Set g
1((+))
Pg
end
end
end
end
3.5 Theoretical insights regarding model update
We now provide theoretical justification how CONTRAST selects the best source model by optimally trading off model accuracy and domain mismatch. At time t, let f( be the set of source models defined as (=1(() 2g()) [N(()] CONTRAST aims learn combination of these models by optimizing weights w on the target domain. For simplicity of exposition, we consider convex combinations w E A where A the N-dimensional simplex.
To learn w E A, CONTRAST runs empirical risk minimization on the target task using loss function E(-) with pseudo-labels generated by w-weighted source models. Let [(f) denote the target population/test risk of a model f (with respect to ground-truth labels) and C+ represent the optimal population risk obtained by choosing the best possible w A (i.e. oracle risk). We introduce the functions: (1) V which returns the distance between two data distributions and (2) yp which returns the distance between two label distributions. We note that, rather than problem-agnostic metries like
Wasserstein, our V o definitions are in terms of the loss landscape and source models ((t) hence tighter. We have the following generalization bound at time step (precise details in Appendix Section A).
<ll) Theorem 2. Consider the model fT with combination weights w (()) obtained via CONTRAST by
minimizing the empirical risk over B IID target examples per Equ. 5. Let I denote the pseudo-
label variable of w-weighted source models and ( = w: D denote weighted source
distribution. Under Lipschitz e and bounded f5 with probability at least 1 - 35 over the target batch, test risk obeys
(V(D)) ( + (l0l) ) + (e)))++VllI (((N+T)/B)
min
WEA
CONTRAST
Optimal
shif
quality
7

---

Proof. Please refer to the Appendix (Section A) for the proof.
Discussion. In a nutshell, this result shows how CONTRAST strikes a balance between: (1) choosing the domain that has the smallest shift from target, and (2) choosing a source model that has high-quality pseudo-labels on its own distribution (i.e. o matches you (()). From our analysis, is evident that, rather than adapting the source models to the target distribution, if we simply optimize the combination weights to optimize pseudo-labels for inference, the left side excess risk term (C(El))) - ( becomes upper bounded by a relatively modest value. This is because the shift and quality terms on the right-hand side are optimized with respect to w. We note that /N/B B is the generalization risk due to finite samples B and search dimension N.
To further refine this, our immediate objective to tighten the upper bound. This can be achieved by individually adapting each source model to the current test data, all the while maintaining the optimized w constant. Yet, such an approach is not ideal since our second goal is to preserve knowledge from the source during continual adaptation. To attain our desired goal, we must relax the upper bound, reducing our search over Å. Here, Á is the discrete counterpart of the simplex A. The elements of Á are one-hot vectors that have all but one entry zero The elements of Å essentially represent discrete model selection. Examining the main terms on the right reveals that: (i) source-target distribution shift and (ii) divergence between ground-truth and pseudo-labels are all minimized when we select the source model with the highest correlation to target. This model, denoted by r ,essentially corresponds the largest entry of (*) and presents the most stringent upper bound within the Á search space. Thus, to further minimize the right hand side, the second stage of CONTRAST adapts fg(t) with the current test data. Crucially, besides minimizing the target risk, this step helps avoids forgetting the source because (5 already does a good job at the target task. Thus, during optimization target data, fgg will have small gradient and will not move much, resulting in smaller forgetting. Please refer to the Appendix (Section A) for more detailed discussion along with the proof of this theorem.
4 Experiments
Datasets. We demonstrate the efficacy of our approach using both static target distribution and dynamic target data distributions. For static case, we employ the Digits and Office- Home datasets [42]. For the dynamic case, we utilize CIFAR-1OOC and CIFAR-IOC [43]. Detailed descriptions of these datasets along with results on segmentation task can be found in the Appendix.
Baseline Methods. As our problem setting is most closely related to test time adaptation, our baselines are some widely used state-of-the-art (SOTA) single source test time adaptation methods: specifically compare our algorithm with Tent [3], CoTTA [40] and EaTA [41]. These methods deal with adaptation from small batches of streaming data and without the source data, which is our
setting, and hence compare against these as our baselines. To evaluate the adaptation performance, we follow the protocol similar to [18), where we apply each source model to the test data from a particular test domain individually, which yields X-Best and X- where "X" is the name of the single source adaptation method, representing the highest and lowest performances among the source models adapted using method "X", respectively. For algorithm, we extend all of the methods "X" in the multi source setting and call the multi-source counterpart of "X" as "X+CONTRAST".
Implementation Details. We use ResNet-18 18 [44] model for all our experiments. For solving the optimization of Eq. (4), we first initialize the combination weights using Eq. (6) and calculate the optimal learning rate using Eq. (7). After that, we use 5 iterations to update the combination weights using SGD optimizer and the optimal learning rate. For all the experiments we use a batch size of 128, as used by Tent [3]. For more details on implementation and experimental setting see Appendix.
Experiment on CIFAR-100C. We conduct a thorough experiment on this dataset to investigate the performance of our model under dynamic test distribution. We consider 3 corruption noises out of 15 noises from CIFAR-100C which are adversarial weather conditions namely Snow, Fog and Frost. We add these noises for severity level 5 to the original CIFAR- training set and train three source models, one for each noise. Along with these models, we also add the model trained on clean training set of 100. During testing, we sequentially adapt the models across the 15 noisy
8

---

domains, each with a severity of 5, from the CIFAR- 100C dataset [40, 41]. We report the results for the experiment Table 2. Moreover, we also conduct an experiment on CIFARIO-C with the exact same experimental settings with CIFAR100-G CIFAR-IOC, results are in Table 5 of Appendixx
Table 2:
Results on CIFAR-100C: We take four source models trained on Clear, Snow, Fog, and Frost. We employ these models for adaptation on 15 sequential test domains. This table illustrates that even in the dynamic environment X+ CONTRAST performs better than X-Best, which is the direct consequence of optimal aggregation of source models as well as better preservation of source knowledge. (Results in error rate (in %))
GN
SN
IN
MB5
Snow
Frosi
Fog
Bright
Contrass
Elasacc
Fivel
JFEG
Mean
Source Want
92.7
98.3
98.2
68.8
78.1
66.1
65.1
53.5
59.3
62.0
55.8
95.4
61.9
71.3
75.2
73.7
Source Best
90.5
89.0
94.5
50.7
48.1
51.9
44.5
30.0
29.5
28.2
19.0
81.9
44.0
58.5
57.1
54.5
Teat Warst
71.2
58.0
15.5
78.2
89.2
92.4
93.7
55.4
5007
96.3
9606
96.3
82.5
65.6
76.7
75.3
91.3
68.2
CONTRANT
40.6
55.3
28.6
31.9
29.6
31.7
32.4
30.9
28.6
41.5
38.5
34.8
49.9
37.1
ETTA Warsi
37.7
34.0
66.5
40.5
53.2
41.4
36.8
44.0
43.5
45.4
34.8
45.4
45.7
3939
55.7
47.0
EaFA Best
48.1
44.7
57.9
37.1
44.1
38.7
34,9
33.7
31.9
31.6
33.2
37.2
40.0
34.7
50.3
39.9
ESTA CONTRAST
43.3
27.5
30.4
27.5
29.3
29.1
28.3
25.9
31.3
334
29.0
34.2
COTTA
37.4
41.4
47.6
CUTTA
49.8
46.6
58.6
34.0
40.7
36.5
34.2
34.0
32.8
33.0
37.8
34.8
35.3
33.6
41.1
38.5
COTTA F CONTRAST
44.6
43.8
57.2
27.8
37.6
30.6
28.0
29.3
29.3
28.2
26.6
30.0
33.5
29.7
41.4
34.4
From the table, we can draw two key observations:
(i) As anticipated, X+CONTRAST consistently outperforms X-Best across each test distribution, underscoring the validity of our algorithmic proposition.
(ii) Given that the CoTTA and EaTA methods are tailored to mitigate forgetting, the average eтror post- adaptation across the 15 noises using these methods is significantly lower than that of Tent, which not de- signed for this specific challenge. For instance, in Table 2, Tent-Bew Best error is approximately 68.2%. while CoTTA and Bast- are around 39.9% and 38.5%, respectively. However, when these adaptation methods are incor- porated into our framework, the final errors are remarkably close: 37.1% % for Tent, 34.2% for EaTA, and 36.9% for COTTA This suggests that even though Tent is more lightweight and faster compared to the other methods and is not inherently designed handle forgetting, its performance within our framework is on par with the results obtained when incorporating the other two methods designed to prevent forgetting. This shows the generalizability of our approach to various single-source methods. Note that identical to the experiment on CIFAR 100-C, the results on CIFAR1O-C in Table 5 follow the same trend where X+CONTRAST outperforms the X-Best.
Table 3: Results on Office-Home. We train three source models using three domains in this dataset and use them for testing the remaining domain under the TTA setting. Our results demonstrate that X+CONTRAST consistently outperfoor outperforms all of the baselines (X) (% error).
Ar
Cl
Pr
Rw
Avg.
Source Worst
61.4
64.9
46.2
43.9
54.1
Source Best
42.5
58.5
29.8
35.7
41.6
Tent Worst
S7.7
60.4
46.3
42.1
S1.7
Tent Best
41.4
54.3
27.9
36.0
39.9
+ CONTRAST
40.7
52.5
27.4
37.0
EaTA Worst
58.4
64.3
48.0
43.5
53.5
EaTA Best
42.1
57.8
30.3
35.9
41.5
EaTA CONTRAST
40.1
53.3
28.3
28.0
37.4
CoTTA Worst
58.3
62.9
47.1
42.8
52.8
CoTTA Best
42.1
55.0
29.0
34,9
40.2
C6TTA CONTRAST
40.6
53.3
28.3
29.0
37.8
Experiment on Office-Home. We report the results of the experiment on static distribution using the Office-Home dataset in Table 3. Each column in the table represents target domain from Office-Home dataset. We train three source models on the remaining Office-Home datasets. For instance, in case of 'Ar' column, 'Ar' is the target domain where three source models trained on "CI' 'Pr' and 'Rw' are adapted in test time. We calculate the test error of each incoming test batch and then report the numbers by averaging the error values over all the batches. The table shows that CONTRAST provides a significant reduction of test eтror compared to the best single source model. This demonstrates that when presented with an incoming test batch, CONTRAST has the capability to effectively blend all available sources using optimal weights, resulting in superior performance compared to the best single source model. It important to note that each test batch in this experiment is drawn from the same stationary distribution, which represents the distribution of the target domain. We conduct a similar experiment with the same experimental settings on Digits dataset that can be found in Table 4 of Appendix.

---

Analysis of Forgetting, Here, we demonstrate the robustness of our method against catastrophic forgetting by evaluating the classification accuracy on the source test set after completing adaptation to each domain [41, 45, 46]. For CONTRAST, we use our ensembling method to adapt to the incoming domain, After adaptation, we infer each of the adapted source models on its corresponding source test set. For the baseline single-source methods, every model is adapted individually to the incoming domain, followed by inference on its corresponding source test set. The reported accuracy represents the average accuracy obtained from each of these single-source adapted models.
From Figure 3, we note that our method consistently maintains its source accuracy during the adaptation process across the 15 sequential noises. In contrast, the accuracy for each individual single- source method (X) declines on the source test set as the adaptation process progresses. Specifically, Tent, not being crafted to alleviate forgetting, experiences a sharp decline in accuracy. While CoTTA and EaTA exhibit forgetting, it occurs at a more gradual pace. Contrary to all of these single-source methods, our algorithm exhibits virtually no forgetting throughout the process.
Ablation Study. We conduct an ablation study in Tables 6, in the Appendix evaluate the impact of various initialization and learning rate strategies on the optimization process described in (4), Our findings demonstrate that the initialization and learning rate configurations generated by our method outperform other alternatives.
Additionally, our experiments in Tables 8, 9 and 10 in the Appendix reveal that selectively up- dating the most correlated model parameters enhances performance compared to updating all model parameters, the least correlated ones,
Tent
CoTTA
EaTA
CaTTA-Durs
70-
Gasziiicatton Accurcc lin %)
08
a selected subset of correlated models or even updating the models according to their combi- nation weights. We report the comparison with
66-
64-
MSDA in Table 11 and Model-Soups in Table
12. We also report the values of the combination weights learned by our method. See Section D of the Appendix for detailed observations.
98
10
gasssaa
impouta
dedecca
briaatmnn
contrest
pieslate
Conclusions
Domains
We propose a novel framework called CON- TRAST, that effectively combines multiple source models during test time with small batches of streaming data and without access to the source data. It achieves a test accuracy that is at least as good as the best individual source model. In addition, the design of CONTRAST offers the added benefit of naturally preventing the issue of catastrophic forgetting. To validate the effectiveness of our algorithm, we conduct experiments on a diverse range of benchmark datasets for classification and semantic segmen- tation tasks. We also demonstrate that CONTRAST can be integrated with a variety of single-source methods. Theoretical analysis of the performance of CONTRAST is also provided.
Figure 3: Comparison with baselines in terms of source knowledge forgetting. Maintaining the same setting as in Table 2, we demonstrate that by integrating single-source methods with CON- TRAST, the source knowledge is better preserved during dynamic adaptation. Unlike all these single- source methods, our algorithm demonstrates virtu- ally no forgetting throughout the entire adaptation process.
6 Broader Impact and Limitations
Implementing multiple models for adaptation on dynamic distribution can yield broad impacts. For instance, this approach could find applications in robot navigation, autonomous vehicles or decision making in dynamically evolving scenarios. In all these cases, the algorithm can intelligently select the optimal combination of models during inference, ensuring sustained performance over extended periods. Our method currently assumes that data sampled within batch comes from the same distribution. In the future, we aim to explore using mixed data samples from different target domains within a batch.
10

---

Acknowledgments
The work was partially supported under NSF grant CCF-2008020. Additionally, research was sponsored by the OUSD (R&EVRT&L and was accomplished under Cooperative Agreement Number W911NF-20-2-0267. The views and conclusions contained in this document are those of the authors and should not be interpreted representing the official policies, either expressed or implied, of the ARL and ( or the U.S. Government. The U.S. Government is authorized to reproduce and distribute reprints for Government purposes notwithstanding any copyright notation herein. This work was also partially supported by the Laboratory Directed Research and Development (LDRD) Program (25-006) of Brookhaven National Laboratory under U.S. Department of Energy Contract No. DE-SC0012704.
References
[1] Eric Tzeng, Judy Hoffman, Kate Saenko, and Trevor Darrell. Adversarial discriminative domain adaptation. In CVPR, 2017.
[2] Yi-Hsuan Tsai, Wei-Chih Hung, Samuel Schulter, Kihyuk Sohn, Ming-Hsuan Yang, and Manmohan Chandraker. Learning adapt structured output space for semantic segmentation. In CVPR, 2018.
[3] Dequan Wang, Evan Shelhamer, Shaoteng Liu, Bruno Olshausen, and Trevor Darrell. Tent: Fully test-time adaptation by entropy minimization. In International Conference on Learning Representations, 2021. URL https: http//ppeeeeiieeeeeeeeeeeeeeeeee
[4] Han-Kat Kai Hsu, Chun-Han Yao, Yi-Hsuan Tsai, Wei-Chih Hung, Hung- Yu Tseng, Maneesh Singh, and Ming-Hsual Hsuan Yang. Progressive domain adaptation for object detection. In WACV, 2020.
[5] Dripta S Raychaudhuri, Sujoy Paul, Jeroen Vanbaar, and Amit K Roy-Chowdhury. Cross- domain imitation from observations. In ICML, 2021.
[6] Mingsheng Long, Yue Cao, Jianmin Wang, and Michael Jordan. Learning transferable features with deep adaptation networks. In JCML, 2015.
[7] Yaroslav Ganin, Evgeniya Ustinova, Hana Ajakan, Pascal Germain, Hugo Larochelle, François Laviolette, Mario Marchand, and Victor Lempitsky. Domain Domain-adversarial training of neural networks. In JMLR, 2016.
[8] Judy Hoffman, Eric Tzeng, Taesung Park, Jun-Yan Efros, and Trevor Darrell. Cycada: Cycle-consistent adversarial domain adaptation. In ICML, 2018.
Yan Zhu, Phillip Isola, Kate Saenko, Alexei A.
[9] Luan Tran, Kihyuk Sohn, Xiang Yu, Xiaoming Liu, and Manmohan Chandraker. Gotta adapt 'em all: Joint pixel and feature-level domain adaptation for recognition the wild. CVPR. 2019.
[10] Shiqi Yang, Yaxing Wang. Joost Van De Weijer, Luis Herranz, and Shangling Jui. Generalized source-free domain adaptation. In Proceedings of the IEEF/CVF International Conference Computer Vision, pages 8978-8987, 2021.
[11] Jogendra Nath Kundu, Naveen Venkat, R Venkatesh Babu, et al. Universal source-free domain adaptation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pages
2020.
[12] Fan Wang, Zhongyi Han, Yongshun Gong, and Yilong Yin. Exploring domain-invariant parameters for source free domain adaptation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 7151-7160, 2022.
[13] Shiqi Yang. Yaxing Wang, Joost Van De Weijer, Luis Herranz, and Shangling Jui. Unsupervised domain adaptation without source data by casting a bait. arXiv preprint arXiv:2010. 20.::0000./222,, (2):5, 2020.
11

---

[14] Rui Li, Qianfen Jiao, Wenming Cao, Hau-San Wong, and Si Wu. Model adaptation: Unsuper- vised domain adaptation without souree data. In Proceedings of the IEEE/CVE conference on computer vision and pattern recognition, pages 9641-9650, 2020.
[15] Ning Ding, Yixing Xu, Yehui Tang, Chao Xu, Yunhe Wang, and Dacheng Tao. Source-free domain adaptation via distribution estimation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pages 7212-7222, 2022.
[16] Dian Chen, Dequan Wang, Trevor Darrell, and Sayna Ebrahimi. Contrastive test-time adaptation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 295-305, 2022.
[17] Jian Liang, Dapeng Hu, and Jiashi Feng. Do we really need to access the source data? source hypothesis transfer for unsupervised domain adaptation. In ICML, 2020.
[18] Sk Miraj Ahmed, Dripta S Raychaudhuri, Sujoy Paul, Samet Oymak, and Amit K Roy- Chowdhury. Unsupervised multi-source domain adaptation without access source data. In CVPR, 2021.
[19] Sk Miraj Ahmed, Suhas Lohit, Kuan-Chuan Peng, Michael J Jones, and Amit K. Roy- Roy-Chowdhury. mross-modal knowledge transfer without task-relevant source data. In Computer ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23,27, 2022, Proceedings, Part XXXIV, pages 111-127. Springer, 2022.
[20] Hao-Wei Yeh, Baoyao Yang, Pong C Yuen, and Tatsuya Harada. Sofa: Source-data-free feature alignment for unsupervised domain adaptation. In WACV, 2021.
[21] Vikash Kumar, Rohit Lal, Himanshu Patil, and Anirban Chakraborty. Conmix for source-free single and multi-target domain adaptation. WACV, 2023.
[22] Haifeng Xia, Handong Zhao, and Zhengming Ding. Adaptive adversarial network for source- free domain adaptation. In ICCV, 2021.
[23] Sicheng Zhao, Bo Li, Pengfei Xu, and Kurt Keutzer. Multi-source domain adaptation in the deep learning era: A systematic survey. arXiv preprint arXiv:2002.12116; 2020.
[24] Jiang Guo, Darsh J Shah, and Regina Barzilay. Multi-source domain adaptation with mixture of experts. arXiv preprint arXiv: 0225602266.............0 2018
[25] Xingchao Peng, Qinxun Bai, Xide Xia, Zijun Huang, Kate Saenko, and Bo Wang. Moment matching for multi-source domain adaptation. In Proceedings of the IEEE/CVF international conference on computer vision, pages 1406-1415, 2019.
[26] Ruijia Xu, Ziliang Chen, Wangmeng Zuo, Junjie Yan, and Liang Lin. Deep cocktail network: Multi-source unsupervised domain adaptation with category shift. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 9964-3973 2018
[27] Yitong Li, David Carlson, al. Extracting relationships by in Neural Information Processing Systems, 31, 2018
matching. Advances
[28] Abhimanyu Dubey, Vignesh Ramanathan, Alex Pentland, and Dhruv Mahajan. Adaptive methods for real-world domain generalization. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 14340-14344 2021.
[29] Zehao Xiao, Xiantong Zhen, Ling Shao, and Cees GM Snoek. Learning to generalize across domains on single test samples. arXiv preprint arXiv:2202.06 2..40000000000455 2022.
[30] Mitchell Wortsman, Gabriel Ilharco, Samir Ya Gadre, Rebecca Roelofs, Raphael Gontijo-Lopes, Ari S Morcos, Hongseok Namkoong, Ali Farhadi, Yair Carmon, Simon Kornblith, et al. Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference time. In International conference on machine learning, pages 23965-23992 PMLR, 2022.
12

---

[31] Zuxuan Wu, Xin Wang, Joseph E. Gonzalez, Tom Goldstein, and Larry S. Davis. Ace: Adapt- ing to changing environments for semantic segmentation. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), October 2019.
[32]
Mohammad Rostami. Lifelong domain adaptation via consolidated internal distribution. Ad- vances in neural information processing systems, 34:11172-1118 2021.
[33] Theodoros Panagiotakopoulos, Pier Luigi Dovesi, Linus Siveenstam-Nielsee and Matteo Poggi. Online domain adaptation for semantic segmentation in ever-changing conditions. In European Conference Computer Vision, pages 128 146. Springer, 2022.
[34] Yanghao Li, Naiyan Wang, Jianping Shi, Jiaying Liu, and Xiaodi Hou. Revisiting batch normalization for practical domain adaptation. arXiv preprint arXiv: 1603.04779, 2016.
[35] M Jehanzeb Mirza, Jakub Micorek, Horst Possegger, and Horst Bischof. The norm must go on: dynamic unsupervised domain adaptation by normalization. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 14765-14775, 2022.
[36] Jeya Maria Jose Valanarasu, Pengfei Guo, Vibashan VS, and Vishal M Patel. On-the-fly test-time adaptation for medical image segmentation. arXiv preprint S.Kiv:2203.055744 2022.
[37] Inkyu Shin, Yi-Hsuan Tsai, Bingbing Zhuang, Samuel Schulter, Buyu Liu, Sparsh Garg, In So Kweon, and Kuk-Jin Yoon. Mm-tta: multi-modal test-time adaptation for 3d semantic segmentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 16928-16937, 2022.
[38] Yuang Liu, Wei Zhang, and Jun Wang. Source-free domain adaptation for semantic segmentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition. pages 1215-1224, 2021.
[39] Minhao Hu, Tao Song, Yujun Gu, Xiangde Luo, Jieneng Chen, Yinan Chen, Ya Zhang, and Shaoting Zhang. Fully test-time adaptation for image segmentation. In Medical Image Com- puting and Computer Assisted Intervention-MICCA. Strasbourg, France, September 27-October 1, 2021, Proceedings, Part 24, pages 251-260. Springer, 2021.
2021: 24th International Conference,
[40] Qin Wang, Olga Fink, Luc Van Gool, and Dengxin Dai. Continual test-time domain adaptation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pages 7201-7211, 2022.
[41] Shuaicheng Niu, Jiaxiang Wu, Yifan Zhang, Yaofo Chen, Shijian Zheng, Peilin Zhao, and Mingkui Tan. Efficient test-time model adaptation without forgetting. In International confer- ence on machine learning, pages 16888-16905. PMLR, 2022.
[42] Hemanth Venkateswara, Jose Eusebio, Shayok Chakraborty, and Sethuraman Panchanathan. Deep hashing network for unsupervised domain adaptation. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 5018-5022 2017.
[43] Dan Hendrycks and Thomas Dietterich. Benchmarking neural network robustness to common corruptions and perturbations. arXiv preprint 1903. 122611, 2019.
[44] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770-778, 2016.
[45] Junha Song, Jungsoo Lee, In So Kweon, and Sungha Choi. Ecotta: Memory-efficient continual test-time adaptation via self-distilled regularization. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 1 11920-11929, 2023.
[46] Goirik Chakrabarty, Manogna Sreenivas, and Soma Biswas. Sata: Source anchoring and target alignment network for continual test time adaptation. arXiv preprint a3X4v:2304.100 10/13, 2023.
[47] Koulik Khamaru and Martin Wainwright. Convergence guarantees for a class of non-convex and non-smooth optimization problems. In International Conference on Machine Learning, pages 2601-2610. PMLR, 2018
13

---

[48] Samet Oymak, Mingchen Li, and Mahdi Soltanolkotabi. Generalization guarantees for neural architecture search with train-validation split. International Conference Machine Learning. pages 8291 8291-8301. PMLR, R, 2021.
[49] Marius Cordts, Mohamed Omran, Sebastian Ramos, Timo Rehfeld, Markus Enzweiler, Rodrigo Benenson, Uwe Franke, Stefan Roth, and Bernt Schiele. The cityscapes dataset for semantic urban scene understanding. In Proceedings of the IEEE conference computer vision and pattern recognition, pages 3213-3223, 2016.
[50] Christos Sakaridis, Dengxin Dai, Simon Hecker, and Luc Van Gool. Model adaptation with synthetic and real data for semantic dense foggy scene understanding. In Proceedings of the european conference on computer vision (ECCV), pages 687-704, 2018.
[51] Xiaowei Hu, Chi- Whig Fu, Lei Zhu, and Pheng-Ann Heng. Septh-attentional features for single-image rain removal. In Proceedings of the IEEE/CVF Conference on computer vision and pattern recognition, pages
2019.
[52] Christos Sakaridis, Dengxin Dai, and Luc Van Gool. Acde: The adverse conditions dataset with correspondences for semantic driving scene understanding. In Proceedings of the IEEE/CVF International Conference Computer Vision, pages 10765-10775, 2021.
[53] Liang-Chieh Chen, Yukun Zhu, George Papandreou, Florian Schroff, and Hartwig Adam. Encoder-decoder with atrous separable convolution for semantic image segmentation. In Proceedings of the European conference computer vision (ECCV), pages 801-818, 2018.
14

---

Appendix Overview:
Contents
Proof and discussion of Theorems and
16
B Results Digits
16
Results on CIFAR-10C
17
Ablation Study
18
Initialization and Learning Rate
18
D.2 Model Update Policy
18
D.2.1 Subset of Model Update
18
D.2.2 Model Update According to Weight
19
D.3
Combination Weight Visualization
20
D.4
Comparison with MSDA
20
D.5
Comparison with Model Soups
20
E
Implementation Details
21
E.1
Stationary Target
21
E.1.1
Digit Classification
E.1.2
Object Recognition
E.2
Dynamic Target
21
E.2.1
CIFAR-10/100-C
Semantic Segmentation
21
F.1
Datasets
22
F.2
Experimental setup
22
F3
Visualization
22
Additional discussion
23
KL divergence between two univariate Gaussians
24
Optimal step size in approximate Newton's method
25
15

---

A Proof and discussion of Theorems and 2
Proof of Theroem I. The optimization (4) has structure similar to class of non convex problems as follows:
minimize X
g(z)- h(x))
(9)
where X closed convex set, g(z) is M, smooth and h(x) continuous convex function. such cases, the optimization converges as follows [47]:
- a(k +
[xlIf(>*)11)
(10)
I++1) ==0
where, f(x) (g(x) h(z)).
In our case g(x) = c, where a constant (smooth and continous) and h(x)) negative of the Shannon entropy, which is continous and convex. Also, X is the n-simplex N, which is a closed
convex set. So, according to the proof derived [47], we can conclude the bound in Theroem O
We adapt the theorem from corollary (corollary 1) in [48]. In this corollary the following result was derived:
Proof of Theorem 2.
f((2) S min (D)) Dr (a) + + (Fa))
aEA
( +t)nm) 6
Here f in the f is the trained model on the training(r) distribution D' and â is hyper-parameter that has been empirically optimized by fine tuning on the validation(v) distribution D. L is the expected risk over the distribution D. DM measures the distribution mismatch via difference of uub-oitimality gap using the training and validation distribution. (Fa)) is the Rademacher complexity of the function class F with as the hyper-parameter The corollary holds for probability of at least 1 - 3e and hefs is the effective dimension of the typer-parameter space. Also is the number of samples under the validation. The bound can be first of all easily extended to the source/target scenario instead of train/validation. In our scenario the source models jointly construct the function class Fa where, the hyper-paramett a is the combination weight w. Effective dimension for case is exactly the number of source model N and instead of we took as the probability variable. For the sake simplicity we omitted ê > which is a positive constant along with the Rademacher complexity. Also n,, B in our setting since we have B number of samples for the target/validation. Now there a new term in our bound which is &o which was not in the original corollary. This term is used to account for the mismatch between actual and pseudo-labels generated by the source. This is done due to the fact that we do empirical minimization of the entropy of the target pseudo-label since the problem is unsupervised and actual labels not available. The left side of the inequality is derived using the test/target pseudo-label. Consequently, we can introduce an added distribution mismatch term. This term can be broken down into three components: mismatch from target pseudo to target ground truth (gt), from target gt to source gt, and from source gt to source pseudo label. Of these components, the first two be readily integrated into the (.) function given that it measures the discrepancy between the weighted source and the target. The remaining third component denoted by the (p.) function. This completes the proof.
B Results on Digits
We report here the results of digit classification in Table 4. Similar to the experiment on Office-Homn dataset, each column of the table represents target domain dataset. We train four source models on the rest of the digit datasets. For instance, in case of 'MM' column 'MM' is the target domain which is adapted using four source models trained on 'MT', 'UP', 'SV' and 'SY" respectively.
We once again calculate the test error for each incoming test batch and report the results by averaging the errors across all batches. The table demonstrates that CONTRAST achieves significant reduction
16

---

Table Results on Digits dataset. We train the souree models using four digit datasets to perform inference on the remaining dataset. The column abbreviations correspond to the datasets as follows: 'MM' for MNIST-M, 'MT" for MNIST, 'UP" for USPS, 'SV' for SVHN, and 'SY' for Synthetic Digits.. The table (reporting % error rate(()) shows that X+CONTRAST outperforms all of the baselines (X- (X-Best) consistently.
MM
MT
UP
SV
SY
Avg.
Source Worst
80.5
59.4
50.3
88.3
84,8
72.7
Source Best
47.7
2.2
16.8
18.3
6.7
18.3
Tent Worst
84.2
46,9
41.1
90.1
85,4
69.5
Tent Best
45.2
2.3
16.7
14.4
6.7
17.1
Teni CONTRAST
37.5
1.9
11.2
14.2
6.7
EaTA Worst
80.1
48.4
42.6
88.0
83.1
68.4
EaTA Best
47.1
2.7
18.2
18.5
7.2
18.7
FaTA CONTRAST
39.5
2.0
11.5
18.0
7.0
15.6
CoTTA Worst
80.0
48.3
42.8
87.9
82.9
68.4
CoTTA Best
47.0
2.8
18.6
18.5
7.2
18.8
C6TTA CONTRAST
39.6
2.0
11.7
18.1
7.1
15.7
in test error compared to the best single source (on average 3% error reduction than the best source). Another baseline exists that simply uses naive ensemble of the source models, without any weight optimization. In situations where there's significant performance gap between the best and worst source models adapted using single-source methods, uniform ensemble of these models produces a predictor that trails considerably behind the best-adapted source, as noted by [18]. Referring Table 4, when testing on the SVHN dataset, the error disparity between the best and worst adapted source models is approximately 70.7%-a substantial margin. Consequently, using a uniform ensemble in such a scenario results in an error rate of roughly 45.5% (experimentally found, not reported in the table). This is strikingly higher than our method's 's error rate of around 14.2%. Given these findings, we deduce that uniform ensembling is not a reliable approach for model fusion. Thus, we exclude it from our experiment section's 's baseline.
C Results on CIFAR-10C
Here, we report the results on dynamic target distribution using CIFAR-IOC dataset. Note that identical the experiment on CIFAR100-C in the main paper the results on CIFARIO-C in Table follow the same trend where X+CONTRAST outperforms the X- X-Best.
Table 5: Results on CIFAR-1OC. We take four source models trained on Clear, Snow, Fog and Frost. We employ these models for adaptation on 15 sequential test domains. This table illustrates that even in the dynamic environment X+CONTRAST performs better than X. which is the direct consequence of better retaining source knowledge. (Results in error rate (in %))
GN
SNN
LN
D8
GB
MB
ZBB
Snew
Frost
Fog
Bright
Contrast
Elastic
Pisel
JPLG
Mean
Source Wanst
84.7
$1.1
89.1
42.6
55.6
36.2
32.2
30.5
39.2
25.7
18.5
76.4
26.9
50.0
52.7
18.3
Source
20.4
8.1
8.2
6.9
10.6
18.5
13.9
23.9
30.1
Test Warst
26.6
22.7
20.0
34.9
28.8
28.7
32.8
34.4
36.1
30.3
38.2
41.7
33.5
Ten: Best
19.3
17.6
27.9
14.5
21.1
17.6
13.5
14.3
12.6
14.4
12.4
17.0
19.0
14.3
20.4
17.1
Teni + CONTRAST
17.2
15.6
25.7
9.1
19.1
11.7
9.0
9.9
10.1
9.7
7.7
11.7
14.5
10.3
17.4
13.2
ESTA Wordi
31.3
30.4
44.8
14.8
33.9
16.1
13.4
20.3
21.6
19.3
11.2
18.9
23.2
19.3
29.6
23.2
Best
9.0
7.5
8.5
15.1
CONTRAST
8.0
9.1
10.0
12.0
CA Wons
30.1
26.8
37.8
13.0
28.5
16.6
14.6
1933
18.6
17.
12.2
13.9
194
154
193
20.5
CoTTA Best
21.0
18.5
28.0
11.2
17.3
13.3
11.1
10.6
10.4
9.5
9.7
11.2
13.1
10.5
15.6
14.1
COTTA CONTRAST
1854
17.0
28.0
8.4
17.7
10.7
7.9
9.1
8.4
8.5
6.8
83
12.1
9.3
153
12.4
In the single-source scenario, one among the four source models achieves the X-Best (for example CoTTA-Best) accuracy for a specific domain. The determination of which individual model (from the four) will attain the best accuracy for that domain remains uncertain beforehand. Furthermore, the individual source model yielding the X-Best accuracy varies across different domains within CIFAR1O-C. However, in our X+CONTRAST approach, the need to deliberate over the selection of out of the four source models is eliminated. X+CONTRAST reliably outperforms any single source X-model that might achieve the X- X-Best accuracy.
Individual TTA methods may have distinct advantages. For example, Tent offers several distinct advantages over CoTTA, including its lightweight nature and faster performance. Conversely, CoTTA presents certain benefits over Tent, such increased resilience against forgetting, Consequently, the
17

---

choice between TTA methods is dependent on the user's preferences, aligning with the specific task hand. In this experiment, we have demonstrated that CONTRAST can be integrated with any TTA method of the user's choosing.
D Ablation Study
D.1 Initialization and Learning Rate
Table 6: Effect of initialization and step size choice. Error rate on Office-Home under different
choices of initialization and step sizes.
Step size
Initialization le
3
le
2
1c
1c0
lel
Ours
Random
40.7
40.9
40.6
39.6
41.5
39.3
Ours
37.9
37.8
37.5
37.4
39.1
37.0
Table presents the eπror rate results on the Office-Home dataset under the same experimental setting as Table (Appendix) with Tent as the adaptation method, but with different initialization and learning rate choices for solving the optimization in (4). It is evident from the table that our chosen initialization and adaptive learning rate result in the highest accuracy gain.
We additionally show another ablation study in Table 7, where we initialize the combination weights based the probability of source model predictions. More precisely, we set the initial weights inversely proportional to the entropy of the source model predictions. In simpler terms, a source model with low entropy receives a higher weight, while one with high entropy receives lower weight.
Table 7: Initialization based on Entropy. The table shows the results of entropy based initialization.
(Results in error-rate % !)
Updare Policy
GN
SN
19
D8
GB
MB
78
Snow
Frost
Fog
Bright
Contrast
Elassic
Pixel
JPEG
Mean
Entropy init
427
41.1
36.9
33.5
463
394
37.
41.0
43.2
50.6
46.7
78.6
77.9
79.5
88.7
53.6
Ours
In the presented table for CIFAR-100C) we note a 16.5% reduction in error resulting from our initialization method. We found that initializing the combination weights using the entropy of the test batch for various sources leads to somewhat uniform initialization. However, when we initialize the combination weights using KL divergence, we achieve a highly effective and peaky prior, favoring the most correlated source model with relatively higher weightage. This clarifies why initializing with entropies fails converge quickly the optimum, resulting significantly poorer outcomes compared to our method.
D.2 Model Update Policy
In Table 8 and 9, demonstrate that by updating only the model with the highest correlation the target domain, our method produces the lowest test accuracy. This is in comparison to scenarios where we either update all models solely the least correlated one. This empirical observation directly supports our theoretical assertion from the theorem: updating the most correlated model is most effective in preventing forgetting, thereby resulting the smallest error during gradual adaptation. We also experiment with another model update policy where a subset of model is updated.
D.2.1 Subset of Model Update
In this approach, rather than focusing solely on the most correlated source model, we identify and update subset of source models that exhibit higher correlation than the rest of the models. Specifically, we select models for updating based on their combination weights, choosing only those whose weights exceed 1/n, with n representing the total number of models. The intuition behind selecting this threshold 1 1/n for subset selection is grounded in the distance of the combination weight distribution with respect to the uniform distribution. A uniform combination weight implies that all
18

---

Table 8: Choice of model update (CONTRAST+CoTTE) In experiments using CoTTA the model update method on CIFAR100-C, we tested four scenarios: updating all models, updating only the least correlated model, updating subset of model, and updating only the most correlated model. Our results indicate that our model selection approach produces the most favorable outcome. (Results in error rate (in %))
Update Policy
Fog
Blassic
IPBG
Mean
All Model Updase
44.0
42.5
54.5
30.1
38.9
31.7
32.1
32.1
32.6
30.2
22.8
34.5
32.0
40.2
36.2
Lesst Coer. Updatc
44.8
44.5
58.9
28.6
38.7
31.0
28.4
29.1
28.9
29.5
26.9
30.9
33.8
30.5
44.0
35.2
Subset of Models Updase
44.5
43.3
57.1
28.1
37.5
30.6
28.4
29.9
29.9
28.8
26.8
30.2
32.4
90.2
40.4
34.5
Most Cor Upsine
44.5
438
57.
27.8
37.
38.6
28.0
29.3
29.3
28.2
26.6
30,0
32.5
29.7
41.4
34.4
Table Choice of model update (CONTRAST+Tent) In our experiments using Tent as the model
update method CIFAR we tested four scenarios: updating all models, updating only the
least correlated model, updating subset model, and updating only the most correlated model. Our results indicate that our model selection approach produces the most favorable outcome. (Results in error rate & (in %))
Update Roicy
GN
SN
178
G8
MB
7B
Snow
Frost
F6g
Bright
Contenst
Elattic
Fixel
IPE3
Mean
All Modil Update
41.6
40.9
57.8
47.1
60.2
60.3
62.1
68.6
73.2
30.9
82.1
92.4
91.2
92.5
94.9
09.7
Least Corz. Update
43.
34.8
31.4
33.5
33.1
37.5
31.5
41.5
37.5
39.3
Subset Modell Update
Updbbee
models equidistant w.r.t the test distribution and should be updated. However, if only model weight surpasses 1/n, it signifies that only one model exhibits a high correlation with the overall model.
Results are shown in Table 8 and 9. Several key observations can be extracted from here. Notably, when utilizing the Tent adaptation algorithm, updating subset of models results in significantly poorer performance compared to updating only the most correlated model. Conversely, with the CoTTA adaptation algorithm, the performance decrement from updating subset of models is relatively minor compared to updating the most correlated model. This discrepancy can be attributed to the varying degrees of resistance forgetting exhibited by these adaptation algorithms. Updating multiple models tends to induce forgetting, leading to a decline in overall performance, especially when the adaptation algorithm not highly resistant forgetting. Despite the adaptation method's robustness to forgetting, it has been consistently observed that updating the most correlated model not only delivers superior performance but also offers computational advantages updating subset of models. This approach simplifies the update process and ensures more efficient use of computational resources.
D.2.2 Model Update According Weight
Here, we update the model j weighted by w/y To do we need to properly devise an approach that updates models in measures according to their correlation with the test data. Drawing inspiration from recent studies that employ variable learning rates for single-source TTA, we devise strategy to adjust the learning rate Y) used in updating model j based on their respective combination weights Wy- Specifically, we assigned the highest learning rate Amaz 0.001 (0.001 is the learning rate used for both Tent and CoTTA in our experiments) to the model with the greatest combination weight, while the lowest learning rate Timin 0.0001, (a tenfold reduction) was allocated to the model with the lowest combination weight. For the remaining models, we interpolated their learning rates proportionally between the highest and lowest rates, based their respective combination weights
x (Mmax Mm + Mmin In the Table 10, we present the resulting error rates for CIFAR-I00G dataset using both Tent and CoTTA
following the formula:
10, Wimin
Table 10: Model Update according to Weight. The table shows results of updating model according their respective weights. (Results in error-rate %
Updine Policy
Bright
JPEG
Tom
41.
59.7
55.0
33.9
43.9
36.8
34.5
37.8
30.3
41.0
36.5
36.1
45.5
41.4
661
43.0
CONTRAST-Test
42.2
60.5
55.3
28.5
90.7
31.9
29.5
31.7
32.4
30.9
28.6
41.5
38.5
34,8
49.9
37.1
CoTTA
64.5
43.0
56.2
28.1
38.1
30.8
28.5
29.9
29.6
28.7
27.0
29.5
31.8
29.0
38.6
31.2
CONTRAST-CATTA
44.6
43.8
57.2
37.6
30.6
20.3
19

---

Our investigation reveals that, in scenarios where the update algorithm exhibits limited robustness against forgetting, such as Tent, updating only the model with the highest combination weight proves more advantageous. This is because even marginal updates to uncorrelated models can lead to detrimental forgetting, resulting poor performance. Conversely, when the update algorithm demonstrates resilience against forgetting (CoTTA), updating the most correlated model impacts per- formance the most. While updating uncorrelated models does not substantially enhance performance, it significantly increases computational costs. It should also be noted that we have found exactly same finding with our ablation study focused on updating subsets of models. Consequently, we assert that updating the single model with the highest combination weight yields optimal performance across all scenarios.
D.3
Combination Weight Visualization
To provide insight into the combination weight distribution, let's consider an example where the source models are trained on the clean, snow, frost, and fog domains using the training data. We then select one of these domains to collect the average weights over all the test data. When the test data from the fog domain, the weight distribution appears follows: [0.05, 0.08, 0.09, 0.78]. the other hand, when the test domain is frost, we observe the following weight distribution: [0.07, 14, 0.69, 0.11]. These results clearly illustrate that the weight distribution accurately reflects the correlation between the source models and target domains.
D.4 Comparison with MSDA
Existing multi-source source-free methods are designed for offline settings where all the target data are available during adaptation. However, in our setting, data is received batch by batch during adaptation. Therefore, theoretically, these methods are expected to perform worse in our setup. Nevertheless, we compared CONTRAST with the seminal paper [18] on source-free multi-source Unsupervised Domain Adaptation (UDA), specifically the DECISION method, demonstrate its effectiveness in an online adaptation setting. We keep the hyperparameters exactly the same as described in the DECISION and perform adaptation on each incoming batch of test data with the number of epochs specified in DECISION.
Table 11: Comparison with MSDA. The table compares the performance of our method with MSDA approach DECISION. (Results in error-rate % J)
Updize Policy
Snow
Frost
Bright
Contrnst 98.7
Pixel
JPEG
DECISION
550
76.2
9015
93.2
97.
97.9
98.2
98.0
98.3
98.4
98.4
99.0
98.9
98.9
93.3
CONTRAST-Ienl
422
40.6
55.
28.6
40.7
31.9
29.5
31.7
32.4
30.9
28.6
41.5
38.5
34,8
49.9
37.1
It is evident from Table 11 that DECISION performs notably poorly in the online setting, with an eтror rate almost 56% higher than CONTRAST. DECISION utilizes clustering of the entire offline dataset based on the number of classes, method not feasible to accurately implement in our setting with very small batch sizes. This highlights the necessity of a multi-source method specifically tailored for our setting.
D.5 Comparison with Model Soups
Model Soups [30] is a popular approach for utilizing set of models by averaging their parameters to create single model for inference on test data. For completeness, we compare our method against Model Soups.
Table 12: Comparison with Model Soups. The table compares the performance our method against model soups. (Results in error-rate % L)
Updine Poliey
ON
SN
IN
D8
GB
M88
28
Snow
Foost
Fog
Bright
Contrast
Elastic
Pisel
JPEG
Mean
Soupe
06.83
36
97.08
05.17
05.33
95.0
65.23
95.17
95.5
05.26
94 96
97
05.04
95.05
95.80
95.73
CONTRAST-
42.2
55.3
28.6
31,9
29.6
32.4
30.9
28.6
38,5
34.8
49.9
37.1
As shown in Table 12, the performance of Model Soups is significantly worse compared to our method. Model Soups averages the parameters of models fine-tuned on the same data distribution. However, in our setting, we have models trained on different source domains, making the averaging of model parameters suboptimal.
20

---

E Implementation Details
In this section, we provide comprehensive overview of our experimental setup. We conducted two sets of experiments: one on a stationary target distribution, and the other on dynamic target distribution that changes continuously. The reported results in the main paper are average of three runs with different seeds.
E.1
Stationary Target
E.1.
Digit Classification
The digit classification task consists of five distinct domains from which we construct five different adaptation scenarios. Each scenario involves four source models, with the remaining domain treated as the target distribution. In total, we construct five adaptation scenarios for our study.
The ResNet-18 architecture was used for all models, with an image size of 64 x 64 and a batch size of 128 during testing. Mean accuracy over the entire test is reported in Table 2 of the main paper. For Tent we use learning rate of 0.01 and for rest of the adaptation method a learning rate of 0.001 is used. We use Adam optimizer for all the adaptation methods. Model parameter update is performed using a single step of gradient descent.
E.1.2 Object Recognition
The object recognition task on the Office-Home dataset comprises of four distinct domains from which we construct four different adaptation scenarios, similar to the digit classification setup. We use the same experimental settings and hyperparameters as the digit classification experiment, with the exception of the image size, which evaluation are reported in Table 3 of the main paper.
224 224 in this experiment, The results of this
E.2
Dynamic Target
E.2.1
CIFAR-10/100-C
In this experiment, we use four ResNet-18 source models trained on different variants of the CIFAR- 10/100 dataset: 1) vanilla train set, 2) train set with added fog (severity = =), 3) train with added snow (severity = 5), and 4) train set with added frost (severity = 5). To evaluate the models, we use the test set of CIFAR- CLFAR-10/1006 10/100C (severity 5) and adapt each of the domains in continual manner. The images are resized to 224 x 224. For all the adaptation methods, a learning rate of 0.001 with Adam optimizer used.
F Semantic Segmentation
Table 13: Result Cityscape to ACDC: In this experiment, test method the test data from individual weather conditions (static test distribution) of ACDC. The source models are trained on the train set of Cityscape and its noisy variants. Our method clearly outperforms baseline adaptation method. (Results in % mloU)
Method
Fog
Rain
Snow
Night
Avg.
Best
25.3
21.0
19.2
12.6
19.5
CONTRAST
27.7
22.8
21.1
14.0
21.4
Our method is not just limited image classification tasks and can be easily extended other tasks like semantic segmentation (sem-seg). We assume access to a set of sem-seg source models [ where each model classifies every pixel of an input image to some class. Specifically, C3 HxW -> TR x where K the number of classes. this case, the entropy in Eqn. 3 of the main paper will be modified as follows:
21

---

Table 14: Result on Cityscapes ACDC for dynamic distribution: This table illustrates that over prolonged cycle of repetitive test distributions, our model can retain performance better than baseline Tent. (Resuults in mloU))
Time
Round
3
All
Conditions
Rain
Snow
Fog
Night
Rain
Snow
Fog
Night
Rain
Snow
Fog
Night
Mean
Tent Best
20.1
21.3
22.3
11.3
18.5
17.2
19.5
8.4
15.8
14.5
17.5
6.8
16.1
CONTRAST
22.1
21.4
24.3
13.4
21.4
18.3
23.5
11.3
18.6
15.5
21.4
10.4
18.6
H
W
K
(w) = -E p(
Vilwe log Vitwee
(11)
hm
w== c=1
Where, (1)
Dihwee is the weighted probability output corresponding class for the pixel at location (h, w) at stam-stamp& t. We modify Equ. the main paper, while keeping the rest of the framework the same.
F.
Datasets
We use the following datasets in our experiments:
Cityscapes: Cityscapes [49] is large-scale dataset that has dense pixel-level annotations for 30 classes grouped into 8 categories (flat surfaces, humans, vehicles, constructions, objects, nature, sky. and void). There are also fog and rain variants [50, 51] of the Cityscapes dataset, where the clean images of Cityscapes have been simulated to add fog and rainy weather conditions.
ACDC: The Adverse Conditions Dataset [52] has images corresponding fog, night-time, rain, and snow weather conditions. Also, the corresponding pixel-level annotations available. The number classes the same as the evaluation classes of the Cityscapes dataset.
F.2 Experimental setup
We use Deeplab v3+ [53] with ResNet-18 encoder the segmentation model for all the experiments. We resize the input images to size of 512 x 512. Following the conventional evaluation protocol [49], we evaluate our model on 19 semantic labels without considering the void label.
We first experiment in a static target distribution setting. Specifically, we train three source models on clean, fog, and rain train splits of Cityscapes. We then evaluate the models the test set of each of the weather conditions of ACDC dataset using CONTRAST and baseline Tent models. We use a batch size of 16 and report the mean accuracy combination weights of CONTRAST with SGD optimizer using iterations. For updating the source model in CONTRAST that has the most correlation with the incoming test batch, we the Adam optimizer with a learning rate of 0.001 and updated the batch-norm parameters with one iteration. The bascline Tent models are also updated with the same optimizer and learning rate. The results in Table. 13 clearly demonstrate that CONTRAST outperforms all the baselines on test data from each of the adverse weather conditions.
all the test batches. Again, we have updated the
We also evaluate our method a dynamic test distribution setting, where we have sequentially incoming test batches from the four weather condition test sets of ACDC dataset. The test sequence includes 5 batches Rain, followed by batches of Snow, 5 batches of Fog, and finally 5 batches images) for total of 5 rounds. We report
of Night. This sequence is repeated (with the same the mean accuracy over the 5 batches and include the results for the 1st, 3rd, and 5th rounds in Table 14. We the same hyperparameters in the dynamic setting of previous experiments with the exception that the batch-size is 16.
F.3 Visualization
In Fig. 4, we present the input images along with the corresponding predicted masks of the baseline models and CONTRAST from the last round. The figure contains rows of input image samples from
22

---

the four different weather conditions of the ACDC dataset, in the order of rain, snow, fog. and night. CONTRAST is compared with baseline adaptation method Tent, and as shown in Fig. is evident that CONTRAST provides better segmentation results compared to the baselines visually.
Input
GT Mask
Tent-Best
CONTRAST
Figure 4: Visual Comparison of CONTRAST with Baselines for Semantic Segmentation Task. Each row the figure corresponds different weather condition (rain, snow, fog, and night from top to bottom). It is evident that CONTRAST outperforms the baselines in terms of segmentation results.
G Additional discussion
The v(.) function implies that trained sources should produce high-quality pseudo-labels within their own distribution. Essentially, this function evaluates the effectiveness of the model's training. For instance, model might still under- mnder-perform on the target. Observe that both the shift and the quality terms are minimized when we broaden our search space over Å. This allows us to select model that exhibits the highest correlation with the test domain, thereby providing us with the most strict bound within the discrete simplex.
if the shift between the source and target is minimal, poorly trained source
Examining the issue through the lens the gradient provides another perspective. By updating source model that is most correlated with the test data, its gradient will be smaller than those of other models. Over time, this ensures that the model's parameters remain closer the original source parameters, thereby preventing catastrophic forgetting. let's examine case mathematically of
23

---

the most correlated source can give us least gradient.
Let us assume a binary classification task with linear regression where the final activation is sigmoid (.) function. Now let's take the pseudo-label for sample x be V, where ij alw x).. Then the entropy h of y will be - (o) Then we take the derivative of the objective w.r. w weight as follows:
0 log(
0h Ow
+ log(ū))) 1)x
Now we can easily verify that if the source model closest to the test domain, then the pseudo-label generated by the model has very small entropy which also means ý is either close to 0 or close to 1. For both of the cases the derivative expression above goes close to zero which validate the claim of having smallest gradient for highest correlated source.
H KL divergence between two univariate Gaussians
During the discussion of initialization of the combination weights in Section 3.5, we come up with 8j which is calculated using the formula for KL divergence between two univariate Gaussians (M1 o and N (H2, oÿ). In this section, provide the detailed derivation of this below: From the definition of KL divergence, we know the distance between two distributions and is given by,
+00
p((x) q(x))
DKL(P,9) =
p(x) log
dr
(12)
+oo p(x) log (9(x)) dr
p(x) log (p(x)) dx
Here in this problem and q are univariate Gaussians and can be expressed follows:
& exp (- (x
m112))
exp (- (x 112)2
p(r)
((x)
{
20
(2maz)
20
Now we compute the second term in Eqn. (12) follows:
J.
100
+00
(x M22 da
p(1) log (1(x)) dr log
p(x)
(2
20
1
x 2²p(x) dx 2u2 - xp(x) dr + p2
log
(2mTaR)
20
E [2 22 [X] + 22
log
(2ma²2)
20
Var [X] + (E (x))
2 [X] -13
log
(2mrz))
20
of + oT 2/12/11 22
log
(2mraz)²
20
of + (Mi 222
log
(2
20
(13)
In similar manner calculate the first term in Eqn. (12) as follows:
24

---

+oo
(x 11)2 dx
p(x) log (p(x)) dx log
p(x)
(2mTa))²
20
- x²p(2²) dx - 2/11 - xp(x)) dx + p
log
(2To+))
20
E [X2] -22 [X] + ur
log
(2map))
20²
Var [X] [x]) 2pm [X]
1r
log
(2mT3)
20
of + M1 2p1 + pt
log
(2To
20
log
(2moç)) 2
(14)
Now combining Eqn. (14) and Eqn. (13), we get the final KL divergence as follows:
+ of + (11 22)
DxL(p.g) = log
(2TO7))
- 2
log (2TT3)
20
(15)
02
of + (w) m2)2
log
20
I Optimal step size in approximate Newton's method
In the main paper, we compute the optimal combination weights by solving the optimization below:
minimize
(()
subject to
Wj 2 (1,22 (1,
(16)
Σ W w
To solve this problem, we begin by initializing w inst as 5(-8t). Next, determine the optimal step size based on the initial combination weights to minimize the loss (() as much as possible. Specifically, we use a second-order Taylor expansion to approximate the loss at the updated point
after taking single step with a step size of (t), Thus, after one step of gradient descent, the updated point becomes:
W
(0)() Snat
= w.
inst
(t) (TwC(t) wintt
(17)
For notational simplicity let us first denote w Winit (1) (e)
(1)(1)
inst ((()) and (V cl0
V Tor Cl (l) We also denote the hessian of (t) w Hw) Now, we write the taylor series
25

---

expansion of c( at w(1) follows:
(W - (w(O) a wios (
- (w())) - aa()) (v (100 (V winn ( (a
( wro) (/20
Hwi (V (www + (((a(()33
f (wll)) (t)) (V (Wrcos L((1) ) (V Ww:o) (/(0 (al1))
(Vw(o) L2(1
(Vmro) f/(1
Hw
2
(18)
In order to minimize Cll (w(l)) we differentiate Eqn. (18) with respect to (1)) and set it zero to get (C) best Specifically,
a...)) (w)
0a(1)
=a (2)
- (v (Vrios c(1)
V (Wwcon c((0) + aaees (wioo) 2(0)
H (Vwros c((
(19)
V woo clu
V Vioros C(C
V Cw
( Vw
best
(v (Vwios Cit
( Vwozs/(tt
(vw
Hho (Vw
Horoo
(0) This is the desired expression of Deeet in Eqn. 10 in the main paper.
Note that (1) does not lie within the simplex. To ensure that the updated remains within the simplex, we project it onto simplex after each gradient step. This can be done by applying the softmax operator (Ó(.) in the main paper), which will ensure that the updated weights are normalized and satisfy the constraints of the simplex. Moreover, an ideal scenario, one would calculate the
(t) optimal step size best after each gradient step, taking into account the updated point. However, (t) for the purpose of our experiment, we calculate best only for the first step and use this value as
the learning rate for the remaining steps in order to avoid hessian caleulation repeatedly. In our
experiment, we limit the number of steps to in order to ensure quicker inference. Empirically, we have observed that using the obtained step size fixed throughout the optimization process works reasonably well.
26

---

NeurIPS Paper Checklist
The checklist is designed to encourage best practices for responsible machine learning research, addressing issues of reproducibility, transparency, research ethics, and societal impact. Do not remove the checklist: The papers not including the checklist will be desk rejected. The checklist should follow the references and follow the (optional) supplemental material. The checklist does NOT count towards the page limit.
Please read the checklist guidelines carefully for information on how to answer these questions. For each question in the checklist:
You should answer [Yes], [No]. [NA].
[NA] means either that the question is Not Applicable for that particular paper or the relevant information is Not Available.
Please provide short (1-2 sentence) justificatin right after your answer (even for NA).
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: Main claims made in the abstract and introduction accurately reflect the paper's contributions and scope.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
The paper discusses limitations of the work performed.
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point any strong assumptions and how robust the results are violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails handle technical jargon.
27

---

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairesss.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community, Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?
Answer: [Yes]
Justification:
The paper provides full set of assumtions and complete proof.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Justification: The paper fully discloses the information needed to reproduce the main experimental results.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken make their results reproducible verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often good way to accomplish this, but reproducibility instructions for how to replicate the results, access to a hosted model (e.g., in the case of large language model), releasing of a model checkpoint, other means that appropriate to the research performed.
also be provided via detailed
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
28

---

(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm.
(b) If the contribution is primarily
model architecture, the paper should descrihe the architecture clearly and fully.
(c) the contribution is a new model (e.g.. a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (c.g., with an open- open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reprodueibility may be tricky in some cases, in which authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should possible for other researchers have some path reproducing verifying the results. 5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, described in supplemental material?
Answer: [No]
Justification: The paper fully discloses all the information needed to reproduce the main experimental results.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https: httppp:://nis..ooo cc/
public/ guuides / /CodeSuomii ss Monoolicy)) for more details.
While encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https: //nips. cc/
cc/ppbll pppppulliie / des CodeSubmis ss i onPol cooocyyyy for more details.
The authors should provide instructions data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code is permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary understand the results?
Answer: [Yes]
Justification: Training details included in the supplementary material.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment Statistical Significance
29

---

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [Yes]
Justification:
The paper provides information about the statistical significance of the experi- ments.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the enror bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall
with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that out of range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification:
The paper provides sufficient information the computer resources.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported the paper (e.g., preliminary or failed experiments that didn't make it into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics https: /ttpriiur.....// cc httppublcc/ttthccccccoll......ll
Answer:
[Yes]
Justification: Yes, we confirm.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws or regulations in their jurisdiction).
30

---

10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: Included in the paper.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
The paper does not pose such risks.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification:
The data and models used properly cited.
Guidelines:
31

---

The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (e.g.. CC-BY 4.0) should be included for each asset. For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
assets are released, the license, copyright information, and terms of use the package should be provided. For popular datasets, papersw swi thcode com/ datasets s has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: The paper does not release new assets.
Guidelines:
The answer NA means that the paper does release assets.
Researchers should communicate the details the dataset/code/mooel as part their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then much detail as possible should be included in the main paper.
According to the NeurlPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification:
The paper does not involve crowdsoureing nor research with human subjects.
32

---

Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with
human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent)
may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such the institution conducting the review.
33


Reviews:
[{"review_id": "1z1cR33tA3", "reviewer": "Reviewer_YLKD", "summary": "The manuscript address continual learning in the context of adaptation to multiple data distributions. The method employs a model ensemble for unsupervised domain adaptation to the dynamically evolving distributions.\n\nThe weights denoting the contribution of each model are calculated through optimization.\nThe bounds for the optimal solution are provided.\n\nThe combination of different models weights for the model ensemble model are calculated and updated during the continual learning. Only the model that correlates the most to the new data is adapted.", "questions": "What happens if many models are correlated with the new data to be learnt?", "limitations": "Broader Impact and Limitations are discussed in Section 6 of the manuscript.", "rating": 7, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "*  The manuscript address continual learning in the context of adaptation to multiple data distributions. \n*  During the test time, multiple source models are adapted such that it optimally blends the sources\n*  The theoretical foundations for the proposed methodology are provided\n*  Extensive experimental results are provided in the manuscript and in the supplemental materials", "weaknesses": "*  Many references have missing bibliographic information. \nFor example reference [36] should be marked as being presented at MIDL 2023 and published in PMLR. Reference [3] was presented at the International Conference on Learning Representations (ICLR), 2021. Also, pages are missing in many references, including [1,2,4,5,6,7] among many others.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**W1**: Thank you for pointing these out. We will fix these references in the camera-ready version.\n\n**Q1**: When multiple source models are highly correlated and have nearly equal weights, updating all the models is an option. However, while updating all models might be effective in the short term, it can lead to an increased rate of catastrophic forgetting over time during continual adaptation. This presents a tradeoff between optimizing performance in the current batch and maintaining overall performance in the long run. Please refer to Section F.2 in the Supplementary for details.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "gXqPjXPgcn", "reviewer": "Reviewer_f99A", "summary": "The work introduces a new framework called CONTRAST, designed for dynamically combining multiple pre-trained source models during testing to adapt to changing target data distributions. For each test batch, CONTRAST learns the optimal combination weights of the source models, ensuring that the test error of the combined model does not exceed that of the best single source model. CONTRAST updates the parameters of the source models most relevant to the current test data, avoiding the forgetting of irrelevant model parameters, thereby maintaining good performance in long-term dynamic distribution adaptation. Theoretical analysis shows that CONTRAST optimizes the combination weights by balancing the distribution shift of the source models and the quality of pseudo-labels, minimizing the risk on the target distribution. Experimental results demonstrate that CONTRAST outperforms single source model adaptation methods on dynamic distribution setup.", "questions": "1. It is recommended that the authors add two baselines: one using the original TTA method with multiple source models, which can be achieved by ensembling multiple TTA-updated models. Another baseline is using the TTA method, independently updating from the original model for each task.\n\n2. It is suggested that the authors discuss the computational complexity of the learning rate selection method. The experiments in the appendix indicate that the effective learning rate range is quite large. The difference between the proposed method and the optimal fixed learning rate is only 0.4. Therefore, to evaluate the necessity of using the proposed method, it is important to consider the additional computational overhead it introduces.", "limitations": "The limitations are discussed in the corresponding section.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The method of the work is clear, intuitive, and easy to implement.\n\n2. The writing is clear, making it easy to understand and read.", "weaknesses": "1. Using multiple source models and performing ensemble is a feasible and reasonable approach to improve test-time adaptation. However, since previous techniques only utilized a single source model, this ensemble approach introduces an unfair comparison in the experiments.\n\n2. In the current task setting, dynamic test-time adaptation requires sequential model updates. However, this may not be a reasonable design or baseline. After the first update, we will have two models: the original model and the model updated on Task 1. Therefore, in practical scenarios, when a new task arrives, we wouldn't simply update the model that was updated on Task 1. Instead, we would at least try to update both the Task 1 updated model and the original model then ensemble them, or even discard the Task 1 updated model and start updating from the original model again.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**W1**: Since there are no prior works on dynamic multi-source adaptation in test time, we do not have a direct baseline for comparison. Therefore, we followed the protocol of the first multi-source Unsupervised Domain Adaptation (UDA) method ([18]), where we compared our approach with the best source model and also with any additional baseline we could create using single-source TTA methods. Additionally, we compared our method with existing multi-source UDA methods. Given that this is the first work on multi-source adaptation in test time, these comparisons represent the most reasonable baselines we could establish.\n\n**W2**: This approach could work well when the task boundaries are known. However, in our setup, we have unlabeled streaming data with no information about the task boundaries. Therefore, this method cannot be directly used in our setup (see below for more details).\n\n**Q1**: For the first baseline, we already have a comparison in Table 8 and 9 (\"All Model Update\") in the Supplementary, where we update all the models using single source TTA methods and then ensemble them with proper weights learned by CONTRAST (since the result would be worse if we naively ensembled these updated models; this also ensures fairness in the comparison). In this table, we can clearly see that updating only the most correlated model outperforms updating all models.  \n    \nFor the second baseline, we need information about the task boundaries in order to reset the models at the proper time, which we do not assume to have in our setup. This is a big assumption to make and is impractical almost always. If we make this assumption, the error rate using resetting the model is about ~3\\% lower than CONTRAST. This is not surprising because of the underlying strong assumption about knowing ground truth task boundaries. If we estimate the task boundaries, performance will fall and will be similar to or worse than CONTRAST, depending upon the estimation error. We will elaborate on these scenarios more thoroughly in the camera-ready version.\n    \n**Q2**: We calculate the Hessian for only $n$ scalar parameters, with $n$ representing the number of source models. Typically, in common application domains, addressing distribution shifts requires only a small number of source models, making the computational overhead negligible.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "t8QPscilrB", "reviewer": "Reviewer_z72Z", "summary": "This paper introduces a new task to consider continual mutli-source test-time adaptation to dynamamically evolving distributions. A framework is proposed consisting of two key step: (1) learning the combination weights and (2) identify the most correlated source model to update. In order to speed up the optimization, this paper further designs a good initialization strategy and selects the optimal step size. Extensive experiments are conducted to validate the effectiveness of the proposed model.", "questions": "Please refer to the weaknesses above.", "limitations": "The limitation is included in the paper.", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "1. Writing quality sounds good.\nThe paper is well-structured. Theoretical insights are also provided.\n2. SOTA performance.\nThe proposed framework can be integrated with various single source TTA models.\nIt can improve the performance of single source TTA models by\ncombining with the proposed method.\n3. Ablations.\nAblation experiments are provided to evaluate the impact of the proposed modules.", "weaknesses": "1. Some parts of the writing are unclear.\na) In Line 182, clarify what the t-th test batch refers to and how it is calculated. Why not compute the distance for each source model independently?\nb) In Line 270, unclear descriptions cause confusion for the reader.\nc) Algorithm 1 in the Appendix contains some typos.\n\n2. The proposed framework largely follows strategies from existing multi-source domain adaptation methods. \nFor example, the approach to learning combination weights and the use of weighted pseudo-labeling strategies are similar to those outlined in [18].\n3. The differences between challenges in the new task and multi-source DA and CTTA are not clearly described.\nThe proposed task appears similar to methods addressing multiple source model adaptation without accessing the source data. The CTTA setting typically deals with catastrophic forgetting in single source adaptation. However, multi-source adaptation inherently mitigates forgetting, as noted in Line 110. It remains unclear how the challenges of the multi-source CTTA task differ from those in multi-source DA and CTTA.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**W1**: (a) There are two variables, $\\( j \\)$ and $\\( t \\)$, where $\\( j \\)$ is the index of the source model and $\\( t \\)$ is the index of the test batch. The $\\( t \\)$-th test batch refers to the batch of data streamed at time step $\\( t \\)$. Thus, $\\( \\theta_j^t \\)$ represents the distance of the $\\( t \\)$-th test batch from the $\\( j \\)$-th source model, and these distances are calculated independently. We will clarify this further in the camera-ready version. \n(b) Here, we wanted to explain that we update each model with single source TTA methods ('X') for a test batch. The model that achieves the best performance on this batch is referred to as 'X-Best,' and the model that performs the worst is referred to as 'X-Worst.' We will improve this explanation to make it more reader-friendly.\n(c) Thanks for pointing these out. We will fix the typos.\n\n**W2**: There is a significant difference between the method in [18] and CONTRAST. The method in [18] operates in a setting where all of the target data is available during the adaptation phase and does not depend on the combination weight initialization. In contrast, our method operates under a test-time scenario where data streams in a batch by batch manner, requiring us to learn combination weights using a few samples from the current batch. Our contribution lies in properly determining these combination weights using an optimization framework, with guarantees on its convergence.\n    \n**W3**: Multi-source domain adaptation (DA) methods operate with multiple sources and require all target data beforehand during the adaptation phase. In contrast, CTTA (Continual Test-Time Adaptation) methods work with a single source and are designed for online streaming data. These methods fall into two categories: (i) those that mitigate forgetting and (ii) those that do not. Our setup combines aspects of both approaches, offering advantages from both worlds: (a) it performs better than the best source model, similar to multi-source DA methods, and (b) it operates effectively in streaming data scenarios. Additionally, our method can be  forget-free if we use the class (i) CTTA methods for model updates. Even if we use the class (ii) methods, which are computationally lightweight, our approach significantly slows down the forgetting process over the long run (Fig. 3}). We will clarify these points further in the camera-ready version.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "D2xs3q9TnO", "reviewer": "Reviewer_C1aF", "summary": "This paper introduces a novel continual multi-source adaptation method to tackle a new Test-Time Adaptation (TTA) task involving dynamic distributions.\nThe method integrates multiple source models to adapt continuously to the evolving test data distribution.\nIt efficiently computes the optimal combination weights for merging the source models and identifies which source model parameters require updating.\nAdditionally, the authors present a thorough theoretical analysis of the optimization convergence and test risk bound, supported by extensive experiments demonstrating the effectiveness of the proposed method.", "questions": "I have some questions and suggestions for the authors:\n1. I suggest that the authors provide a toy experiment by constructing a dynamic test data distribution and conducting experiments using ground truth weights to validate the effectiveness of the proposed method.\n1. According to the current test settings, the authors train four source models (vanilla, fog, snow, and frost) and test them in a continual manner. For the results in the fog, snow, and frost domains, it is expected that directly using the corresponding source model would yield the best results, i.e., X-best would be the best. However, this is not the case in the experimental results. Could the authors provide an explanation?", "limitations": "Yes", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The authors propose a novel continual multi-source adaptation method to address the TTA task with dynamic distributions, which is a challenging and practical problem.\n1. The theoretical analysis of the optimization convergence and test risk bound is thorough and well-supported.", "weaknesses": "1. The proposed method is not adequately supported by the experimental results. The authors claim that the method can adapt to dynamic test data distributions, i.e. sunshine interspersed with rain, modeled as a linear combination of source distributions. However, the authors did not construct a corresponding test dataset reflecting these conditions, but instead used a continual manner for the experiments.\n1. As a plug-in method, the proposed approach lacks experiments based on recent TTA methods. The experiments only included three methods, Tent, CoTTA and EaTA. The authors should incorporate more recent TTA methods in realistic scenarios, such as RoTTA$^{[1]}$, TRIBE$^{[2]}$, and ROID$^{[3]}$.\n\n- [1] Longhui Yuan, Binhui Xie, and Shuang Li. Robust test-time adaptation in dynamic scenarios. In CVPR, 2023.\n- [2] Yongyi Su, Xun Xu, and Kui Jia. Towards real-world test-time adaptation: Tri-net self-training with balanced normalization. In AAAI, 2024.\n- [3] Robert A Marsden, Mario Döbler, and Bin Yang. Universal test-time adaptation through weight ensembling, diversity weighting, and prior correction. In WACV, 2024.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**W1**: To demonstrate the linear combination of source distributions, we devise the following experiment. We linearly combine or blend same images from the test set of the Snow and Fog domains of CIFAR100-C using two different sets of weights. We then use CONTRAST to predict on the test set and average the combination weights obtained over all test batches. The results are presented below:\n\n**Table**: Images from the Snow and Fog domains are blended using the ground truth (GT) weights. The CONTRAST row displays the combination weights predicted by our method, which closely align with the GT weights. We also highlight the best single-source model along with its accuracy. Additionally, we report the multi-source accuracy obtained using our method, which significantly outperforms the best single-source results.\n\n|                      | Snow | Fog | Best Model | Single-source Model Acc. | Multi-source Acc. |\n|----------------------|------|-----|------------|--------------------------|------------------------|\n| **GT**               | 0.9  | 0.1 | Snow       | 68.5                     | -                      |\n| **CONTRAST**         | 0.88 | 0.12| -       | -                        | 70.9                   |\n| **GT**               | 0.1  | 0.9 | Fog        | 70.3                     | -                      |\n| **CONTRAST**         | 0.13 | 0.87| -        | -                        | 72                     |\n\nIt can be observed that the predicted combination weights across the test set are very close to the ground truth values. Additionally, the multi-source accuracy achieved is superior to the best single-source adaptation performance.\n\n**W2**: Our main goal in this paper is to demonstrate how to combine multiple source models during test time using any general single source method as our update strategy. Since our method is general enough to integrate any of the single source methods, we can also incorporate the recent methods mentioned by the reviewer into our framework. Upon using RoTTA as the update method, in the CIFAR100 $\\rightarrow$ CIFAR100C experiment, with the same experimental setup given in the paper, we get around $\\sim$2\\% reduction in error rate over the best-single source baseline. We will include results with the other methods in the camera-ready version.\n\n**Q1**: Please refer to W1.\n\n**Q2**:\nThis is a very reasonable question and we provide an explanation for the results. The goal of any multi-source method is to ensure that the performance is always equal to or better than X-Best [18]. For example, fog domain data might be most highly correlated with the fog model, but it still has some correlation, however small, with other models as well. In such a scenario, the other models also contribute positively to the performance on the target data.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
In this paper, we propose CONTRAST, a novel method for continual adaptation to dynamic streaming data using multiple source models, without requiring access to source data. CONTRAST combines these models to adapt to test data that arrive in small batches without access to the original source data. It features two main innovations: calculating optimal combination weights for continuous adaptation and updating only the source model most correlated with the test data to prevent forgetting. Theoretical insights about the performance are provided. Experiments show that CONTRAST performs as well as the best source model with hindsight and maintains robust performance as the test data distribution changes over time. As stated by the reviewers, this paper addresses a novel, challenging, and practical problem (C1aF). The writing is clear, easy to understand, and well-structured (f99A, z72Z). The method is clear, intuitive, and easy to implement (f99A). Additionally, the paper provides thorough and well-supported theoretical foundations and bounds (YLKD, z72Z, C1aF). It also presents extensive results with numerous ablations and SOTA performance (YLKD, z72Z).

Meta-review: Four expert reviewers with backgrounds in domain adaptation, test-time adaptation, and model editing side largely with acceptance (YLKD: 7, f99A: 6, z72Z: 5) and one vote for borderline rejection (C1aF: 5). The authors provide a rebuttal, globally and individually, but unfortunately only one reviewer responds. The AC has accordingly paid close attention to the points in each review and rebuttal to determine if potential reasons for rejection have been satisfactorily resolved. 

- f99A was principally concerned with the fairness of comparing multi-source adaptation against single-source adaptation and the construction of proper baselines from existing methods. The authors have provided sufficient explanation and results in the supplement to justify their choice of ensembled single-source TTA methods, but they are encouraged to prioritize how this is explained in the main text to achieve better clarity, because the doubt by this reviewer is indicative of insufficient explanation.
- z72Z was concerned with clarity, the contribution relative to multi-source DA [18], and the need to articulate the specifics of multi-source CTTA vs. multi-source DA and single-source CTTA. The rebuttal clarifies these points, and the AC encourages the authors to incorporate this into the work. Given the success of offline DA does not imply the success of online TTA, this submission makes a contribution.
- C1aF was concerned about (1) inadequate experimental design to test dynamic distributions of mixed shifts and (2) lack of recent and state-of-the-art TTA methods for more realistic settings like RoTTA, TRIBE, and ROID. The authors provide explanations and a rebuttal experiment on RoTTA that resulted in a positive re-evaluation and raised score.

Given this accounting of the rebuttal, and the raised score by C1aF, the AC sides with acceptance.

Note: The AC suggests the authors consider the following concurrent and prior work for discussion, but makes clear that these have no bearing on the decision, and are provided purely to better connect findings across the community.

- BeCoTTA https://arxiv.org/abs/2402.08712 is concurrent work on merging or *blending* models for adaptation published at ICML 2024.
- Seasoning Soups https://arxiv.org/abs/2302.10164 merges models in a few-shot supervised setting without gradient optimization by selecting weighted averages of multiple models.
- Both methods prevent forgetting by routing and merging while the latter never updates the original models themselves.
Decision: Accept (poster)