Cross-Device Collaborative Test-Time Adaptation
Authors: Guohao Chen
Shuaicheng Niu
Deyu Chen
Shuhai Zhang
Changsheng Li
Yuanqing Li
Mingkui Tan
Venue: NeurIPS 2024 poster (2024)
Keywords: Test-Time Adaptation
Out-of-distribution Generalization
Collaborative Adaptation
Primary area: other
TL;DR: Propose a test-time Collaborative Lifelong Adaptation (CoLA) paradigm to enable knowledge accumulation, sharing, and utilization across multiple devices, where both resource-abundant and resource-limited devices are included in the collaboration.

Abstract: In this paper, we propose test-time Collaborative Lifelong Adaptation (CoLA), which is a general paradigm that can be incorporated with existing advanced TTA methods to boost the adaptation performance and efficiency in a multi-device collaborative manner. Specifically, we maintain and store a set of device-shared _domain knowledge vectors_, which accumulates the knowledge learned from all devices during their lifelong adaptation process. Based on this, CoLA conducts two collaboration strategies for devices with different computational resources and latency demands. 1) Knowledge reprogramming learning strategy jointly learns new domain-specific model parameters and a reweighting term to reprogram existing shared domain knowledge vectors, termed adaptation on _principal agents_. 2) Similarity-based knowledge aggregation strategy solely aggregates the knowledge stored in shared domain vectors according to domain similarities in an optimization-free manner, termed adaptation on _follower agents_. Experiments verify that CoLA is simple but effective, which boosts the efficiency of TTA and demonstrates remarkable superiority in collaborative, lifelong, and single-domain TTA scenarios, e.g., on follower agents, we enhance accuracy by over 30\% on ImageNet-C while maintaining nearly the same efficiency as standard inference. The source code is available at https://github.com/Cascol-Chen/COLA.

Cross-Device Collaborative Test-Time Adaptation
Guohao Chen 2+ Shuaicheng N Deyu Chen' Shuhai Zhang' Changsheng Li Yuanqing Li Mingkui Tan 21 'oouth China University Technology. PPazhou Laboratory, Nanyang Technological University, "Beijing Institute of Technology.
Abstract
In this paper, we propose test-time Collaborative Lifelong Adaptation (CoLA), which general paradigm that can be incorporated with existing advanced TTA methods to boost the adaptation performance and efficiency in multi-device collaborative manner. Specifically, we maintain and store a set device-shared domain knowledge vectors, which accumulates the knowledge learned from all devices during their lifelong adaptation process. Based on this, CoLA conducts collaboration strategies for devices with different computational resources and latency demands. 1) Knowledge reprogramming learning strategy jointly learns new domain-specific model parameters and a reweighting term to reprogram existing shared domain knowledge vectors, termed adaptation on principal agents. 2) Similarity-basee knowledge aggregation strategy solely aggregates the knowledge stored in shared domain vectors according to domain similarities in an optimization- free manner, termed adaptation on follower agents. Experiments verify that CoLA is simple but effective, which boosts the efficiency of TTA and demonstrates remarkable superiority collaborative, lifelong, and single-domain TTA scenarios, egg. on follower agents, we enhance accuracy by over 30% on ImageNet-C while maintaining nearly the same efficiency as standard inference. The source code is available at httpps//github..om/CAA.
Introduction
The conventional pipeline of deep learning typically trains model and deploys it across numerous devices with frozen parameters. This pipeline has demonstrated great success in various applications. such as autonomous driving cars [35] 14], embodied robots [18], and many other smart devices 28 29 However, during deployment, the model on each device may encounter test samples drawn from a domain different from the training one. In some cases, the testing environment even changes continuously and periodically, such as changes in weather. Unfortunately, the deep model often struggles to generalize to unseen testing domains and its performance may degrade significantly.
To resolve domain shifts, test-time adaptation (TTA) 51 21 54
59
26
39
25
has emerged as promising research field. TTA updates a given model w.r.t. testing sample using entropy
self-/unsupervised objectives, such as rotation prediction 13 contrastive learning 2 30
50
minimization [54]
64
25
37,, etc. Compared to conventional domain adaptation 31
45
24 or
fine-tuning [61
33 methods that require performing offline model learning on the whole pre-collected
target dataset, TTA distinguishes itself with minimal overhead by utilizing each test sample only once for immediate post-infes isst-infreenc adaptation. This renders TTA more adaptable in real-world applications.
However, prior TTA methods mainly validate their effectiveness on a single device, i.e., re-adapting the model from scratch on each. In practice, models often deployed across multiple devices.
"Equal contribution. Email: secasper @ scut.edu.cn en, shuaicheng, niu @ ntu.edu.sg t Corresponding author. Email: mingkuitan scu.edu.cn, les@bit.edu.cn
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

Principal Agent
Follower Agent
+ TTA
Knowledge */ Different Domain Knowledge
Shared
Knowlodge
Degrade
(a) Prior single-device TTA
(b) Our collaborative TTA
Comparison w.r.t. (a) prior single- device TTA vss. (b) our collaborative TTA. Prior TTA operates on each device independently and may be infeasible in resource resource-limited devices. In contrast, our collaborative TTA allows devices to share knowledge. Based on this, on different devices, one can choose to solely aggregate the shared knowledge for TTA (Follower Agents), or further conduct backpropagation for knowledge aggregation and new domain knowledge learning (Principal Agents).
Figure I:
As shown in Figure I. in the multi-device adaptation scenario, single-device adaptation methods suffer from the following limitations. First, single-device TTA neglects useful knowledge Iearned from other devices and adapts independently. Since different devices may frequently encounter similar even identical testing domains, ignoring this shared knowledge often leads to suboptimal adaptation performance, as demonstrated in Table 2 Second, due to limited resources or latency demands, some devices may not support the backpropagation operation required by learning- based TTA methods [54] (38], rendering single-device TTA infeasible. Third, even on a single device, models may also encounter dynamically and periodically changing domain shifts. Although recent works have proposed continual TTA to mitigate the catastrophic forgetting issue, such as anti- forgetting regularizer (38 or restoration schemes (56), these methods still struggle accumulate previously learned knowledge over a long-term adaptation process, as shown in Table []
In this paper, we propose test-time Collaborative Lifelong Adaptation (CoLA) method to enable knowledge accumulation, sharing, and exploitation across devices. Our approach exploits both the previously learned knowledge from other devices and the device itself to achieve efficient and collaborative TTA. Specifically, we represent the knowledge learned on each domain of each device by domain vector, and automatically detect domain changes on each device during its continual adaptation process. These domain vectors are stored and shared across devices upon domain changes for collaborative TTA and catastrophic forgetting mitigation. Based on the shared domain vectors, we first introduce a knowledge reprogramming learning method for Principal Agents, egg., the resource- abundant devices, where we enhance TTA performance and efficiency by leveraging available shared knowledge while learning new Nomain-speecif parameters in case existing knowledge is insufficient. The newly learned parameters/knowledeee are subsequently stored for shared domain vector set updating. Furthermore, we devise an optimization- free collaborative TTA method, to reduce the computation consumption of TTA and thus enable TTA in latency- limited devices, which we term as Follower Agents in CoLA. We achieve this by directly aggregating the domain knowledge shared by principal agents according to domain similarities.
scenarios or resource-
Main novelty and contributions. 1) We introduce a novel and practical collaborative lifelong adaptation paradigm to TTA. This paradigm addresses practical demand in real-world applications to perform effective adaptation on numerous devices with varying resources and latency requirements simultaneously, meanwhile keeping privacy preserved and communication efficient. 2) We devise domain vectors to explicitly store the domain knowledge and share them across devices for collabo- rative TTA. Based on this, we devise two collaborative strategies, i.e., knowledge reprogramming principal agents and similarity- -based knowledge aggregation for follower agents. 3) Extensive experiments demonstrate the superiority of our CoLA regarding the scenarios of collaborative, lifelong, and single-domain TTA in a plug-and- and-pla manner. By leveraging available shared knowledge, on principal agents, we achieve an up to 78.0x speed up on ETA compared with the baseline without collaborative learning on ImageNet-C. On follower agents, we enhance the accuracy by over 30% while maintaining nearly the same computation and memory efficiency standard inference on ImageNet-C.
learning for resource-abundee limited or latency-sensinn

---

Shared Domain Vectors T
Follower Agents Collaborative Optimization-fres TTA
Weights d0
Weights 60, LN,
Source of
Weights 20,
Donmin
LNN,
LN,
LN,
..
:
Yix
:
Dimalla
I.N,
LN.
Domain 0
Domain
Follower Agents
Similarity basedl Agyregation
Domain Similarity
Principal Agents Collaborative Knowledge Reprogramming Learning
Firbedding
Source of
Weights A6,
Weight ae, W
Weights
T
L.N.
T
LN
Assention
:
LN,
LN
LN,
LN,
+50
Principal Agents
Frozen
Learnable Knowledge Aggregation
New Domain
Leurnable
Figure 2: An illustration of our proposed CoLA. We maintain shared domain vector set T to explicitly store the knowledge learned by each principal agent during adaptation. Based on T, for Principal Agents, we jointly learn the domain-specie via backward propagation, where the learned knowledge is then stored T. For Follower Agents, adaptively aggregate the shared knowledge in T in forward-only manner, based the domain similarities, which prioritizes knowledge derived from domains that are similar to the testing domain.
parameters 2 and the reweighting term a
2 Problem statement and motivation
Let sa(-) be the model trained on a labeled dataset Derain = [(x:,y;)} and x; ~ (xx) After training. so(-) is often deployed various devices, each device fo(:) may encounter test samples drawn from shifted and dynamically changing domain distribution (xx), where Q(x) # P(x). Under such domain shifts, deep models are often very sensitive and suffer from severe performance degradation. To address this, on each device, one can adapt fa(-) to by optimizing some self- /unsupervised learning objective at test time:
min L(x: O,, Ot), Q(x), 0:
where y and 0; denote frozen and learnable model parameters, respectively.
Motivation. Eqn. ( known single-device test-time adaptation (TTA) method, which naively re-adapts fo(-) from scratch on each device. In our multi-device adaptation scenario, this independent adaptation manner neglects the valuable knowledge learned from other devices and often obtains limited performance, as in Figure 3(a). Therefore, there is urgent demand to devise multi-device collaborative TTA methods, to enhance the adaptation performance and efficiency. To this end, the key technical challenge lies in devising a collaboration scheme that effectively exploits the knowledge from other devices while ensuring privacy preservation and communication efficiency.
Domain knowledge vectors. In real-world scenarios, a model is often deployed in environments that may change continuously and cyclically, egg., day night day. Moreover, the model deployed different devices may encounter similar environments. experiencing similar domain shifts. In such cases, when device encounters test samples drawn from domain previously other devices, is unnecessary to conduct adaptation from scratch. Instead, leveraging the previously acquired knowledge can achieve enhanced adaptation. Inspired by this, we seek to explicitly store the knowledge learned on each domain of each device, and then exploit this knowledge for collaborative TTA. We term this knowledge as domain vectors and introduce its definition below.
by itself by
domain vector ~ 0 to denote the knowledge learned on the i-th domain the j-th device. Here, 0 denotes the learned parameters the i-th domain of the j-th device, and 0f denotes the corresponding original learnable parameters. For privacy and efficiency considerations, we select the affine parameters of the norm layers learnable parameters 0 and transmit domain vectors between devices for knowledge sharing, which consumes negligible extra cost as in Table
Formally, given devices with each having domains,
We store each knowledge vector A0% in set

---

Algorithm The overall pipeline of CoLA.
Input: Test samples {D"M) mm=1 where D== > 1t=1 denotes the batches of test samples the m-th device, the model (-) and its stem (first) layer fe, (-), threshold 2. Initialize: shared domain vectors (= Φ==0 for each device.
2: for t = 1, ,,..., do
3: for each device (in parallel) do
4: Calculate batch statistics é; over fo. (xx²
5: Update distribution statistics da by Eqn.
6
6: For Principal Agent:
IT knowledge reprogramming learning, Sect. 3.7
7:
if D(os; 6e) > then
domain change detection, Eqm. Update domain vectors T by storing the newly learned 2 and reset Qa = 0.
8:
9:
end
10:
Predict 91"" by (x!") based on Eqn. (2)
11:
Update a and AO via Eqn. (2) with back packpropagttion:
12:
For Follower Agent:
similarity-based aggregation, Sect. 3.2
13:
Update Dj for different domain knowledge via Eqn. {
Predict dt." by fx(" based on Eqn. (3)
15:
end for
end for
Output: Predictions (11th) |t I,... T and m = I,..,,M... M}.
T= n.m /=l.j=1- During the continual adaptation process, we dynamically expand T by storing new AO in T once A0 is learned. For simplicity of presentation, we omit T as (A0.) where N and exploit T to devise collaborative TTA strategies the following sections.
Cross-device collaborative test-time adaptation
In this paper, we propose test-time Collaborative Lifelong Adaptation (CoLA) method. In CoLA, we seek to conduct collaborative TTA across multiple devices by exploiting set of shared domain vectors, termed T. We automatically detect the domain changes on each device during a continual adaptation process and explicitly store the current domain knowledge in T once the testing domain changed (c.f. Sect. (.3). Then, based on T, we develop two distinct collaboration strategies. In practice, users can determine which strategy to use according to the computational resource of their device or their latency requirements. First, the collaborative knowledge reprogramming learning strategy (c.f. Sect. [.I] is designed for "principal agents", i.c., the devices that will dominate the learning of new knowledge and have sufficient resources for backpropagation-ben based model updates. This strategy jointly learns new domain-specific model parameters and a reweighting term to reprogram the knowledge learned from previously encountered distributions from both the device itself and other devices, through backpropagatton-basse based optimization. Second, the optimization-fre free collaborative TTA (c.f. Sect. (.2)) is designed for "follower agents", i.e., the devices that are resource- limited latency-sensitive. This strategy mainly exploits the knowledge shared from principal agents, by aggregating the valuable shared knowledge according to distribution similarities. We summarize the pseudo-code in Algorithm I and illustrate the overall pipeline of CoLA in Figure 2
3.1 Collaborative test-time adaptation via knowledge reprograming learning
We conduct collaborative adaptation with both the knowledge learned from other devices and the device itself. This latter one is particularly advantageous for lifelong adaptation since, in practice, a deployed model may encounter diverse and evolving domains. By storing and reprogramming all knowledge learned from previously encountered domains, we naturally mitigate the issue of catastrophic forgetting during lifelong TTA (see results and analysis in Table []. To this end, on each device, we detect domain changes and store the learned model parameters for each specific domain once the test domain changes. We then reprogram the knowledge stored in these model parameters by reweighting as below, facilitating scamless adaptation to changing environments.

---

As aforementioned, we assume there exist N sets of parameters, i.e., domain vectors, learned from previously encountered domains across all devices, denoted as (=0 ] where - 0g. For flexibility, we denote null knowledge as 20,= and include it in T. We use T to illustrate our collaborative learning scheme here and put details for detecting and storing each A0; in Sect. 3.3] Based on T. we learn a reweighting term that adaptively aggregates shared knowledge via backpropagation, while learning new knowledge simultancously if existing knowledge is insufficient. The overall optimization problem is given by:
min L(x; 0y, where 0 0g + Σ a, + AO and AO, E T = o,A0
(2)
Here, Ao denotes learnable parameters for current round of adaptation, and a denotes the normalized weights for different domain knowledge, i.e. Σ, a; = Thus, knowledge reprogramming and new knowledge learning are decoupled into the optimization of a and A0. Note that we introduce Adg to ensure more flexibility in reprogramming, e.g., by setting 0gg = 1, one can entirely disregard previously learned knowledge when it is not beneficial for adapting to currently encountered domain. Moreover, when no knowledge has been accumulated, i.e., T {0). Eqn. (2) simplifies to the conventional TTA. We put details of the initialization for a and T in Appendix B due to page limits.
Adaptive temperature scaling for fast adaptation. Promptly re- weighting the appropriate knowl- edge for aggregation is key to fast adaptation when the testing distribution suddenly changes. However, this process hindered when the logits of are sharpened. making re-weighting difficult favor another knowledge. To mitigate this, we further introduce an adaptive scaling temperature during the optimization phase, which helps adjust the sharpness of a adaptively while maintaining the smoothness of the original logits. The calculation of a can be thus expressed as a softmax(B.17 where , is the logit vector and Ti a learnable cempeeree temperature.
3.2 Collaborative test-time adaptation via similarity- -based knowledge aggregation
The computational constraints of devices, combined with the real-time demands of various applica- tions, often necessitate TTA be as efficient as possible. To this end, we propose a forward-only collaborative TTA strategy for devices operating in the follower agent mode. Here, the follower agent adapts given model by aggregating the previously learned knowledge and the knowledge shared from principal agents without learning new Nomain-specifie parameters, aiming to maintain almost the same efficiency as pure inference. Formally, given domain vectors T=(A6,] } shared from m principal resource-abundan devices with each having n encountered domains and A O, the goal of adaptation is to find appropriate normalized weights 7 ER such that:
N
>" = arg min L(x; 0y, 0r), where 0, = 0f + Σ 1=0
AA; and A0, ET= (A0,) (3)
7
Here, since backpropagation- based learning is not supported, we directly assign the specific value of each "/ approximately according to distribution similarities. We estimate the distribution by calculating the feature statisties, i.e., the mean and standard deviation of the features from the first stem layer. Formally, let Φu denote the statistics of the current distribution that is online estimated via Eqn. (6). and de be the distribution statistics on which A0, is adapted (i.e., the corresponding da while learning AO,), we re-wetght the knowledge from different distributions by:
Y = softmax(p), where p. = 1 (D(da, Φ.) +e).
Here, p is a logit vector, D(..)) is a distance for which we adopt KL divergence as in Eqn. 7 and e is a small constant for numerical stability. In this way, learned from distributions that are similar to the current distribution. Note that principal agent with abundant resources may also leverage Eqn. (3) for efficient TTA in real-time scenarios.
adaptively prioritize shared knowledge
Exploiting diverse knowledge for aggregation. Aggregating the advantages of different knowledge is the key to achieving satisfying robustness under various distribution shifts, as shown in (b). However, when distributions are highly similar, the re-weighting logit μ; shall become sufficiently large (e.g., Pi > 10) and the softmax function tends to simplify to the max function, which hinders the potential of Eqn. (3) from aggregating a diverse set of knowledge. To alleviate this, we further in- troduce pre-defined temperature scaling factor Tr to soften pt, thereby encouraging the aggregation
5

---

of more existing knowledge. Then, P: is re-defined as:
Pi 1/ (T₃ D(os, de) +e).
Remark, It is worth noting that our CoLA can be incorporated with existing TTA techniques as a plug-and-play module for a more effective solution (as in Table I aad Table Furthermore, unlike prior methods [4] that impose intensive transmission of both data and model weights. CoLA offers several merits for real-world implementation: 1) CoLA involves only the transmission of learned parameters ~0, which preserves user privacy and imposes much less communication burden (e.g.. the affine parameters of the norm layers ViT-Base [8] which are typically updated in TTA [54] 38), occupy only 0.15 MB). 2) the domain vectors are preserved and shared intermittently with a shift detector, which further reduces the communication burden by a considerable margin. 3) CoLA is decentralized and flexible, which allows all agents to join or leave the collaboration at any time.
3.3 Automatic domain shift detection for constructing domain knowledge vectors? T
In this section, we introduce the construction of the domain knowledge vectors T shared across multiple devices. As aforementioned, explicitly accumulating/storing the learned knowledge from each domain of each device in T is a key step in our CoLA for collaborative learning. However, in practice, during the lifelong adaptation process of each device, we do not have any prior information on the domain labels regarding given test sample stream. To conquer this, we devise an efficient distribution shift detector to identify whether the test distribution changes, and then automatically store the currently learned model weights to the domain vector set T once the domain change is detected. We achieve this by measuring the discrepancy between the distribution's 's statistics Φa and the statistics of the current test batch 6r. Formally, let fe, (-) be the stem layer of fo(-), i.e. the first layer. &e comprises the mean fú and standard deviation on calculated over fo, (xx). Then, estimate the statistics du from the observed test samples [x,]== via exponential moving average:
Φd Xoi + (1 A)oa.
where A is moving average factor belongs to [0. 1]. Inspired by existing distance-based detection methods [19] we capture the magnitude of distribution shifts by distance function D(:-)) as follows.
H
D(da, bi)
Σ
KL(0il)02) = 20²
(o + (11 - p2)3), (7)
H
where denotes the dimension of statistics, and KT((II-) is the Kl.-divergence simplified from Here, a distribution shift is detected when D( d.) > 2, where 2 is a pre-defined threshold. This simple design offers several merits: 1) It imposes minimal computational and memory costs without necessitating data preservation. 2) By leveraging the features from the stem layer, we can promptly detect and respond to distribution shifts, rendering it well-suited for the online nature of TTA.
4 Experiments
Datasets and models. We conduct experiments on the ImageNet-lk (6), well as five benchmarks for OOD generalization, i.e., ImageNet-C (16) (contains corrupted images in 15 types of 4 main categories and each type has 5 severity levels), ImageNet-R (various artistic renditions of 200 ImageNet classes) [15, ImageNet-Sketch (55), ImageNet-A [17] and ImageNet-V2 V2 [44]. We use ViT-Base [8] as the source model unless stated otherwise. The model is trained on the source ImageNet- IK| 6 training set and the model weights are obtained from the timm repository [60].
Compared methods and implementation details. We compared our proposed CoLA with 1) Backpropagattio -based methods: CoTTA (56), ETA [38], EATA (38], SAR 39), and DeYO 25).
2) Backpropagation free methods: LAME 3 and T3A (2I). For Eqn. (2) We directly leverage the learnable test-time objectives from the integrated TTA methods. A is optimized by following the update rules of the integrated baseline as listed in Appendix C a is updated via the AdamW optimizer with learning rate of 0. The shift detection threshold is set to 0. For follower agents, we consistently set T in Eqn. 33 to 5 for all experiments. More details are put in Appendix Aaand

---

Table 1: Comparison ImageNet-C (level 5) regarding Accuracy (%) under lifelong adaptation for 10 rounds, in total of 150 corruptions, on a single principal resource-abudaat device. We report the average aceuracy of 15 corruptions in each round here and put more results in Appendix D
Time:
Round
2
3
4
5
6
7
8
9
10
Average
NoAdapt
29.9
29.9
29.9
29.9
29.9
29.9
29.9
29.9
29.9
29.9
29,9
COTTA [56
44.9
40.6
35.8
32.7
30.4
28.9
27.7
27.1
27.2
26.5
32.2
EATA 33
60.4
60.0
59.6
59.4
59.3
59.1
59.0
58.8
58.7
58.6
59.3
SAR 39
59.1
60.6
60.9
61.2
61.3
61.4
58.3
60.4
60.8
61.1
60.5
CoLA (Ours)
59.1
62.4
63.6
64.3
64.7
64.9
65.2
65.1
65.3
65.4
64.0(43.55)
ETA 38
61.4
58.7
54.5
50.2
46.2
44.1
38.8
38.0
36.7
35.1
46.4
CoLA (Ours)
62.0
63.9
64.8
65.1
65.3
65.3
65.3
65.3
65.4
65.4
64.8 (41.44
DeYO 25
59.8
48.8
0.1
0.1
0.1
0.1
0.1
0.1
0.1
0.1
10.9
+ CoLA (Ours)
61.7
62.5
63.6
64.5
65.0
65.1
65.3
65.5
65.5
65.5
64.4(463.5)
Table 2: Effectiveness under collaborative adaptation across principal resource-abundant devices w.r.t. Accuracy ( (%). Results are evaluated on ImageNet-C. (level 5, containing 15 corruption types
of 4 groups). We share learned weights across devices post-adaptation each group of corruptions.
Device (Adapt +)
Device 2 (Adapt -)
Device 3 (Adapt -))
Method
Noise
Blur
Weat.
Digit.
Blur
Noise
Digit.
Weat.
Weat.
Digit.
Blur
Noise
Avg.
NoAdapt
8.2
28.4
36.1
41.7
28.4
8.2
41.7
36.1
36.1
41.7
28.4
8.2
28.6
CoTTA [56
28.9
41.3
50.2
47.6
36.3
37.1
50.4
52.7
50.4
55.0
42.5
38.7
44.2
EATA 38
53.5
57.0
68.1
67.2
58.1
52.2
67.0
68.5
69.4
67.7
57.9
51.5
61.5
SAR (3
50.4
54.4
66.3
64.5
55.1
48.3
64.0
66.4
66.5
64.3
55.4
47.7
58.6
+ CoLA (Ours)
50.4
58.0
69.4
68.7
55.0
55.0
67.1
70.5
66.3
65.3
58.8
55.5
61.7
ETA [38
55.2
56.9
67.5
66.0
59.8
51.7
65.0
67.4
70.3
67.8
58.0
49.4
61.2
+ CoLA (Ours)
55.2
60.0
70.9
69.3
59.5
56.3
68.8
70.9
70.2
68.1
59.7
55.3
63.7
DeYO 25]
56.3
49.9
68.1
67.8
55.6
46.7
67.2
69.0
71.1
68.8
51.1
4.3
56.3
+ CoLA (Ours)
56.2
55.1
71.2
70.2
54.8
54.5
70.0
71.5
71.0
69.0
53.7
54.3
62.6
4.1
Comparison with state-of-the-art methods
Results under lifelong test-time adaptation. We evaluate the long-term effectiveness of our CoLA on ImageNet-C [44) within challenging lifelong TTA scenario where the model is online adapted to 15 corruptions over 10 rounds (total 150 corruptions), and the parameters will never be reset. We put more details of the experimental protocol in Appendix due to page limits. From Table [ we derive the following observations.
1) Our CoLA achieves new state-of-the-art results on the first round, last round, and the average of adaptation, suggesting our superiority. 2) Most methods, including CoTTA (56] and EATA (38) with an anti-forgetting strategy, experience performance degradation as the number of adaptation rounds increases (e.g., ETA's performance degrades from 61.4% to 35.1% on the average accuracy), indicating the difficulty of the evaluated scenario. 3) By integrating our CoLA with existing methods, we enhance the performance steadily with more adaptations, demonstrating our effectiveness in accumulating and exploiting learned knowledge for long-term adaptation. 4) Although EATA mitigates performance degradation by introducing anti-forgetting regularization, it suffers from the stubilit--lastiiily trade-off, i.e., the average accuracy drops from 61.4% (ETA) to 60.4% (EATA) in the first round. In contrast, our Col.A enhances ETA's performance even at the first round of adaptation, i.e., 61.4% (ETA) vs. 62.0% (ETA+CoLA), indicating that CoLA does not limit the learning ability. The sensitivity analyses on threshold are provided in Appendix
B
Results under collaborative test-time adaptation. To evaluate our CoLA under the collaborative TTA scenario, we first assess its performance across multiple principal absuurce-abundant devices. From Table our CoLA outperforms the integrated baseline from the adaptation to the second group of corruptions, e.g., the accuracy of 58.0% (SAR+CoLA) vs. 54.4% (SAR) on 'Blur' in Devices 1. Moreover, this improvement becomes increasingly more pronounced as more knowledge is shared across devices, e.g., improving the accuracy from 47.7% (SAR) to 55.5% (SAR+CoLA)) on "Noise" in Device 3. This demonstrates our effectiveness facilitating knowledge sharing and exploitation across principal devices via our knowledge reprogramming learning scheme, i.e., with Eqn. 2
7

---

Table 3: Comparison on ImageNet-C (level 5) regarding Accuracy (%) under lifelong adaptation
on resource-limited follower devices. T3A* resets the model after adaptation on each Corruption.
CoLA exploits the learned weights from Table 2
(e.g. ETA + CoLAA at the 7-th row) for Eqn. 3
Noise
Blur
Weather
Digital
Method
Gaus. Shot Imp. Def. Glass
Mot.
Zoom Snow Frost Fog Brit. Contr.
Elas.
Pix. JPEG
Avg.
NoAdapt
9.5
6.7
8.2
29.0
23.4
33.9
27.1
15.9
26.5 47.2 54.7
44.1
30.5 44,5
47.8
29.9
T3A 120
9.5
7.0
8.7
23.3
23.3
31.2
25.9
11.9
24.2
44,0
52.2
41.0
30.1
43.0
47.0
28.2
T3A* 12
9.5
6.5
8.1
29.8
24.1
34.3
28.2
16.0
26.9
49.0
55.5
44.5
33.1
44,5
48.2
30.5
LAME 33
9.3
6.5
8.0
28.6
23.0
33.3
26.6
15.2
26.0
45,9
54.1
43.6
29.3
44.0
47.4
29.4
CoLA (SAR)
55.2
56.0
56.8
57.3
49.1
59.9
58.5
65.8
65.8
72.2
77.1
66.2
65.9
72.2
69.4
63.2
CoLA (ETA)
55.7
57.3
56.9
58.5
46.2
59.4
63,4
69.1
66.5
73.1
77.6
66.3
69.2
73.1
69.9
64.1
CoLA (DeYO)
56.6
57.7
57.5
58.2
47.7
55.5
39.0
69.6
67.2
73.5
78.0
67.0
70.4
73.5
70.3
62.8
Table 4: Comparison under single-domain TTA (on one principal device) w.r.t. Ace (%). Results are averaged over 15 corruptions on ImageNet-C (level 5). L.S denotes label distribution shifts, M.S denotes mixed domain shifts per SAR [39.
Table 5: Comparison w.r.t. wall-clock time and memory on ImageNet-C (Gaussian, level 5) on an A100 GPU. C/R refers to accuracy on ImageNet-C/R. BP is short for back- pack-propggttion. CoLA utilizes weights of ETTA+ in Table
Method
Mild
L.S.
M.S
Avg.
Method
BP
C
R
[T. (s)
Mem. (MB)
NoAdapt
29.9
29.9
29.9
29.9
NoAdapt
X
9.5
43.1
50
816.6
SAR (3
54.5
56.7
57.1
56.1
T3A 21
X
9.5
42.1
158
909.9
CooLA (Ours)
57.7
58.5
58.0
58.1
CoLA (Eqn.
X
55.7
51.5
51
821.9
ETA 38
63.3
47.6
57.4
56.1
HATA 38
49.5
56.8
113
7439.3
+ CoLA (Ours)
64.4
55.2
58.3
59.3
SAR 39
44.0
51.8
202
7429.9
DeYO 25
64.1
61.3
59.1
61.5
ETA
38
51.9
57.5
109
7429.6
CoLA (Ours)
64.7
63.5
59.3
62.5
CoLA (Eqn.
54.3 59.0
112
7435.3
Given learned knowledge from resource-abundant principal devices (totaling 34 weights occupying 5.0 MB), we further evaluate the effectiveness of CoLA on resource-limited follower devices. From Table 3 we observe that existing TTA methods struggle to improve the performance of the source model without model updates, highlighting the urgent need for a more effective solution. In contrast, by exploiting shared knowledge adaptively in forward-only manner with Eqn. B CoLA achieves substantial performance gain, e.g., enhancing the average accuracy from 29.9% to 64. 1% in CoLA (ETA). Note that we also verify CoLA's sample efficiency well as its computation and memory efficiency in Figure B (a) and Table [5] These results collectively underscore the importance of cross-device collaboration and our effectiveness regarding the scenario of collaborative TTA.
Results under single-domain test-time adaptation. Following Yo 25 we validate our CoLA in both the wild scenario (i.e., imbalanced label distribution shifts and mixture of distribution shifts) and the mild scenario of single-domain TTA, where the model is reset post-adaptation to each corruption. Here, Col.A saves learned weights for every adaptation to 10 batches of samples while maintaining a maximum of 32 weights (totaling 4.7 MB) by discarding the unused ones according to ag-
From Table 4 within all evaluated scenarios, incorporating CoLA consistently improves the perfor- mance by considerable margin (e.g., +2.0% on SAR w.r.t. overall average accuracy). Interestingly, the enhancement from CoLA may even help surpass a stronger baseline, e.g., the average accuracy of 64.4% (ETA+CoLA) vs. 64. 1 YO) on the mild scenario, demonstrating our effectiveness. This improvement mainly stems from our ability to alleviate егrоr accumulation. Given multiple saved weights, instead of naively selecting the newest weight that may have adapted to noise, CoLA dynamically favors the more optimal one via loss optimization. This renders Col.A more robust to scenarios where perturbations may occur. We also visualize o; Appendix F to offer more insights.
4.2 Ablation studies and more discussions
Effectiveness of Ti on sample efficiency in Eqn. (2). Sample efficiency is particularly important in scenarios where the availability of target data is limited or early adaptation performance is paramount. As shown in Figure 3 (a). by leveraging the learned knowledge from other devices, ETA+CoLLA achieves an up to 78.0x speed up compared with ETA, i.e., 51.7% accuracy with 640 samples (ETA+CoLA) 51.37% accuracy with 49,920 samples (ETA), demonstrating the importance
8

---

(a) Effects of T in Eqn. (2)
(b) Effects of Ty in Eqn. (5)
78,0x Speedup
Accuracy (%6
50
Accuracy (%)
40
30
40
ETA CoLA (Ours, WD
NoAdapt (S)
NoAdapt (R)
(Ours)
Our
Our CoLA (R)
10
1k
10k
20k
30k
40k
50k
inf
#Online Samples
The Value of Temperature Tr
Figure 3: Ablation study of CoLA. In left, compare the sample efficiency Device2 in Table 2 Gaussian. Model's accuracy is recorded on the entire test set after adapting to N test samples. In right, we evaluate the effectiveness of y on seen (i.e., ImageNet-C, Gaussian) and unseen distributions (i.e., ImageNet-R/Sketch), where CoLA exploits weights of ETA+CoLA in Table 2 for Eqn.
Table 6: Effectiveness of CoLA (Eqn. 2
Table 7: Effectiveness of CoLAA on prompt tuning. Re- sults reported on ImageNet and its variants with CLIP- RN50 43). CoLA exploits hard prompts for Eqn.
unseen distributions with weights
learned on ImageNet-C from Table
Method
R
Avg.
Method
A
V2
Avg.
SAR 39
51.9
33.8
42.8
NoAdapt Ensemble
58.2
21.8
51.4
56.2
33.4
44.2
+ CoLA (Ours)
55.1
39.3
47.2
59.8
23.2
52.9
60.7
35.5
46.4
ETA 38
57.7
43.1
50.4
TPT [47
60.7
26.7
54.7
59.1
35.1
47.3
CooLA (Ours)
59.0
43.2
51.1
CoLA (Ours)
62.2
28.0
55.4
60.4
34.7
48.1
of cross-device collaboration and our effectiveness to facilitate knowledge sharing and utilization. Moreover, upon integrating Ti in Eqn. (2). CoLA demonstrates superiority without necessitating too many test samples, i.e., 53.2% accuracy (ETA+CoLA) vs. 22.2% accuracy (ETA+CoLA w.o. Ti) with 960 samples, suggesting the effectivencss of 7i to promote swift adaptation under distribution shifts. We also provide more comparison in Appendix H regarding sample efficiency unseen distributions.
Effectiveness of T on robustness in Eqn. We evaluate the robustness of Eqn. (3) on seen/unseen distributions (i.e., whether resource- abundant devices have encountered the evaluated distribution). From Figure (b), our CoLA consistently outperforms NoAdapt regardless of Ty, demonstrating our effectiveness. On seen distribution, our improvement is particularly significant while the performance insensitive to T in a reasonable range. However, when Ty is set to infinity (i.e., averaging different weights), performance experiences degradation appropriate knowledge plays less significant role. On unseen distribution (e.g., ImageNet-R), aggregating minority of knowledge may be insufficient to address distribution shifts. In this case, T', plays important role in enhancing robustness by aggregating the strengths of diverse knowledge. We fix Ty to 5 in experiments without careful tuning.
Memory and computation consumption of CoLA. Besides achieving strong performance across various scenarios, we demonstrate that CoLA incurs negligible computation and memory costs. From Table 5 on resource-limited follower devices, CoLA (Eqn. 3 substantially outperforms T3A in terms of accuracy and runtime memory (i.e., 55.7% vs. 9.5%, and 821.9 MB vs. 909.9 MB) while maintaining nearly the same efficiency as NoAdapt On resource-abundant principal devices, CoLA enhances the performance of ETA by a considerable margin (i.e., +2.4% on ImageNet-C) while incurring only 5.7 MB of additional runtime memory and 3s of latency, indicating that CoLA is even lighter than the regularizer introduced EATA. Note that CoLA (Eqn. 3 determines appropriate re- weighting for different knowledge with batch of test samples, outperlorming ETA+CoLA w.r.t average accuracy on ImageNet-C. This underscores the potential of collaborative TTA in real world, where device may adapt effectively using only negligible costs given adequate shared knowledge. We also show that CoLA can efficiently scale to over 10,000 domain vectors in Appendix E
Generalization of Eqn. 2 on unseen distributions. Following Figure 3 (b), we further validate the
effectiveness of CoLA (Eqn. 2 on unseen distributions. From Table 6 performance unseen distributions, yielding notable 4.4% improvement SAR in terms of
CoLAA consistently enhances

---

average accuracy. Such improvement can be attributed to the transferability of knowledge learned from similar domains [63]. These findings collectively indicate that the effectiveness of CoLA not limited to previously encountered distributions on both principal agents and follower agents.
Prompt tuning with CoLA using multiple hard prompts. Besides aggregating learned knowledge from other devices, we demonstrate that CoLA can also benefit from aggregating diverse knowledge from humans (i.e., manual -desineed hard prompts), As shown in Table 7 compared with TPT which is limited to leveraging a single hard prompt, CoLA enhances the adaptation performance on 4 out of datasets (e.g.. +1.5% on ImageNet w.r.t. accuracy). These results collectively indicate that CoLA effectively exploits both the knowledge of humans and the knowledge from optimization, which may bring new perspectives the design of learning algorithms when introducing diverse human prior knowledge is beneficial, e.g., chain-of-thone of-thoughts [58]. All used prompts are listed in Appendix C
Differences and advantages over federated TTA 1 23). The main difference is that our CoLA conducts collaborative learning at the testing phase, whereas federated TTA conducts collaborative learning during federated source training. For instance, FedTHE+ [23] federatedly trains global and a local personalized model for each client, then adaptively ensemble their outputs at test time. ATP M federatedly learns module-specific adaptation rates across clients during training for test-time adaptation. However, these methods still conduct TTA independently on each devices during testing, and thus inherits the limitation of the single-device TTA methods. Moreover, in federated TTA I 23]. the training phase and test-time adaptation phase are highly correlated, which means they can only use their own federated trainned models during TTA. This makes these methods restricted for real-world applications. In contrast, our CoLA enhances test-time model adaptation performance and efficiency by leveraging knowledge from multiple devices in the application environment, which essentially establishes new unsupervised on-time TTA paradigm. Moreover, our CoLA paradigm can be applied to any pre-trained models, and thus offers much better flexibility in deployment. Additional comparisons with FedAvg (32 for collaborative adaptation are also provided in Appendix E
Conclusion
In this paper, we propose a multi-device Collaborative Lifelong Adaptation (CoLA) paradigm for test-time adaptation (TTA), which addresses practical scenario where multiple devices with different computational resources and latency requirements need to perform TTA simultaneously. In particular, we first accumulate set of shared domain knowledge vectors with an efficient domain shift detector. Based on this, we develop knowledge reprogramming learning strategy on principal agents, which leverages backpropagation-bing baased optimization aggregate existing knowledge while learning new domain-specific parameters simultaneously. To further improve adaptation efficiency, we introduce an optimization- free TTA strategy on follower agents, which solely aggregates the shared domain vectors based on domain similarity. In CoLA, all devices/agents work collaboratively while keeping privacy preserved and communication efficient. Experiments verify that CoLA boosts the performance and efficiency of existing TTA solutions in collaborative, lifelong, and single-domain TTA scenarios.
Acknowledgments
This work was partially supported by National Natural Science Foundation of China (NSFC) 62072190 and TCL. Science and Technology Innovation Fund. The authors thank Jinwu Hu and Yu Hu for discussions on domain knowledge reprogramming, and Yaofo Chen for consultations on cloud-edge test-time adaptation.
References
[1] W. Bao, Wei, H. Wang, and He. Memo: Test time robustness via adaptation and augmenta- tion. In Advances in Neural Information Processing Systems, volume 36, 2024.
[2] A. Bartler, A. Bühler, F. Wiewel, M. Döbler, and B. Yang. Mt3: Meta test-time training for self-supervised test-time adaption. In International Conference on Artificial Intelligence and Statistics, pages 3080-3090. PMLR, 2022.
10

---

[3] M. Boudiaf, R. Mueller, I. Ben Ayed, and L. Bertinetto. Parameter- free online test-time adaptation. In IEEE Conference on Computer Vision and Pattern Recognition, pages 8344- 8353, 2022.
[4] Y. Chen, S. Niu, S. Xu. H. Song, Y. Wang, and M. Tan. Towards robust and efficient cloud-edge clastic model adaptation via selective entropy distillation. In International Conference on Learning Representations, 2024.
[5] Z. Chen, J. Chen, Z. Xie, E. Xu, Y. Feng, and S. Liu. Multi-expert attention network with unsupervised aggregation for long-tailed fault diagnosis under speed variation. Knowledge- Based Systems, 252:109399 2022.
[6] J. Deng, W. Dong, R. Socher, L.-J. K. Li, and L. Fei-Fei. Imagenet: A large-scale hicrarchical image database. In IEEE Conference on Computer Vision and Pattern Recognition, pages 248- 255, 2009.
[7] Z. Deng, Z. Chen, S. Niu, T. Li, B. Zhuang, and M. Tan. Efficient test-time adaptation for super-resolution with second-order degradation and reconstruction. In Advances in Neural Information Processing Systems, 2023.
[8] A. Dosovitskiy, L.. Beyer, A. Kolesnikov, D. Weissenborn, X. Zhai, T. Unterthiner, M. Dehghani, M. Minderer, G. Heigold, S. Gelly, Uszkoreit, and N. Houlsby. An image is worth 16x16 words: Transformers for image recognition at scale. In International Conference on Learning Representations, 2021.
[9] Y. Du, S. Luo, Y. Xin, M. Chen, S. Feng, M. Zhang, and C. Wang. Multi-source fully test-time adaptation. Neural Networks, page 106661, 2024.
[10] F. Fleuret et al. Test time adaptation through perturbation robustness. In Advances in Neural Information Processing Systems Workshop, 2021.
[11] Y. Gandelsman, Y. Sun, X. Chen, and A. Efros. Test-time training with masked autoencoders. In Advances in Neural Information Processing Systems, volume 35, pages 29374-29385, 2022.
[12] Gao, J. Zhang, X. Liu, F. Shelhamer, T. Darrell, and D. Wang. Back to the source: Diffusion- driven test-time adaptation. In IEEE Conference on Computer Vision and Pattern Recognition, 2023.
[13] S. Gidaris, P. Singh, and N. Komodakis. Unsupervised representation learning by predicting image rotations. In International Conference on Learning Representations, 2018.
[14] S. Grigorescu, B. Trasnea, T. Cocias, and G. Macesanu. A survey of deep learning techniques for autonomous driving. Journal of field robotics, 37(3):362-33)) 2020.
[15] D. Hendrycks, S. Basart, N. Mu, S. Kadavath, F. Wang. E. Dorundo, Desai, T. Zhu, S. Parajuli, M. Guo, et al. The many faces of robustness: A critical analysis of out-of-distributiin general- ization. In JEEE Conference on Computer Vision and Pattern Recognition, pages 2021.
[16] D. Hendrycks and T. Dietterich. Benchmarking neural network robustness to common corrup- tions and perturbations. In International Conference on Learning Representations, 2019.
[17] D. Hendrycks, K. Zhao, S. Basart, Steinhardt, and D. Song. Natural adversarial examless. In IEEE Conference on Computer Vision and Pattern Recognition, pages 15262-15271, 2021.
[18] Y. J. Heo, D. Kim, W. Lee, H. Kim, J. Park, and W. K. Chung. Collision detection for industrial collaborative robots: A deep learning approach. IEEE Robotics and Automation Letters, 4(2):740-746, 2019.
[19] J. Hong, L. Lyu, J. Zhou, and M. Spranger. MECTA: Memory-economic continual test-time model adaptation. In International Conference on Learning Representations, 2023.
[20] E. J. Hu, Y. Shen, P. Wallis, Z. Allen-Zhu, Y. Li, S. Wang, L. Wang. and W. Chen. Lora: Low-rank adaptation of large language models. In International Conference on Learning Representations, 2022.
[21] Y. Iwasawa and Y. Matsuo. Test-time classifier adjustment module for model-agnostic domain generalization. In Advances in Neural Information Processing Systems, volume 34, 2021.
[22] I.. Jiang and T. Lin. Test-time robust personalization for federated learning. In International Conference Learning Representations, 2022.

---

[23] L. Jiang and T. Lin. Test- time robust personalization for federated learning. In International Conference on Learning Representations, 2023.
[24] J. N. Kundu, N. Venkat, R. V. Babu, et al. Universal source- source-free domain adaptation. In IEEE Conference on Computer Vision and Pattern Recognition, pages 4544,4553, 2020.
[25] J. Lee, D. Jung, S. Lee, J. Park, J. Shin, U. Hwang, and S. Yoon. Entropy is not enough for test-time adaptation: From the perspective of disentangled factors. In International Conference on Learning Representations, 2024.
[26] J. Liang, R. He, and T. Tan. A comprehensive survey test-time adaptation under distribution shifts. International Journal of Computer Vision, pages 1-34, 2024.
[27] J. Liang, D. Hu, and J. Feng. Do we really need to access the source data? source hypothesis transfer for unsupervised domain adaptation. In International Conference on Machine Learning, pages 6028-6033 2020.
[28] Y. Liang, Z. Cai. J. Yu, Q. Han, and Y. Li. Deep learning based inference of private information using embedded sensors in smart devices. IEEE Network, 32(4):8 14, 2018.
[29] J. Lin, W.M. Chen, Y. Lin, C. Gan, S. Han, al. Mcunet: Tiny deep learning on iot devices. In Advances in Neural Information Processing Systems, volume 33, pages 11711-11722 2020.
[30] Y. Liu, P. Kothari, B. van Delft, B. Bellot-Gurlet, T. Mordan, and A. Alahi. Tu+++: When does self-supervised test-time training fail or thrive? In Advances in Neural Information Processing Systems, volume 34, 2021.
[31] M. Long, H. Zhu, J. Wang, and M. I. Jordan. Unsupervised domain adaptation with residual transfer networks. In Advances in Neural Information Processing Systems, volume 29, 2016.
[32] B. McMahan, E. Moore, D. Ramage, S. Hampson, and B. A. y Arcas. Communication--ffiienn learning of deep networks from decentralized data. In Artificial intelligence and statistics, pages 1273-1282. PMLR, 2017.
[33] Y. Ming and Li. How does fine-tuning impact -ifto---ditrtiutiin detection for vision-language models? International Journal of Computer Vision, 132(2):596-61) 609, 2024.
[34] M. J. Mirza, P. J. Soneira, W. Lin, M. Kozinski, H. Possegger, and H. Bischof. Actmad: Activation matching to align distributions for cest-time-tainning In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 14152-2461,, 2023.
[35] K. Muhammad, A. Ullah, J. Lloret, J. Del Ser, and V. H. C. de Albuquerque. Deep learning for safe autonomous driving: Current challenges and future directions. IEEE Transactions on Intelligent Transportation Systems, 22(():4316 4336, 2020.
[36] Z. Nado, S. Padhy, D. Scullcy, A. D' Amour, B. Lakshminarayanan, and J. Snoek. Evaluat- ing prediction- batch normalization for robustness under covariate shift. arXiv preprint arXiv:2006. 10963, 2020.
[37] S. Niu, C. Miao, G. Chen, P. Wu, and P. Zhao. Test-time model adaptation with only forward passes, In International Conference on Machine Learning, 2024.
[38] S. Niu, Wu, Zhang, Y. Chen, S. Zheng, P. Zhao, and M. Tan. Efficient test-time model adaptation without forgetting. In International Conference on Machine Learning, pages 16888- 16905, 2022.
[39] S. NIu, J. Wu, Y. Zhang, Z. Wen, Y. Chen, P. Zhao, and M. Tan. Towards stable test-time adaptation in dynamic wild world. In International Conference on Learning Representations, 2023.
[40] Y. Oh, J. Lee, J. Choi, D. Jung, U. Hwang, and S. Yoon. Efficient ceffusioo-driven corruption editor for test-time adaptation. arXiv preprint arXiv:2403. 10911, 2024.
[41] Z. Pei, Z. Cao, M. Long, and J. Wang. Multi-adversarial domain adaptation. In AAAI Conference on Artificial Intelligence, 2018
[42] Z. Qiu, Y. Zhang, H. Lin, S. Niu, Y. Liu, Q. Du, and M. Tan. Source-free domain adaptation via avatar prototype generation and adaptation. In International Joint Conference on Artificial Intelligence, 2021.
[43] A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell, Mishkin, J. Clark, et al. Learning transferable visual models from natural language supervision. PMLR, 2021.
In International conference on machine learning, pages
12

---

[44] B. Recht, R. Roelofs, L. Schmidt, and V. Shankar. Do imagenet classifiers generalize to imagenet? In International Conference on Machine Learning, pages 5389-5400, 2019.
[45] K. Saito, K. Watanabe, Y. Ushiku, and T. Harada. Maximum classifier discrepancy for unsuper- vised domain adaptation. In IEEE Conference on Computer Vision and Pattern Recognition, pages 3723-3732, 2018.
[46] S. Schneider, E. Rusak, L. Eck, O. Bringmann, W. Brendel, and M. Bethge. Improving robustness against common corruptions by covariate shift adaptation. In Advances in Neural Information Processing Systems, volume 33, pages 1539-11551, 2020.
[47] M. Shu, W. Nie, D.-A. Huang, Z. Yu, T. Goldstein, A. Anandkumar, and C. Xiao. Test-time prompt tuning for zero-shot generalization in vision-language models. In Advances in Neural Information Processing Systems, volume 35, pages 14274-14289, 2022.
[48] Song, J. Lee, I. S. Kweon, and S. Choi. Bcotta: Memory-efficient continual test-time adaptation via self-distileed regularization. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pages 11920-11929) 2023.
[49] J. Song, K. Park, I. Shin, S. Woo, C. Zhang, and I. S. Kweon. Test-time adaptation in the dynamic world with compound domain knowledge management. IEEE Robotics and Automation Letters, 2023.
[50] Y. Su, X. Xu, T. Li, and K. Jia. Revisiting realistic test-time training: Sequential inference and adaptation by anchored clustering regularized self-training. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2024.
[51] Y. Sun, X. Wang. Z. Liu, J. Miller, A. Efros, and M. Hardt. Test-time training with self- supervision for generalization under distribution shifts. In International Conference on Machine Learning, pages 9229 9248, 2020.
[52] M. Tan, G. Chen, J. Wu, Y. Zhang. Y. Chen, P. Zhao, and S. Niu. Uncertainty-calibratee dest-time model adaptation without forgetting. arXiv preprint arXiv:2403. 2024.
[53] A. Tang, L. Shen, Y. Luo, N. Yin, L. Zhang, and D. Tao. Merging multi-task models via eeight-ensembling mixture of experts. In International Conference on Machine Learning. 2024.
[54] D. Wang, E. Shelhamer, S. Liu, B. Olshausen, and T. Darrell. Tent: Fully test-time adaptation by entropy minimization. International Conference on Learning Representations, 2021.
[55] H. Wang, S. Ge, Z. Lipton, and E. P. Xing. Learning robust global representations by penalizing local predictive power. In Advances in Neural Information Processing Systems, pages 10506- 10518, 2019.
[56] Q. Wang, O. Fink, L. Van Gool, and D. Dai. Continual test-time domain adaptation. In IEEE Conference on Computer Vision and Pattern Recognition, 2022.
[57] X. Wang, Y. Tsvetkov, S. Ruder, and G. Neubig. Efficient test time adapter ensembling for low-resource language varieties. arXiv preprint arXiv:2109.0445
2021.
[58] Wei, X. Wang, D. Schuurmans, M. Bosma, F. Xia, Chi, Q. Le, D. Zhou, al. Chain- of-thought prompting elicits reasoning in large language models. In Advances in Neural Information Processing Systems, volume 35, pages 14824-24831 2022.
[59] Z. Wen, S. Niu, G. Li, Q. Wu, M. Tan, and Q. Wu. Test-time model adaptation for visual question answering with debiased self-supervisions. IEEE Transactions on Multimedia, 2023.
[60] R. Wightman.
Pytorch image models.
serrchh-iaage-odde
2019.
[61] Y. Xin, S. Luo, Zhou, J. Du, X. Liu, Y. Fan, Li,, and Y. Du. Parameter-efficint fine-tuning for pre-trained vision models: A survey, arXiv preprint arXiv:2402.02244, 2024.
[62] E. Yang, Z. Wang, L. Shen, S. Liu, G. Guo, X. Wang, and D. Tao. Adamerging: Adaptive model merging for multi-task learning. In International Conference on Learning Representations, 2024.
[63] L. Zhang and X. Gao. Transfer adaptation learning: A decade survey. IEEE Transactions on Neural Networks and Learning Systems, 2022.
(64] M. M. Zhang, S. Levine, and C. Finn. Memo: Test time robustness via adaptation and augmentation. In Advances in Neural Information Processing Systems, 2022.
13

---

[65] Y. Zhang, B. Hooi, L. Hong, and J. Feng. Self-supervised aggregation of diverse experts for test-agnostic long-tailes recognition. Advances in Neural Information Processing Systems, volume 35, pages 34077-34090, 2022.
[66] Y. Zhang, S. Niu, Z. Qiu, Y. Wei, P. Zhao, J. Yao, J. Huang, Q. Wu, and M. Tan. Covid-da: deep domain adaptation from typical pneumonia to covid-19. arXiv preprint arXiv:2005.01577, 2020.
[67] Y. Zhang, Y. Wei, Q. Wu, Zhao, S. Niu, J. Huang, and M. Tan. Collaborative unsupervised domain adaptation for medical image diagnosis. IEEE Transactions Image Processing, 78444347744,, 2020.
[68] H. Zhuang. Z. Weng, H. Wei, Xie, K.-A. Toh, and Lin. Acil: Analytic class-incremental learning with absolute memorization and privacy protection. In Advances in Neural Information Processing Systems, volume 35, pages 11602-11614, 2022.
14

---

Supplementary Materials for " "Cross-Device Collaborative Test-time Adaptation
"
Contents
A Related work
16
More design details of CoLA
18
More implementation details
20
More details on datasets
20
More experimental protocols on evaluation
20
More experimental protocols on methods
22
More experimental results
23
Additional discussions
25
F
Limitations and future works
29
Broader impact
29
15

---

Table A: Characteristies of problem settings that adapt a trained model to a potentially shifted domain. 'Online' adaptation predicts a single or batch of incoming test samples immediately. '#Devices' is the number of devices involved in TTA. 'Learned Knowledge' is the knowledge from model adaptation.
Target Data
Testing Loss
Leurned
Resource-Limited
Data Transmission
Setting
Online
#Devices
Privacy
Knowledge
Devices
Fine-tuning
x', of
Not Considered
Not Considered
Continual learning
x² of
X
Accumulated
Not Considered
Unsupervised DA Test-time training
X
Accumulated
Not Considered
xf
((x²)
Not Considered
Not Considered
Fully TTA
xf C(x))
Not Considered
Purtly Applicable
Federated TTA 2
xf
[(x²)
Not Considered
Not Considered
Preserved
Intensive
Cloud-Edge TTA
4
x!
((x))
Not Considered
Applicable
Violated
Intensive
CoLA (Ours)
x
((x²)
M
Accumulated
Applicable
Preserved
Intermittent
A Related work
We summarize our main differences in Taale|AJand discuss the related works the following.
Test-time adaptation (TTA) seeks to enhance the model performance on unseen, potentially shifted test data, by directly learning from the test data itself. We categorize the related TTA works into the following four groups for discussion, according to 1) the number of devices involved in adaptation; 2) their dependence on backward propagation; and 3) the availability of multiple models.
TTA. Test-Time Training (TTT) [5 first proposes this pipeline. During the training phase, TTT methods train source model with both supervised and self-supervised branch. Given a test sample during testing, they typically update the shared encoder with the self-supervised objectives, such rotation prediction (5]), contrastive learning [30 2), reconstruction learning [[]] To avoid altering the model training phase and access to source data, Fully TTA methods directly update on-the-fly model via unsupervised learning objectives, including, but not limited to, entropy minimization [54] 39). prediction consistency maximization [64]
Single-device backpropagattonn-asse
10 and feature distribution alignment
34
In pursuit of efficient backpropagatioon-basce TTA, the attempts of existing methods can be generally categorized into: 1) Sample Efficiency. As test data are not equally important for adaptation, some recent works [38 39 47 25 have devised various sample selection strategies to identify reliable and non-redundant samples for test-time learning. It reduces the noise in the gradient and the number of samples for TTA, thereby enhancing adaptation performance and efficiency. 2) Memory Efficiency: ВсоTTA (48) reduces run-time memory by optimizing only the parameter-efficient adapters. MECTA (19 reduces the batch size at testing, while further proposes a domain-aware batch normalization layer stabilize TTA using only a small batch size.
Nevertheless, these methods focus on single-device adaptation, where all devices adapt from scratch. In this sense, valuable knowledge Iearned from other devices is neglected, damaging the adapta- tion performance and efficiency. Moreover, these methods still rely on computationally intensive backpropagation for model updates, which hinders their applicability in resource- limited devices or iutency-sensitive scenarios. To address this, we propose a gradient- based and forward-ond collaboration paradigm, facilitating knowledge accumulation, sharing, and utilization across devices.
Recently, TTA-CDKM 49 proposes to learn and reuse multiple groups of model parameters for better adaptation, with each group representing the knowledge of a domain. However, TTA-CDKM exploits only a parameter group during inference, and thus fails to aggregate the strength of diverse domain knowledge. On the other hand, TTA-CDKM updates the stored parameters when adapting each batch of test samples, which introduces intensive communication costs for weight synchronization that impractical for multi-device collaboration. In contrast, our CoLA effectively learns to aggregate diverse strengths of domain vectors at testing while ensuring communication efficiency.
Single-device forward-only TTA. In the development of BP-free TTA, early research mostly focused on calibrating the statistics of batch normalization layers by leveraging the test data to estimate test statistics [36 46). Nevertheless, these strategies only conduct adaptation on batch normalization layers, limiting their applicability to various architectures. pursuit of more general forward-only TTA solution, existing methods can be generally divided into: 1) Input-Level Adaptation,
16

---

where the corrupted test images are reconstructed before making predictions based on a diffusion model [ 40) 2) Output-Leve Level Adaptation, where T3A [2T] proposed a prototype-basee classifier for adaptive predictions, and LAME 33 directly corrects the predicted logits. However, since BP-free TTA does not leverage model feedback, i.e.,
learning objectives, for knowledge acquisition, it often results in suboptimal performance when dealing with out-of-distribution testing data. In our paper, we address this by facilitating knowledge sharing and utilization across devices, where we adaptively aggregate the learned knowledge of other devices in a forward-only manner.
Single-device test-time aggregation. Given multiple pretrained models, one can leverage unsuper- vised objectives at test time to adaptively aggregate their diverse strengths. Based on the level on which aggregation is performed, existing solutions can be generally divided into:
1) Output-wise test-lime aggregation: Farly research mainly focuses on output-wise test-time aggrega- tion, where the aggregation is conducted on the output logit of each model. EMEA (57) first proposed this pipeline. Given multiple models trained on different datasets or different label distributions, this paradigm introduced a reweighting vector to aggregate the outputs logit of various models during testing, where the reweighting vector is optimized based on entropy minimization [57] 65], consistency loss [5], etc. More Recently, Mute [)] jointly updates the reweighting vector and all pretrained models at test time. Nevertheless, these strategies necessitate forward passes for each candidate model, i.e., (NN) forward passes with N the number of candidate models. Thus, they typically consume substantially more computation power, impeding their feasibility at edge devices.
2) Parameter-wise test-time aggregation. In contrast, parameter-wiss test-time aggregation methods directly merge several candidate parameters into a single model without necessitating additional forward passes, i.e., ((1) forward passes neglect of N, rendering it more efficient in computation. To this end, Adamerging [62] first introduces the learnable parameter-wise weighting vectors, which aggregate various models before performing forward passes. Building upon this, WEMOE (53) further introduces the mixture of experts (MoE) into this paradigm, where the parameters from different models are considered as different experts, while the parameter-wise weighting vector is generated by router that is trained during testing, with the entropy minimization objective.
Nevertheless, existing test-time aggregation solutions assume the availability of pretrained models from the target domain, which struggles to fulfill the conventional TTA settings. Unlike these methods, we perform both new knowledge learning and existing knowledge aggregation simultaneously at test time, where knowledge is accumulated across all devices from previous learning.
Multi-device test-time adapiation. Existing TTA methods typically conduct adaptation on single device, where multi-device test-time adaptation is heavily overlooked. In recent academics, FedTHE+ [22] proposes a global-local scheme for federated test-time adaptation. Specifically, during each round of federated training, each device trains the global model on the local data, where the trained model is then sent back to the server. During testing, it adaptively merges the output logit of the global and local model based on entropy minimization and feature alignment loss. Nevertheless, it necessitates alteration to model training, limiting their applicability to scenarios where training data is unavailable. ATP 1 obtains module-specific adaptation rates via federated learning across clients and applies the adaptation rates in TTA. However, these methods still conduct TTA independently on each devices, and thus inherits the limitation of the single-device TTA method. More recently, CEMA [4] proposes cloud-edge collaboration paradigm for TTA. In CEMA, edge devices perform pure inference to filter reliable and informational samples to reduce communication burden, where the computationally intensive model updates are offloaded to the cloud server. Nevertheless, CEMA necessitates intensive transmission of both data and model weights. which introduces a heavy communication burden and may violate user privacy. Unlike these methods, we conducts collaborative adaptation at test time and necessitate only the intermittent transmission of updated model parameters, which is more practical in real-world implementation.
Unsupervised domain adaptation (UDA). Conventional UDA tackles distribution shifts by jointly optimizing source model on both labeled source data and unlabeled target data to learn domain-
invariant features [3] 41 45
67 (66) To avoid access to source data, recently CPGA [42) generates feature prototypes for each category with pseudo-labeling. SHOT (27] learns target-specific feature extractor by information maximization for representation alignment. Nevertheless, these methods necessitate pre-collected target datasets for offline adaptation, which limits their applicability in the real world. In contrast, our method adapts in an online manner and does not access the model training phase or access source data, which facilitates more practical adaptation paradigm.
17

---

B More design details of CoLA
In the section, we further elaborate more details of our methods regarding the design of saving domain knowledge upon distribution shifts
Initialization of and A in Eqn. (2) for stable adaptation warm-up. Since ground-truth labels are absent in our online TTA scenario to provide stable learning signals for Eqn. (2). careful initialization of and A is crucial upon distribution changes. In this sense, ensure the stability of model performance pre and post encountering a new domain while enabling flexible aggregation of existing knowledge, we devise the initialization strategy by considering the following requirements: 1) the overall model weights 0f initialized after encountering the domain n+1 should remain the same as 0f' that learned on the n-th domain; 2) A should be as small as possible to enable flexible samples all from in in-distribution,
obtain the solution of 0-=0f by setting Og 0g=1 1 if A is negligible; 3) for a==oftmax(3)) (3), the initial B3 should not be very sharp, e.g., (0, inf, 0, -.... 0), it will hinder the model learning from selecting diverse knowledge.
adaptation, e.g.. when the
Formally, let { ( |=1 be the learned parameter on the i-th layer in AO. We define A s magnitude the maximum scale of parameters across different layers & { J is the number of parameters in each layer. To this end, satisfy the above constraint by defining the reweighting logit A0n+1 as:
A0())) 1 where
1 In (s
5 Wm
Bxt1
Σ T.T
max
Here, Sn+1 is designed so that the magnitude 5' of the recalculated AO satisties 5' < Wms where Wm is a constrained value set to 0.01. Moreover, to improve numerical stability in case Ti is O, clip the value of Ba+1 between [0,,, 10]. The preserved knowledge A is then shared across devices.
We next provide a proposition to validate the reasonableness of the design for Bn+1:
Proposition 1. Given Pn-1 in Eqn. 8 the constraints that 0, 07++1 and g' S Wm holds.
Proof: Assume that the model has learned from previously domains. According to Eqn. @ when the model adapts on the (n+1)-th domain, we have
0 + 0. + 20. a A0 + ...... A0, A0, where softmax(3 Ti).
When encountering the (n+2)-th domain, should save learned knowledge AO from the (n+11-00 domain. To adaptively aggregates prior knowledge and saved knowledge, we initialize 0x+1 for A0,.. n+ Besides, we initialize A0' in order learn new knowledge from the (n+2)-th domain. So
we have
0p+2 00 + +66000 + a' +... 20, + an A0n+1 + AO'
After adding a,41, maintain the vector normalized weights, ag will change
e:77
a,
" Cr eB
+ B04ill
(B.
ΣØ Bi7:
1= B3/T
ei1 eBn+iTi
Σ
(Σn e
As we want to ensure the stability of model performance, the overall model weights 0 should be the same as + that learned on the (n+11)-th domain, so A0,+1 = 0 09 0 0
18

---

Thus, we expand 2?-2 as:
Ou+²= 0f og A00 + a0,00 + +0, an+1100m + A0'
Σ B
= 0. + Li= 0i=0 e
(a0200 + aj 10 + + a,,00,)
+ ei
+ -i= e..T))
A0.+1 + A0'
Ei=o B:2
08 +
(0 Of -0)
EE= e
+ @0111
6 Ti
(0)+)
0e) + A
-i= eBi2))
ennt1ll
Di=0 ei:Ti
0
A0 + -0'
2i= eb22))
e0nt1li
Considering 0,+2 0pt!, we have
Ri B:.
0
A0 + A0' =0, 0
i= eB:T)) + ButIT
AO'
1
AO.
11 ===0
+ e
As A should be as small as possible to enable flexible adaptation, the magnitude of A0' should be no larger than Wms i.c. e' S Wm. So we scale AO by a factor s, i.e. A0' A0. the
magnitude of g already satisfies g' S Wm. we set Thus, we have = m { and
1. Otherwise, we set E
Σ aTi
E/ efiTi)
+ @m 8
Bn+1
In
(s - 1) Σ 3,7
Ti
1=0
19

---

C More implementation details
C.1
More details on datasets
In this paper, we conduct experiments on ImageNet-IK IK out-of-distribttion generalization ability, i.e., ImageNet-C
[6 and its five variants to evaluate the
T6 ImageNet-R
[5].
ImageNet-A [77.
ImageNet-V2 and ImageNet-Sketch [55
ImageNet
ImageNet-C
ImageNet-VZ
ImageNet-A
ImageNet-R
ImageNet-S
Original
Noise
Blur
Adversarial
Rendition
Sketch
Original
Weather
Digital
V2
Adversarial
Rendition
Sketch
Figure Visualizations images in ImageNet and ImageNet-CC/22///R/Kaee
ImageNet-CC consists of various versions of corruption applied to 50,000 validation images from ImageNet. The dataset encompasses distinct corruption types of main groups, including Gaussian noise, shot noise, impulse noise, defocus blur, glass blur, motion blur, zoom blur, snow, frost, fog, brightness, contrast, elastic transformation, pixelation, and JPEG compression. Each corruption type characterized by 5 different levels of severity, with higher severity levels indicating more severe distribution shift. In our experiments, we specifically utilize severity level 5 for evaluation.
ImageNet-RR contains 30,000 images featuring diverse artistic renditions of 200 ImageNet classes. These images are predominantly sourced from Flickr and filtered by Amazon MTurr annotators.
ImageNettA comprises 7,500 images covering 200 ImageNet classes. These images are derived from real-world, naturally occurring examples that lead a notable degradation in classifier performance.
ImageNet-V2 newly collected test dataset extracted from the same test distribution as ImageNet. It comprises three test sets, each containing 10,000 images and covering 1000 ImageNet classes. Following previous TTA methods [36), we utilize the Matched Frequency subset ImageNet- V2 for evaluation, in which the images are sampled match the class frequency distributions of the original ImageNet validation dataset.
Imaeehet-Sketch consists of 50,899 images represented as black and white sketches, encompassing 1000 ImageNet classes. Each class contains approximately 50 images.
C.2 More experimental protocols on evaluation
We use ViT-Base g as the source model for all experiments except Table The model is trained on the source ImageNet- IK training set and we directly obtain the model weights from time! repository [60]. In Table 7 we adopts CLIP-RN50 [3] as the source model by following TPT [4].
The model weights are directly obtained from the original C [43]. All experiments conducted on a single NVIDIA A100 GPUS, using Py PoToh framework with version 1.8.0.
Evaluation on lifelong TTA. In Table | the model is online adapted to 15 corruptions over 10 rounds (total 150 corruptions), where the parameters will never be reset. Specifically, the corruptions
do_ hhttps:/storage.googleeee com/vit models/augreg/BB 16-i21k 300ep-r 0.001-aug mediuml-we 0.1-
bbtps//ggthbb.co//ggen/CCLLLL
20

---

comes in the following order for each round: Gaussian Noise + Defocus Blur + Snow - Contrast Shot Noise - Glass Blur + Frost + Elastic Transform + Impulse Noise + Motion Blur - Fog - Pixelate - Brightness - Zoom Blur - JPEG Compression. Here, subsequent corruptions differ in type, which poses a more significant challenge for TTA methods to leverage previously learned knowledge for adaptation.
Evaluation on collaborative TTA. In Table 2 we evaluate our effectiveness in collaborating on multiple resource-abundant principal devices, where the learned knowledge on each device is shared with others post-adaptation to each type of corruption. Specifically, the corruption order in each type of coruptiin is as follows. 1) Noise: Gaussian Noise - Shot Noise + Impulse Noise. 2) Blur: Defocus Blur- - Glass Blur > Motion Blur + Zoom Blur. 3) Weather: Snow Frost + Fog > Brightness. 4) Digital: Contrast + Elastic Transformation - Pixelate - JPEG Compression.
Then, given learned knowledge from Table we further verify effectiveness on resource-limited follower devices in Table e.g., CoLA (SAR) utilizes the learned weights of SAR+CoL.A from Table 2 and CoLA (ETA) utilizes the learned weights of ETA+CoLA from Table? 2 Here, the resource-limited follower devices adapt in lifelong manner per EATA [38], where the corruptions come in the following order: Gaussian Noise Shot Noise Impulse Noise Defocus Blur Glass Blur + Motion Blur - Zoom Blur - Snow - Frost - Fog + Brightness - Contrast - Elastic Transformation + Pixelate > JPEG Compression.
Evaluation on single-domain TTA. In Table we validate our CoLA in both the wild scenario (i.e., imbalanced label distribution shifts and mixed domain shifts) and the mild scenario of single-domain TTA, where the model reset post-adaptation to each corruption. Here, imbalanced label distribution in class order, mixed domain shifts denotes scenarios
shifts denotes scenarios where test data where test data are drawn from multiple randomly mixed domains with different distribution shifts.
Evaluation on prompt tuning. In Table 7 CoLA aggregates 78 diverse hard prompts with the class token at the end for Eqn. 2). where different prompts are padded to the same length by inserting empty spaces the beginning. Note that adaptation is conducted in episodic manner by following TPT [47], where we reset AO 0 post- adaptation to each test sample in Eqn. (2). while a is continually optimized without reset due to its stability benefit from normalization. Results in Table further demonstrate our effectiveness across various TTA scenarios. The utilized hard prompts for Eqn. 2 are originally developed by [3 as listed below:
List of Hard Prompts:
a drawing of the [class], ) art of a (class), J. itap of the [class], a drawing of a [class), J, a origami [class], photo of a nice (class), ). blurry photo of [class), close-up photo of the (class), a photo of a clean [class), a photo of a weird [class], J, a photo of a small (class), ), a photo of the large (class), pixelated photo of the (class), embroidered (class), photo of the clean (class), the origami (class), ), the plushie (class), a photo of cool (class), ), a sculpture of the /class], ), low resolution photo of the (class]. bad photo of the (class), jpeg corrupted photo of a (class), a rendition of the (class). a photo of the cool (class), a low resolution photo of (class). cropped photo of the [class), the plastic (class), sculpture of (class), a pixelated photo of a [class), itap of a [class), a doodle of a [class), a sketch of a [class), a plastic [class). itap of my (class), close-up photo of [class), ), a bright photo of a [class). art of the [class], J. graffiti of the (class), tattoo of a [class], sketch of the [class], J. a dark photo of (class), tattoo of the [class), a photo of the dirty (class), a black and white photo of the (class), ) a photo of a (class), a painting of the [class], a cropped photo of (class), a photo of large (class), a photo of the weird (class), ), graffiti of (class) ), a painting of a (class), ) cartoon (class), |, the cartoon (class), ) good photo of the [class), J, jpeg corrupted photo of the (class), a bad photo of a (class), photo of the small [class), ), rendering of the (class), a photo of dirty (class), rendition of a (class), ). a blurry photo of the (class), the toy {class), the embroidered (class). a rendering of a (class), a photo of hard to see {class), a dark photo of the (class), doodle of the (class), good photo of a (class), ), photo of the (class), a photo of many cclass). a plushic (class), ). a photo the nice (class), J, a bright photo of the (class), J. toy {class), J. photo of the hard to see (class), ), photo of one {class), photo my (class), ) a black and white photo a (class), sketch of a (class)
21

---

C.3 More experimental protocols on methods
CoLA (Ours), For principal agents, We directly leverage the learnable test test-time objectives from the integrated TTA methods, as listed below. In Eqn. 2 A6 is optimized by following the update rules and the hyper-parameters, of the integrated baseline, and Ti is updated via the Adam optimizer with learning rate of 0.1, and we introduce weight decay of 0. on a. We set the shift detection threshold to 0.1 for all experiments. Moreover, we introduce a weight decay on 2O for Table [ in
and Table K iee. 0. 1/0.4/0..4 for SAR/ETA/De YO in [able and 0/0.4/10 for EATA/ETA/SAA Table respectively. This weight decay, which can be viewed as a simple implementation of the regularizer in EATA 38), helps learn compact and non-redundant knowledge in the domain vectors, thereby benefiting knowledge accumulation. For follower agents, we consistently set T in Eqn. (3) to 5 for all experiments. Moving average factor À is set to 0.2.
SAR 33). We follow all syper-parametess that are set in SAR unless does not provide. Specifically, we use SGD as the update rule, with a momentum of 0.9, batch size of 64, and a learning rate of 0.001. The entropy threshold Eg is set to 0.4x x In C, where C is the number of task classes. The trainable parameters are the affine parameters of the layer normalization layers from blocks to blocks 8.
ETA & EATA (38] We follow all hyper- papee--araeeterss that are set in ETA/EATA unless it does not provide. Specifically, we use SGD as the update rule, with a momentum of 0.9, batch size of 64, and learning rate of 0.001. The entropy threshold Eg is set to 0.4 In C, where C is the number of task classes. For EATA, i.e., ETA with an anti-forgetting regularizer, we use 2,000 samples to estimate the importance of each parameter. The trainable parameters are all affine parameters of layer normalization layers.
De (25). We follow all hyper-parameter that are set in SAR unless it does not provide. Specifi- cally, we use SGD as the update rule, with a momentum of 0.9, batch size of 64, and a learning rate of 0.001. The entropy threshold Eg is set to 0.4x In C and the entropy factor Tim is set to 0.5 In C, where C' the number of task classes. The Pseudo-Label Probability Difference (PLPD) threshold TPLPD set to 0.3 in Table 4 and 0.2 for other experiments by following the original paper. Trainable parameters are the affine parameters of the layer normalization layers from blocks to blocks 8.
COTTA 10 (56.. We follow all hyperparameters that are set in CoTTA unless it does not provide. Specifically, we use SGD as the update rule, with a momentum of 0.9, and a batch size of 64. The learning rate is chosen from (0.05, 0.01, 0.005, 0.001, 0.0005, 0.0001] and the augmentation threshold Pts. is chosen from (0.1, 0.2). In all experiments, we consistently set the learning rate to 0.001 and Pt.. to 0.1 given the optimal accuracy observed in Table For images below the threshold, we conduct 32 augmentations including color jitter, random affine, Gaussian blur, random horizontal flip, and Gaussian noise. The restoration probability of is set 0.01 and the EMA factor for teacher update is set to 0.999. The trainable parameters are all the parameters in ViT-Base.
For fair comparison, we maintain consistent batch size of 64 for LAME, aligning it with the same batch size used by other methods in our evaluation. We use the kNN affinity matrix with the value of k chosen from (1, 5, 10, 20), and for all experiments, we consistently set it to 5 based on the optimal accuracy observed in Table 3
LAMI IT
3
T3A 12 [2 We follow all pepee-pprameters that are set in T3A unless it does not provide. Specifically, the batch size is set to 64. The number of supports to restore M is chosen from (1, 5, 20, 50, 100), and for all experiments, we set it to 20 based on the optimal accuracy observed in Table
TPT [7]. We follow all hyper-parameters that are set in TPT unless does not provide. Specifically, we use AdamW as the update rule, with batch size of I and learning rate of 0.005. Learnable tokens initialized from the hard prompt of 'a photo of a". The confidence threshold p is set 0.1 and the number of TPT steps is set to 1.
https://githubb www//ocoo/////eeeeeeeeeee howps//ggggoogggglas...k
https:://ithhii http://tthhuucooomrrreeerr htmm/mm-oggglaa//AAAA
httpss//gthuubbom////y/////////
https//gthub..oo//gnnneeggg////
hhtps://ithuh.co///iiw//////
https//github.co//zzuueeeee
22

---

D More experimental results
In the main paper, we only report the results averaged over 15 corruptions in ImageNet-C (i.e., in Table [ and T) or over each group of corruptions (i.e., in Table 2) due to page limit. In this section, we offer additional results to enable a more comprehensive comparison.
Results under lifelong test-time adaptation. The knowledge sharing scheme in CoLA can also be used to mitigate catastrophic forgetting in lifelong TTA, similar the aim of conventional supervised continual learning [68] To further verify this, we provide more detailed results of Table regarding the Accuracys on the first and last rounds of adaptation. From Table our CoLA outperforms the integrated baseline in the first round of adaptation, e.g., the accuracy of 62.0% (ETA+CoLA) vs. 61.4% (ETA). This mainly stems from the latter corruptions encountered in the first round, where CoLA demonstrates superiority by leveraging the learned knowledge from previous adaptation with Eqn. 2 More importantly, our enhancement becomes particularly significant at the last round of adaptation, e.g., the accuracy of 65.4% (ETA+CoLA) vs. 35.1% % (ETA), further indicating our effectiveness in accumulating and utilizing learned knowledge for long-range adaptation. Note that CoLA applies weight decay on AO as claimed in Section C thus achieving a lower performance
on the first corruption. Here, the relatively limited performance of COTTA [56] is attributed to its sensitivity to corruption order, as shown in Table
Results under collaborative test-time adaptation. We provide more detailed results of Table 2 regarding the Accuracys on each device. From T our CoLA outperforms the integrated baseline from the adaptation the second group of Corruption, e.g. the accuracy of 53.6% (SAR+CoLA) vs. 38.9% (SAR) on 'Gaussian' in Device 2. Moreover, this improvement becomes increasingly more pronounced as more knowledge is shared across devices, e.g., improving the accuracy from 37.5% (SAR) to 53.9% (SAR+CoLA) on 'Gaussian' in Device 3. This phenomenon underscores the importance of cross-device collaboration and our effectiveness regarding collaborative TTA.
Results under single-domain test-time adaptation. We provide more detailed results of Table 4 regarding the Accuracys under mild scenarios and online imbalanced label distribution shifts. From Table B incorporating CoLA with existing TTA solutions enhances the adaptation performance on most corruptions under both scenarios, demonstrating our effectiveness. Specifically, CoLA showcases a more pronounced improvement in SAR and ETA. This can be attributed to the inadequacy of prediction entropy to identify reliable samples for model updates, thereby suffering more from error accumulation. CoLA alleviates this by dynamically favoring more optimal checkpoint based on loss minimization instead of the newest saved one that may have learned from erroneous predictions, rendering CoLA more robust to noise. We also visualize a; in Appendix E to offer more insights.
Table B: Comparisons on ImageNet-C (level 5) regarding Accuracy (%) under single-domain TTA. In mild scenarios, the test samples come in random order by following Tent (54). Label Shifts is short for online imbalanced label distribution shifts, where test samples come in class order per SAR [39].
Mild Scenarios Gaus, Shot Imp. Def. Glass Mot. Zoom Snow Frost Fog Brit. Contr. Elas. Pix. JPEG|
Avg.
NoAdapt
9.5
6.7
8.2
29.0
23.4
33.9
27.1
15.9
26.5
47.2 54.7
44.1
30.5
44.5 47.8
29.9
FATA 38
49.5
48.9
50.2
54.8
54.8
58.9
56.2
61.6
60.7
70.6 75.2
66.9
63.7
69.7
66.8
60.6
SAR 39
44.0
24.2
45.3
53.0
49.9
55.7
51.0
57.4
45.1
66.6
74.7
64.5
55.3
66.7
64.0
54.5
+ CoLA (Ours)
48.4
27.1
49.0
54.9
53.7
58.7
55.1
60.9
50.5
69.2
76.0
66.2
60.5
69.2
66.4
57.7
ETA 38
51.9
51.9
52.8
57.6
57.6
62.2
60.0
66.1
65.1
72.5
77.4
67.6
66.1
72.0
69.2
63.3
Col.A (Ours)
53.4
53.7
54.3
58.2
58.6
63.2
61.5
67.2
66.0
73.0
77.7
68.2
68.1
73.0
70.0
64.4
DeYO 25
53.2
53.4
54.1
58.3
58.3
63.2
56.6
67.2
66.0
73.4
78.1
68.0
67.9
73.2
70.1
64.1
+ CooLA (Ours)
54.3
50.6
55.0
58.4
59.4
64.2
60.1
68.1
66.5
73.9
78.1
68.3
68.8
73.9
70.4
64.7
Label Shifts
Gaus. Shot Imp. Def. Glass Mot.
Zoom
Snow
Frost
Fog Brit. Contr. Elas.
Pix.
JPBG|
Avg.
NoAdapt
9.5 6.7 8.2 29.0
23.4
33.9
27.1
15.9
26.5
47.2
54.7
44.1
30.5
44.5
47.8
29.9
EATA 38
36.1
35.7
35.4
45.0
42.8
52.0
45.1
55.0
48.7
62.1
73.0
42.9
55.8
63.9
62.7
50.4
SAR [39
47.9
30.7
48.4
55.4
54.2
58.8
54.6
43.5
48.3
69.4
76.3
66.2
60.8
69.4
66.6
56.7
CoLA (Ours)
49.6 50.4
50.7
56.4
56,4
60.3
57.3
40.9
36.6
71.5
77.2
67.0
64.2
71.3
68.5
58.5
ETA 38
31.0
34.4
32.0
30.5
44.8
49.6
46.4
54.9
53.0
56.3
74.1
25.1
57.8
64.2
59.4
47.6
Col.A (Ours)
40.2 37.6
42.4
47.1
49.6
52.5
53.0
59.7
58.4
65.0
74.6
50.9
61.3
68.9
66.3
55.2
Yo 25
53.0
53.9
54.5
57.8
59.0
63.9
12.7
68.0
66.1
73.2
77.9
66.6
68.9
73.7
70.6
61.3
CoLA (Ours)
52.9
54.5
54.9
58.0
59.2
63.6
43.1
68.3
66.0
73.3
78.0
66.8
69.1
73.8
70.7
63.5
23

---

Table C: Comparison on ImageNet-C (severity level 5) regarding Accuracy (%) under lifelong
adaptation for 10 rounds. Here, we provide additional results on the first and last rounds of adaptation.
Round 1
Gaus. Def. Snow Contr. Shot Glass Frost Elas. Imp. Mot. Fog Pix.
Brit. Zoom JPEG|
Avg.
NoAdapt
9.5 29.0
15.9
44.1
6.7
23,4
26.5
30.5
8.2
33.9
47.2
44.5
54.7
27.1
47.8
29.9
CoTTA [6
24.4
35.2
34,6
48.3
419
40.0
52.6
49.8
47.0
44.5
48.8
54,6
63.7
35.7
53.1
44.9
EATA 38
49.4
53.6
61.3
65.8
49.6
$4.5
60.5
64.9
50.5
57.9
69.2
69.5
75.2
56.6
67.3
60.4
SAR 33
44.0
51.9
58.5
63.3
48.8
53.0
61.0
63.5
51.2
57.2
66.9
69.0
76.5
54.0
67.2
59.1
+ CoLAA (Ours)
41.9
51.4
57.8
63.9
48.8
52.7
61.0
62.6
52.6
57.3
68.2
69.3
76.5
54.2
67.7
59.1
ETA 38
51.9
55.6
64.5
64.6
53.2
55.5
62.4
66.4
52.1
57.9
67.5 69.8
75.3
57.7
66.2
61.4
CoLA (Ours)
48.0 55.2
63.5
66.2
51.8
55.6
63.7
65.8 54.3 59.8 71.0 71.1 76.7
58.3
68.5
62.0
De [25]
52.8
56.4
65.9
65.5
54.3
57.0
63.8
68.2 54.1 60.3 69.5 71.8
76.7
20.2
60.0
59.8
+ CoLA (Ours)
49.7
55.9
65.0
66.4
53.2
56.9
65.0
67.4
55.2 61.4 72.0 72.3
77.7
37.8
69.7
61.7
Round 10
Gaus.
Def.
Snow Contr.
Shot Glass Frost Elas.
Imp. Mot. Fog Pix. Brit. Zoom
JPEG
Avg.
NoAdapt
9.5
29.0
15.9
44.1
6.7
23.4
26.5
30.5 8.2 33.9 47.2 44.5 54.7
27.1
47.8
29.9
COTTA 56
22.1
20.7
27.2
22.4
25.2
22.0
29.6
29.4
25.8
24.4 28.3 31.3
38.2
20.2
31.5
26.5
EATA 58
47.6
51.5
58.7
63.9
46.8
52.4
59.3
62.6
48.3
56.2
67.0
68.3
74.2
55.8
66.1
58.6
SAR 33
51.3
55,4
62.5
63.3
52.7
55,9
62.0
64.3
53.2
59.1
67.8
70.4
75.9
55.9
67.1
61.1
CoLA (Ours)
56.0
58.4
67.8
67.8
57.2
59.2
66.2
69.4
57.4
63.7
72.7
73.3
77.8
63.1
70.4
65.4
ETA 38
32.6
28.8
33.4
17.5
31.4
31.9
36.8
42.3
32.1
29.2
30.6
46.0
57.7
34.9
41.7
35.1
CoLA (Ours)
55.8
58.3
68.4
67.9
57.1
59.2
66.3
69.7
56.9
64.2
73.1
73.2
77.5
64.2
69.8
65.4
DeYO 25
0.1
0.1
0.1
0.1
0.1
0.1
0.1
0.1
0.1
0.1
0.1
0.1
0.1
0.1
0.1
0.1
+ Col (Ours)
56.9
58.9
69.7
68.2
58.1
59.8
66.9
70.8
58.2
64.7
73.7
73.9
78.0
53.9
70.9
65.5
Table D: Effectiveness under collaborative adaptation across resource-abundant principal devices w.r.t. Ace. (%). Results are evaluated on ImageNet- C (severity level 5, containing 15 corruption types of 4 groups). We share learned weights across devices post- adaptatonn to each group of corruptions.
Device
Gaus. Shot Imp.
Def.
Glass Mot. Zoom Snow Frost
Fog
Brit.
Contr. Elas, Pix. JPEG|
Avg.
NoAdapt
9.5
6.7
8.2
29.0
23.4
33,9
27.1
15,9
26.5
47.2
54.7
44.1
30.5
44.5 47.8
29,9
CoTTA (56
17.7
28.4
40.5
37.0
43.4
46.5
38.1
39.1
49.1
48.6
63.8
41.5
42.5
54.0
52.2
42.8
EATA 38
50.4
54.3
55.7
53.8
56.1
59.6
58.6
62.3
63.8
70.5
75.9
66.2
64.1
70.1
68.3
62.0
SAR 33
44.5
51.8
54.8
51.3
53.6
57.7
55.0
59.7
62.0
67.2
76.3
63.1
59.4
68.5
67.2
59.5
CoLA (Ours)
44.5
51.8
54.8
56.4
56.5
61.2
57.9
64.7
64.4
71.7
76.8
66.4
66.7
72.2
69.6
62.4
ETA [88]
51.9
56.3
57.2
53.1
56.7
58.8
58.9
62.2
63.3
68.9
75.5
62.7
64.3
69.9
67.1
|
61.8
+ CoLA (Ours)
52.1
56.3
57.1
57.2
58.0
62.1
62.6
67.7
66.3
72.7
77.1
66.0
69.0
72.5
69.7
64.4
DeYO 25
52.9
57.7
58.3
54.9
58.0
61.4
25.5
61.5
63.9
70.1
77.0
63.5
67.0
71.6
69.2
60.88
CoLA (Ours)
52.9
57.6
58.2
57.9
58.3
62.5
41.6
67.8
66.4
73.0
77.5
66.4
70.5
73.3
70.5
63.6
Device 2
Def.
Glass Mot. Zoom Gaus.
Shot
Imp.
Contr:
Elas.
Pix.
JPEG
Snow
Frost
Fog.
Brit.
Avg.
NoAdapt
29.0
23.4
33.9
27.1
9.5
6.7
8.2
44.1
30.5
44.5
47.8
15.9
26.5
47.2
54.7
29.9
CoTTA [56
31.4
31.7
43.9
38.1
27.0
37.7
46.7
45.2
46.3
56.2
54.0
47.0
50.9
49.3
63.5
44.6
EATA 38
55.4
57.2
60.5
59.5
48.5
53.1
55.0
65.0
64.0
70.5
68.4
62.9
64.0
71.0
76.1
62.1
SAR 33
53.0
53.4
58.7
55.3
38.9
51.5
54.7
62.2
57.6
68.2
68.0
59.9
61.9
67.6
76.3
|5.1
+ CoLA (Ours)
52.9
53.4
58.6
55.1
53.6
55.4
56.1
66.4
61.9
71.1
69.2
66.2
65.6
72.9
77.3
62.4
ETA 38
57.6
58.8
61.7
61.1
47.4
53.2
54.4
57.3
64.6
70.5
67.7
62.1
62.7
69.0
75.6
61.6
+ CoLA (Ours)
57.5
58.3
61.2
61.0
54.8
57.0
57.1
64.6
68.4
72.2
70.1
67.5
66.0
72.8
77.3
64.4
DeYO
25
58.1
59.5
63.0
41.8
39.3
50.5
50.4
61.4
66.8
71.6
68.9
64.6
63.9
70.5
76.9
60.5
CoLAA (Ours)
58.3
59.1
62.6
39.3
47.9
57.5
58.2
66.6
69.9
73.0
70.5
68.3
66.9
73.0
77.7
63.3
Device 3
Snow Frost Fog
Brit.
Contr Elus.
Pix.
JPEG
Def.
Glass
Mot.
Zoom Gaus. Shot Imp.
AAg.
NoAdapt
15.9 26.5
47.2
54.7
44.1
30.5
44.5
47.8
29.0
23.4
33.9
27.1
9.5
6.7
8.2
29.9
CoTTA 56
26.1
48.0
56.5
71.2
53.1
45.3
60.9
60.6
42.3
42.8
46.8
38.2
30.2
39.8
46.2
47.2
EATA 38
63.8
65.3
71.8
76.6
66.9
64.7
70.5
68.8
55.7
56.7
60.2
59.1
47.5
52.7
54.3
62.3
SAR 39
57.9
63.1
68.6
76.3
64.6
58.0
67.2
67.7
54.5
54.3
58.3
54.6
37.5
$1.0
54.6
59.2
+ CoLA (Ours)
57.9
63.0
68.0
76.2
64.6
59.7
69.1
67.8
56.5
57.9
61.6
59.2
53.9
56.3
56.5
61.9
FTA 38
66.0
66.3
71.9
77.0
66.0
65.6
70.7
68.8
55.3
56.8
60.1
59.9
44.3
50.9
52.9
62.2
CoLAA (Ours)
66.0
66.0
71.9
76.9
65.4
66.4
71.6
69.1
56.7
58.0
61.8
62.1
54.0
56.0
55.9
63.9
De
2
67.2
66.9
72.7
77.7
66.3
67.4
71.9
69.6
56.6
58.1
61.6
28.2
11.7
1.0
0.1
51.8
+ CoLA (Ours)
67.0
66.6
72.7
77.6
66.1
67.6
72.5
69.7
56.5
57.5
61.4
39.6
49.1
56.9
56.8
62.5
24

---

E
Additional discussions
Robustness of CoLA against potential harmful knowledge. Note that CoLA conducts test-time learning to adaptively aggregate prior knowledge. If the prior knowledge shared from some devices harmful, using such knowledge shall not decrease the test-time objective. Thus, such prior knowledge will not be used for model adaptation. In this sense, CoLA remains stable even with harmful prior knowledge (e.g., the domain knowledge in some devices is not helpful for principal agents to do TTA). This stability also benefits from our initialization strategy as shown Table E which carefully initializes and A to prevent the aggregation of potentially harmful knowledge for stable warm-up.
Table Robustness of our CoL.A using only pre-trained parameters and N harmful prior knowledge, Le., the randomly initialized domain vectors. Results are obtained on ImageNet-CC (Gaussian, level 5).
Method
N =0 0
N ==
N == 4
N = 100
N = 10000
+TA+COLA (Equal (x)
51.97
0.10
0.10
0.10
0.10
+TA+CoLA (Random o))
51.97
10.46
0.15
0.10
0.10
ETA+CoLA (Ours)
51.97
52.02
52.04
52.04
52.04
Scalability of CoLA with more collaborative devices. CoLA scales well with an increasing number of devices, i.e., with more shared domain vectors. From Table F ETA/ETAA+CLL. consistently benefits from additional participating devices, egg., ETA+CoLA achieves an accuracy 65.2% with 11 devices compared to 61.8% with one device. This highlights the importance of cross-device collaboration and CoLA 's effectiveness under more large-scale multi-device collaborative TTA.
Table F: Effectiveness of CoLA with an increasing number of principal devices (with back propaapoonn capability). Here, each device continuously encounters 15 domains from ImageNet-C (level in different domain orders. Results are averaged over all principal devices.
#Devices
1
5
7
9
11
SAR+CoLA
59.7
62.8
63.6
63.8
(3.9(114))
ETA+CoLA
61.8
63.9
64.3
64.9
65.0
65.2(13.4)
Efficiency of CoLA with increasing domain vectors. Our CoLA both computation and memory efficient at exploiting domain vectors. From Table CoLA efficiently scales to over 10,000 domain vectors, incurring only an additional 11s of runtime and 1,502MB of extra memory, yet remains substantially more efficient than CoTTA. We believe that 10,000 domain vectors should be adequate for handling most real-worke applications with proper management.
Table G: Efficiency comparison on ImageNet-C (Gaussian, level 5) using a single A 1000. N is the number of domain vectors, which initialize random parameters in this table.
Method
ETA
+CoLA (N =) +CoLA (N 100) +CoLA (N 10, 000)
CoTTA
Time (s)
109
110
112 7,448.2
120
937
Memory (MB)
7,433.2
7,433.7
8,935.5
21,628.6
Sensitivity of threshold 2 for shift detection. CoLA remains effective among a wide range of threshold 2, as shown Table A. From the results, while a stricter threshold saves more domain vectors, CoLAA achieves a stable performance of around 64.7%. When threshold increases, CoL.A saves significantly fewer domain vectors and still enhances the performance significantly, i.e., the average accuracy of 62.2% in ETA+CoLA (==10) vs. 46.4% in ETA.
Table H: Sensitivity of threshold z. Experiments follow the settings of Table | i.e., single-device lifelong adaptation, and CoLA is incorporated with ETA. We report average accuracy over 10 rounds, each comprising 15 corruptions of ImageNet-C. The average accuracy of the ETA baseline 46.4% 4%.
CoLA (==0.01)
CoLA (==0.05)
CoLA (2=0.1)
CoLA (2=1)
CoLA (==10)
Avg. Ace. #Saved Vectors
64.7
64.6
64.8
63.8
62.2
20740
369
169
110
48
25

---

Advantages of CoLA over FedAvg [32] in collaborative TTA. We further compare our CoLA with FedAvg [22 in cross-device collaborativeffederated learning. From Table CoLA consistently outperforms FedAvg when incorporated with the baseline for collaborative TTÄ. More importantly, FedAvg, which simply averages the model parameters on different devices, may deteriorate model performance (ie., the average accuracy of 61.2% in ETA vs. 58.0% in FTA++edA This is because different devices may encounter different distribution shifts, and thus the knowledge from other devices may not be beneficial for adapting to the current domain. In contrast, our CoLA addresses this by learning at test time to optimize the aggregation of knowledge from different devices.
Table I: Effectiveness of CoLA and FedAvg 32
for cross-device collaborative TTA w.r.t. Acc. (%).
The experiments follow the settings of Table in the main paper.
Device (Adapt -+)
Device 2 (Adapt ->
Device 3 (Adapt -+) Digit.
Method
Noise
Blur
Weat.
Digit.
Blur
Noise
Digit.
Weat.
Weat.
Blur
Noise
Avg.
NoAdapt
8.2
28.4
36.1
41.7
28.4
8.2
41.7
36.1
36.1
41.7
28.4
8.2
28.6
EATA 38
53.5
57.0
68.1
67.2
58.1
52.2
67.0
68.5
69.4
67.7
57.9
51.5
61.5
SAR 39
50.4
54.4
66.3
64.5
55.1
48.3
64.0
66.4
66.5
64.3
55.4
47.7
58.6
+ FedAvg
32
50.4
56.6
67.6
66.1
55.1
52.2
65.6
68.1
66.3
64.9
57.2
51.8
60.2
+ CoLA (Ours)
50.4
58.0
69.4
68.7
55.0
55.0
67.1
70.5
66.3
65.3
58.8
55.5
61.7
ETA 38
55.2
56.9
67.5
66.0
59.8
$1.7
65.0
67.4
70.3
67.8
58.0
49.4
61.2
+ FedAvg 32
55.2
58.9
65.1
62.5
59.6
51.0
63.4
63.7
70.3
46,2
54.0
46.4
58.0
+ CoLA (Ours)
55.2
60.0
70.9
69.3
59.5
56.3
68.8
70.9
70.2
68.1
59.7
55.3
63.7
Robustness of CoLA under small batch sizes. The stability of CoLA under small batch sizes is primarily determined by the base algorithms, such as ETA and SAR, rather than CoLA itself. This is because CoLA is plug-and-ple provide further empirical results to verify this robustness. From Table CoLA achieves a consistent result when incorporated with SAR, demonstrating reduces from to 2. Meanwhile, CoLA can also help improve stability under small batch sizes. For instance, as the batch size reduces from 4 2, ETA+COLA achieves nearly no performance degradation while the baseline ETA's performance degrades by 1.4%.
module designed to be incorporated with existing methods. We
performance degradation as the batch size
Table J: Robustness of our CoLA under various batch sizes. We follow the same settings of Table in the main paper and report average accuracy over all devices and corruptions here.
Method
BS = 64
BS = 16
BS
BS 2
SAR 39
58.6
58.8
58.9
58.7
+ CoLA (Ours)
61.7
61.6
61.7
61.7
ETA
61.2
60.5
58.5
57.1
+CoLA (Ours)
63.7
62.8
61.7
61.5
Effectiveness of CoLA on ResNet models. Table demonsstatsss the effectiveness of our CoLA on ResNet-50, where CoLA updates and stores the affine parameters of batch normalization layers in ResNet. From Table CoI.A consistently enhances the performance of ETA/EATA/SAR throughout 10 rounds of adaptation and addresses the issue of performance degradation in long-term adaptation. These results consistent with Table 1 in the main paper using ViT-Base. further indicating CoLA's effectiveness in accumulating and exploiting learned knowledge with diverse model architectures.
Table K: Effectiveness of CoLA on ResNet- 50 in the lifelong TTA scenarios following Table 1.
Time:
Round
3
4
5
6
7
8
9
10
Average
NoAdapt
18.0
18.0
18.0
18.0
18.0
18.0
18.0
18.0
18.0
18.0
18.0
CoTTA [6
33.4
23.6
9.6
2.2
1.2
1.2
1.2
1.2
1.2
1.2
7.6
SAR 39
35.9
17.5
36.2
18
13.6
35.8
16.5
13.5
36.1
24.4
CoLA (Ours)
39.4
42
43
43.5
43.9
44.4
44.7
45
45.2
45.3
43.6
ETA 38
42.4
40.4
38.9
37.6
37
36
35.9
35.2
35.1
34.7
37.3
CoL (Ours)
46.5
46.5
49.5
49.5
49.7
49.8
49.9
49.9
49.9
49.8
49.3(412.0)
EATA [38]
47.5
47.3
47.1
46.7
46.8
46.6
46.4
46.3
46.2
46.2
46.7
CoLA (Ours)
48.2
49.5
50.0
50.1
50.2
50.2
50.3
50.3
50.3
50.2
49.9(+3.2)
26

---

Effectiveness of CoLA in sample efficiency on unseen distributions. We validate the effectiveness of Eqn. (2) to facilitate sample-efficient TTA on unseen distributions. From Figure [ CoLA consistently enhances the sample efficiency and the overall performance, i.e, with an up to 30.0x speed up on both ImageNet-R and ImageNet-Sketch, indicating that the effectiveness of CoLAA not limited to previously encountered distributions. More interestingly. compared with SAR, CoLA helps mitigate overfitting on ImageNet-Sketee This can be attributed to that, in the unsupervised adaptation, SAR learns more erroneous predictions since its performance is particularly limited at the beginning of adaptation. This phenomenon further demonstrates the importance of sample efficiency and our effectiveness in leveraging shared knowledge for efficient TTA.
(a) Comparison on ImageNet-R
Comparison on ImageNet-Sketdh
40
55
Accuracy (%)
30.0x Speedup
50.0x Speeddi
Accuracy
50
30
SAR SAR CoLA (Ours)
SAR
SAR CoLA (Ours)
45
20
1K
10K
20K
30K
1k
10k
20k
30k
40k
50k
# #Online Samples
#Online Samples
Figure B: Additional comparisons w.r.t sample efficiency on unseen distributions. Here, CoLA leverages learned weights ImageNet-C (i.e., from SAR+CoLA in Table the fifth row) while the effectiveness in sample efficiency is evaluated on ImageNet-R -R and ImageNet-Sketch.
Effectiveness of CoLA in mitigating error accumulation. We provide visualization of the learned 3 in Figure Here, all domain vectors are learned on the same domain, i.e., Gaussian Noise, where save learned weights post-adaptation to 10 batches of samples. Generally, later saved domain vector should be prioritized as it learns on more samples. Nevertheless, in the context of unsupervised TTA, model may learn from erroneous pseudo-labels, where the performance would significantly deteriorate, known as error accumulation. From Figure CoLA adaptively assigns lower 3; on domain vectors that may have accumulated error, e.g., from the 80-th the 90-th domain vectors, according to loss optimization. Thus, CoLAA demonstrates potential effectiveness in mitigating error accumulation, enhancing ETA's performance by +7.6% on ImageNet-C under label distribution shifts.
Value of B,
3
20
40
60
80
100
120
140
160
i th domain vector
Figure C: Visualization of 9; for ETA+CoLA on ImageNet-C(severie level 5, Gaussian) under online imbalanced label distribution shifts. CoLA saves learned weights for every adaptation to batches of samples while no weights are discarded for visualization. Learned temperature Ti is 0.49.
Effectivvness of our domain shift detector. We provide additional results to demonstrate the effectiveness of our domain distance function, i.e., Eqn. (7), for shift detection. In our experiments,
27

---

we consistently set Ou to 0.1 for shift detection. From Figure D our domain shift detector demonstrates sensitivity between corrpptions from different groups, e.g.. with a distance of 99 between 'impulse noise' and "contrast', and distance of 32 between "gaussian noise' and 'defocus blur'.
10f
gaussian noise
0.0058
0000
8.6
75
1.1
2.2
shot noise- 0.0058
0.
0066
34
21
11
9.2
0.74
1.5
26
0.91
1.2
2.4
1.3
impulse
0.0066
15
10
defocus blur-
0.44
2.2
glass ass blur-
21
20
0.2
0.059
0.11
3.7
1.9
0.22
2.7
2.3
2.1
1.1
2.1
109
11
zoom blur-
8.6
9.2
8.6
0.44
0.11
0.016
1.6
0.76
0.35
1.2
3.8
0.83
0.36
0.9
snow
0.65
0.74
0.64
6.8
3.7
2.1
1.6
0.17
3.8
0.11
26
0.17
0.41
0.17
10-1
fog
16
15
0.22
0.22
0.27
0.35
1.8
3.1
2.5
1.4
2.7
16
brightness
0.91
0.8
0.9
2.7
1.5
1.2
0.11
0.17
3.1
0.32
0.26
0.13
elastic
1.1
1.2
11
2.1
1.1
0.83
0.17
0.21
2.5
0.12
0.11
0.008
-1-->
pixelate
2.2
2.4
22
22
0.51
0.36
0.41
0.2
1.4
0.26
11
0.31
0.14
JPEG
gaussian noise
shot noise
impulse noise-
defocus blur-
blur-
motion blur-
zoom bur-
snow-
fag
brightness.
elastic
pixelate
JPEG-
Figure D: Effectiveness of our domain distance function, i.e., D(...) defined in Eqn. (7). to capture the magnitude of domain shift. Here, estimate the domain distance between each corruption in ImageNet-C (level 5, with 15 corruptions). The statistic of each domain estimated via Eqn. (6).
Statistical comparison. We re-run Table 2 in the main paper with different seeds and report the mean and std of each method in Table L The results show that CoLA performs stably with small stds and it lowers the ETA&DeYo, suggesting CoLA's 's stability.
Table L: Statistical comparison. Experiments follow the settings of Table
Device (Adapt ++
Device (Adapt +)
Device (Adapt -+)
Method
Noise Blur Weat.
Digit.
Blur Noise
Digit.
Weat.
Weat.
Digit.
Blur
Noise
Avg.
ETA 38
55.1
56.8
67.3
58.5
59.7
51.6
52.2
66.0
70.4
67.8
57.9
49.5
60.2+1.8
+ CoLA (Ours)
55.1
59.7
70.3
68.7
59.5
56.3
65.2
70.3
70.3
68.0
59.2
54.6
63.1
SAR
50.3
54.4
66.3
64.7
55.1
48.0
63.9
66.5
66.5
64.4
55.6
47.0
58.6.0.1
+ CoL.A (Ours)
50.3
57.9
69.1
68.1
55.0
54.8
67.2
70.1
66.4
65.1
58.7
54.5
61.420.3
DeYO 25
56.3
50.5
67.6
67.9
55.4
44,9
53.5
55.4
71.1
68.9
52.0
26.6
55.8
CoLA (Ours)
56.2
54.7
70.4
69.2
54.1
55.9
68.8
70.6
71.1
68.7
52.4
54.2
62.2
28

---

Limitations and future works
Finetuning LLM. We mainly verify our CoLA with vision models in the context of test-time adaptation. Since our formulation in Eqn. (2) does not necessitate being optimized during testing, it's an interesting future work to verify CoLA on more scenarios. For instance, finetuning the large language models, where multiple finetuned weights, e.g., using LoRA (20), are publicly available.
Shrinking the shared vectors. Our CoLA continuously expands the size of the shared domain vectors across devices. Intuitively, on the same domain, one can reduce memory consumption by preserving only the best-performing domain vector. Although have demonstrated feasible strategy for shared vectors shrinking on the single-domain adaptation, where we discard the unused ones according a, Table Nevertheless, it's still challenging to perform share vectors shrinking across multiple devices and we leave it for future works.
G Broader impact
This paper aims to advance the field of test-time adaptation for out-of-distribution generalization. The societal impact of our work lies primarily in its potential to expand the usability of machine learning models in real-world settings, particularly on self-driving cars, embodied agents/robots, etc. By enhancing the performance of machine learning models various real-world devices, our method helps make AI technology more broadly accessible. Ethically, our approach eliminates the need for data transfer between devices, thereby improving data privacy and security.
29

---

NeurIPS Paper Checklist
1. Claims
Question:
Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: This paper proposes CoLA, a collaborative lifelong adaptation framework for deep models, enabling efficient cross-device domain adaptation by sharing and accumulating knowledge across devices during test-time adaptation, significantly enhancing performance in varying domain scenarios without sacrificing efficiency.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation as long as is clear that these goals attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification:
We have discussed the limitations of our methods in Appendix F
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect the scope of the claims made, e.g., if the approach was only tested on a few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
30

---

Answer:
[Yes]
Justification:
We have provided a proposition and its complete proof in Section B
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proof's can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided not)?
Question:
Answer: [Yes]
Justification: We have fully disclosed all information needed to reproduce the main experi- mental results of the paper in Appendix C
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, descríbing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm. (b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is new model (e-g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should possible for other researchers to have some path to reproducing verifying the results.
5. Open access to data and code
31

---

Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Question:
Answer: [Yes]
Justification:
We have provided all source code in the introduction.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips... cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https
/ "cipps.c/ppulli//guu /ttoe//uuuddeeoooouuuubiiiiiii ssionPolicy) for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (c.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
We have provided detailed experimental settings in Appendix
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [Yes]
Justification: Error bars are reported in Appendix
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
32

---

The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative enror rates).
If error bars reported in tables plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information the puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: We have provided detailed information about our compute resources in Ap- pendix C
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary failed experiments that didn't make into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics
Answer: [Yes]
Justification:
The paper meets the NeurIPS Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special cireumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g.. if there is a special consid- eration due to laws or regulations their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification:
We have discussed this in Appendix G
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
33

---

The conference expects that many papers will be foundational research and not tied particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [No]
Justification:
This paper poses such risks.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
The creators or original owners assets (e.g., code, data, models) used the paper are properly credited, and the license and terms of are explicitly mentioned and properly respected.
Justification:
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version the asset used and, possible, include URI.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g.. website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms use in package should be provided. For popular datasets, paperswithcone com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
34

---

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's s creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [No]
Justification: We will release assets upon acceptance.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details the dataset/code/moodl as part their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included the main paper.
According the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country or institution) were obtained?
Answer: [No]
Justification: The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
35


Reviews:
[{"review_id": "rcBHwIPLRV", "reviewer": "Reviewer_sr9b", "summary": "Prior test-time adaptation (TTA) methods focused on single-device setups only, despite models being deployed to multiple devices in the real world. This paper presents Collaborative Lifelong Adaptation (CoLA) to leverage this situation and improve performance. Specifically, this paper introduces Principal Agents and Follower Agents concepts and proposes knowledge reprogramming, similarity-based knowledge aggregation, and automatic domain shift detection for efficient collaboration. Experiments with various settings and state-of-the-art (SOTA) TTA methods demonstrate the effectiveness and applicability of CoLA.", "questions": "1. What is the impact of this initialization strategy compared to randomized/equal weights?\n2. Where does the communication cost reduction come from? Is it only from utilizing BN layers?\n3. What is the accuracy of automatic domain detection?\n4. Why is CoLA better than the baseline in the first round as well, before the shift happens (Table 1)?", "limitations": "The authors adequately addressed the limitations.", "rating": 5, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. Interesting problem of utilizing knowledge from multiple devices for collaborative TTA.\n2. The method considers practical scenarios such as communication cost, devices without training capability, etc.\n3. The method is plug-and-play with existing TTA methods.", "weaknesses": "1. Lack of deeper analysis on when prior knowledge helps and when it harms. As environments are diverse and KL-divergence-based distance metrics are not always accurate in neural networks, utilizing knowledge from other devices might be harmful. It would be better to have a deeper investigation into when prior knowledge is beneficial, how much prior knowledge is required to get the benefit, the extent of the benefit, and how many such cases are observed in the experiments.\n2. The domain shift setup is a bit unrealistic. In real-world scenarios, domain shifts are often gradual rather than sudden. It is unclear whether the proposed method can handle gradual domain shift scenarios.\n3. No error bars are reported, which weakens the statistical significance of the results.\n4. This submission doesn't include code, which limits the validity and transparency of the work.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "> Q1. Analyses on when prior knowledge helps and when it harms.\n\nA. **Analyses on when prior knowledge helps**: In our paper, we seek to improve test-time model adaptation performance and efficiency by exploiting useful knowledge from multiple devices in the application environment. In practice, different devices may frequently encounter similar (e.g., Gaussian blur and Motion blur) or even identical testing domains. For these cases, our CoLA paradigm can reprogram these similar/useful knowledge via learnable re-weighting term when similar domain data are encountered. This enables CoLA to adapt quickly while simultaneously enhancing adaptation performance.\n\nIn fact, prior knowledge does not help only if all knowledge/domain vectors learned from previously encourtered domains perform no better than the source pretrained model, which is however unrealistic given the similarities in real-world environments.\n\n**Analyses on when prior knowledge harms**: Note that our paradigm conducts model adaptation using prior knowledge to minimize the test-time objective. **If the prior knowledge of some devices is harmful, using such knowledge shall not decrease the test objective.** Thus, such prior knowledge will not be used for model adaptation. In this sense, our method is stable even with harmful prior knowledge (*e.g.*, the domain knowledge in some devices are not helpful for principal agents to do TTA). Our stability is also benefited from our initialization, please refer to Q5 for more discussions.\n\n\n> Q2. CoLA's effectiveness under gradual domain shift (GDS) scenarios.\n\nA. CoLA works well under GDS scenarios as shown in Table C. Here, we follow the benchmark [A,B] to evaluate our CoLA under GDS, in which the severity of each corruption gradually changes during a lifelong adaptation process. From the results, CoLA effectively enhances the accuracy of ETA by 1.8%.\n\nTable B. Accuracy of CoLA on ImageNet-C under gradual domain shifts [A,B].\n|NoAdapt|ETA|+ CoLA(Ours)|\n|:-:|:-:|:-:|\n|53.2|71.2|73.0|\n\n[A] Gradual test-time adaptation by self-training and style transfer\n\n[B] Robust Mean Teacher for Continual and Gradual Test-Time Adaptation\n\n\n> Q3. No error bars are reported.\n\nA. Please refer to Q5 for Reviewer L5Gu.\n\n> Q4. Code Release.\n\nA. We will release the source code upon acceptance.\n\n> Q5. What is the impact of this initialization strategy compared to randomized/equal weights?\n\nA. Our initialization strategy helps stabilize TTA, particularly when the shared domain vectors contain harmful prior knowledge. Randomly or equally initializing $\\alpha_i$ may select harmful knowledge that deteriorates model performance, as in Table C. In contrast, our strategy achieves consistent adaptation performance even when the proportion of harmful parameters exceeds 96%, demonstrating its superiority. Additionally, this strategy also encourages fast adaptation in the early phase of new domain data arrival.\n\nTable C. Effects of different initialization strategies w.r.t. $\\alpha$. CoLA uses only N harmful prior knowledge (random weights). Results (Accuracy) obtained on ImageNet-C (Gaussian, level 5) with ETA+CoLA.\n||N=0|N=2|N=4|N=100|N=1000|\n|-|:-:|:-:|:-:|:-:|:-:|\n|Ours|51.97|52.02|52.04|52.04|52.04|\n|Random $\\alpha$|51.97|10.46|0.15|0.1|0.1|\n|Equal  $\\alpha$|51.97|0.1|0.1|0.1|0.1|\n\n> Q6. Where does the communication cost reduction come from? Is it only from utilizing BN layers?\n\nA. The reduction comes from both **transmitting Norm Layers** and **transmitting only when a domain shift occurs**. For example, using ViT and ImageNet-C data, CoLA requires only 0.15MB per domain (with accurate domain shift detection). In contrast, the previous method, CEMA, transmits both raw data and norm layers for each iteration, resulting in a communication cost of about 568.85MB for a single domain.\n\n> Q7. What is the accuracy of automatic domain detection?\n\nA. Our detector achieves 100% domain detection accuracy in Table 1 of the main paper (single-device lifelong adaptation) and 71.4% in Table 2 of the main paper (multi-device TTA). However, we emphasize that the lower detection accuracy in Table 2 does not compromise CoLA's performance. As in Table D, CoLA achieves the comparable final accuracy as CoLA with an oracle domain detector. This is because certain domains in Table 2 share similarities. Therefore, it is not necessary to identify and save all domain vectors accurately, which helps reduce redundant domain vector storage.\n\n\nTable D. Avg accuracy comparison with CoLA+[Oracle domain shift detector]. Experiments follow the settings of Table 2 in main paper.\n|ETA|ETA+CoLA(Ours)|ETA+CoLA(Oracle)|\n|:-:|:-:|:-:|\n|61.3|63.7|63.7|\n\n\n> Q8. Why is CoLA better than the baseline in the first round as well, before the shift happens (Table 1)?\n\nA. This is because each round consists of 15 different corruptions, and each element in Table 1 represents their average result. The 15 corruptions fall into four main types: noise, blur, weather, and digital. Within each category, the corruptions share certain similarities. Thus, for example, adaptation to Impulse Noise can benefit from knowledge learned on Gaussian Noise. CoLA leverages this similar knowledge to aid subsequent adaptation, thus achieving better performance.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "PmRmfns8RF", "reviewer": "Reviewer_1r2f", "summary": "This paper considers test-time adaptation in multi-device scenarios and proposes Collaborative Lifelong Adaptation (CoLA). Domain knowledge vectors, as interactive objects on devices, are maintained and stored by measuring KL-divergence between the current test batch and the online estimated distribution. This paper categorizes different devices into principal agents and follower agents based on computational resources and latency demands, and uses different strategies for online prediction or parameter updates during testing. Experiments in various TTA settings demonstrate that CoLA has good adaptability and anti-forgetting capabilities. Rigorous ablation studies well support the viewpoints in the paper.", "questions": "It appears that domain knowledge vectors play a pivotal role in CoLA. Apart from using the affine parameters of norm layers, are there any other potential methods to determine these vectors?  (Just in terms of discussion)\n\nI hope that the code can be released.", "limitations": "Yes", "rating": 7, "confidence": 5, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "CoLA is simple and efficient, demonstrating excellent performance in collaborative, lifelong, and single-domain TTA tasks. It also considers the application for devices with different computational resources and latency demands in practice. This paper is well-organized and easy to follow.", "weaknesses": "- The hyperparameter z is used for detecting distribution shift and influences the storage of new domain vectors. The paper lacks ablation studies on the threshold z.\n- In the setting of lifelong TTA, the smooth and stable updating of the affine parameters of norm layers relies on a sufficiently large batch size. It is necessary to analyze the robustness of CoLA under conditions where the batch size is extremely small.\n- It’s insufficient to use only ViT-Base as the source model in the main experiments. Analyzing the effects of CoLA on different source models can provide more comprehensive evidence, such as ResNet and MobileNet.\n- (minor) The experiments are all targeted at classification. It is intriguing to explore the effectiveness of CoLA in detection and segmentation tasks.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "> Q1. Ablation studies on the threshold z.\n\nA. CoLA remains effective among a wide range of threshold z, as shown in Table A. From the results, while a stricter threshold saves more domain vectors, CoLA achieves a stable performance of around 64.7%. When threshold z increases, CoLA saves significantly fewer domain vectors and still enhances the performance significantly, *i.e.*, the average accuracy of 62.2\\% in ETA+CoLA (z=10) *vs.* 46.4% in ETA.\n\n\n\nTable A. Sensitivity of threshold z. Experiments follow the settings of Table 1 in the main paper, i.e., single-device lifelong adaptation, and CoLA is incorparated with ETA. We report average accuracy over 10 rounds with each round consists of 15 corruptions of ImageNet-C. The average accuracy of ETA baseline is 46.4%. \n||CoLA, z=0.01|CoLA, z=0.05|CoLA, z=0.1|CoLA, z=1|CoLA, z=10|\n|-|:-:|:-:|:-:|:-:|:-:|\n|Avg Acc.|64.7|64.6|64.8|63.8|62.2|\n|#Saved domain vectors|20740|369|169|110|48|\n\n\n> Q2. The robustness of CoLA under small batch sizes.\n\n\nA. The stability of CoLA under small batch sizes is primarily determined by the base algorithms, such as ETA and SAR, rather than CoLA itself. This is because CoLA is a plug-and-play module designed to be incorporated with existing methods. In the response, we provide further empirical results to verify CoLA under small batch sizes. From Table B, SAR+CoLA achieves consistent results across different batch sizes, without performance degradation as the batch size reduces from 16 to 2. These results show that CoLA is less sensitive to small batch sizes.\n\n\nTable B. Performance of CoLA under various batch sizes. We follow the same settings of Table 2 in the main paper and report **average accuracy** over all devices and corruptions here.\n|Method|BS=16|BS=4|BS=2|\n|-|-|-|-|\n|SAR|58.8|58.9|58.7|\n|**SAR+CoLA**|61.6|61.7|61.7|\n|ETA|60.5|58.5|57.1|\n|**ETA+CoLA**|62.8|61.7|61.5|\n\n> Q3. Analyzing the effects of CoLA on different source models, such as ResNet.\n\nA. Table C demonstrates the effectiveness of our CoLA on ResNet-50, where CoLA updates and stores the affine parameters of batch normalization layers in ResNet.\n\nFrom Table C, CoLA consistently enhances the performance of ETA/EATA/SAR throughout 10 rounds of adaptation and addresses the issue of performance degradation in long-term adaptation. These results are consistent with Table 1 in the main paper using ViTBase, further indicating CoLA's effectiveness in accumulating and exploiting learned knowledge for lifelong adaptation.\n\nTable C.  Effectiveness of CoLA on ResNet in the lifelong TTA scenarios. We follow the same settings as Table 1 in the main paper.\n|Round|1|4|7|10|Avg (over 10 rounds)|\n|-|:-:|:-:|:-:|:-:|:-:|\n|NoAdapt|18.0|18.0|18.0|18.0|18.0|\n|CoTTA|33.4|2.2|1.2|1.2|7.6|\n|ETA|42.4|37.6|35.9|34.7|37.3|\n|**ETA+CoLA**|46.5|49.5|49.9|49.8|49.3|\n|EATA|47.5|46.7|46.4|46.2|46.7|\n|**EATA+CoLA**|48.2|50.1|50.3|50.2|49.9|\n|SAR|35.9|36.2|35.8|36.1|24.4|\n|**SAR+CoLA**|39.4|43.5|44.7|45.3|43.6|\n\n\n\n> Q4. The experiments are all targeted at classification. It is intriguing to explore the effectiveness of CoLA in detection and segmentation tasks.\n\n\nA. Thank you for your valuable suggestion. We fully agree that applying CoLA to more tasks is very intriguing. However, due to the time limits of the rebuttal period, we plan to further explore this in the future.\n\n\n> Q5. Apart from using the affine parameters of norm layers, are there any other potential methods to determine these vectors? (Just in terms of discussion)\n\nA. Thanks for your interesting question. In addition to affine parameters of norm layers, CoLA can utilize any parameter-efficient fine-tuning methods to select learnable parameters as the knowledge vectors to maintain high memory efficiency. This includes LoRA-based methods and Prompt Tuning methods. Table 7 in our main paper also verifies the effectiveness of CoLA with prompt tuning, enhancing TPT's accuracy by 1.5% on ImageNet.\n\n\n> Q6. I hope the code can be released.\n\nA. Yes, we will release our code as soon as the paper is accepted.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Z0GWbvfhDc", "reviewer": "Reviewer_SsoW", "summary": "This paper explores test-time adaptation (TTA) in scenarios involving varying computational resources across multiple devices. To address this, the paper proposes storing and sharing domain knowledge among devices to improve adaptation. For resource-abundant devices (referred to as principal agents), domain knowledge is either newly learned or reprogrammed based on current inputs. Conversely, for resource-limited devices (referred to as follower agents), existing domain knowledge is efficiently aggregated according to domain similarities. The proposed method demonstrates superior performance across diverse experimental settings.", "questions": "1. Have you considered moving Figure A from the supplementary material to the main document? Figure A could help readers quickly grasp the overall concept of the proposed method.\n\n2. Could the proposed methods be applied to the EATA method? Given that EATA is a strong baseline on several benchmarks, the synergy of the proposed method and EATA might set a new state-of-the-art.", "limitations": "This paper faithfully describes its limitations and potential negative impacts in general.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. The proposed methods are technically sound.\n2. The paper is well-organized and easy to understand.\n3. Extensive analysis of various aspects and scenarios is provided, aiding in a better understanding of the proposed framework.", "weaknesses": "1. Consideration of Multi-Device Test-Time Adaptation Literature:\n\n- While multi-device test-time adaptation literature is mentioned in the supplementary material, there are no direct mentions or experimental comparisons in the main paper. This omission may confuse readers about the paper's originality.\n- The multi-device test-time adaptation literature should be more accurately described in the main paper, including conceptual comparisons in the introduction section.\n- Benchmarking multi-device test-time adaptation methods (e.g., Cloud-Edge TTA [4], CEMA [5], FedTHE+ [21]) within the same experimental settings is crucial to demonstrate the superiority of the proposed method. The concepts of cloud and edge in [4,5] align with the notions of principal and follower agents in this paper.\n\n2. Similarity to Prior Continual Test-Time Adaptation Work [A]:\n\nThis paper and prior continual test-time adaptation work [A] show high similarity in the following aspects:\n- Both papers store and reuse domain knowledge for better adaptation.\n- Both store domain knowledge in the affine parameters of norm layers.\n- Both retrieve related domain knowledge based on the measured domain similarity between the current input and previous domains.\nGiven these similarities, it is essential to correctly reference the prior work [A] and include experimental comparisons on detailed design aspects.\n[A] Test-time Adaptation in the Dynamic World with Compound Domain Knowledge Management, IEEE Robotics and Automation Letters, 2023\n\n3. Additional Ablation Study and Analysis:\n\nTo improve the completeness of the paper, I suggest including the following:\n- An ablation study on the impact of domain knowledge \"stored from multi-devices\" in multi-device scenarios. A baseline where each device has restricted access to its own domain knowledge. Comparing this baseline to the proposed method would highlight the impact of domain knowledge \"stored from multi-devices.\"\n- An analysis of the proposal's efficiency in terms of memory overload and inference time.\n- An analysis of the ratio of selected domain knowledge for reprogramming/aggregation.\n- Experimental results on the diverse order of domains for multiple devices.\n\n4. Limited Representation of Real-World Scenarios:\n\nWhile this paper targets more realistic scenarios, the experiments are somewhat limited in representing real-world conditions. It would be beneficial to include experimental results from (for example) autonomous driving scenarios under diverse weather conditions.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "> Q1. Comparison with Multi-Device Test-Time Adaptation.\n\nA. Please refer to our General Response.\n\n> Q2. Differences between CoLA and previous continual TTA work TTA-CDKM [A]. \n\nA. The main differences between our CoLA and TTA-CDKM [A] are in the following aspects:\n\n1. **Our CoLA is more feasible for adaptation on both computational resource-abundant and resource-limited devices than TTA-CDKM [A]**: TTA-CDKM [A] mainly focus on the adpdation on **one single resource-abundant device** in lifelong TTA scenarios. While our CoLA addresses the practical computation demands of both resource-abundant and resource-limited devices. More importantly, our CoLA facilitates a collaborative TTA community **across multiple devices with diverse resource budgets** rather than a single device.\n2. **Our CoLA is more suitable for bandwith-limited senarios than TTA-CDKM [A]**. TTA-CDKM [A] updates the stored domain parameters when adapting to **each batch of test samples**. This introduces intensive communication cost for weight synchronization across devices post-adaptation. Conversely, our CoLA synchronizes the stored domain parameters only **when domain shift occurs**. It largely reduces the communication overhead to enable multi-device collaboration.\n3. **Our CoLA makes better use of off-the-shelf parameters than TTA-CDKM [A]**. TTA-CDKM [A] assumes that all parameters are learned from domain adaptation and retrieves based on domain similarity, **failing to exploit off-the-shelf parameters without domain information**. In contrast, our CoLA adaptively learns the weighting $\\alpha_i$ without relying on domain information. This enables our CoLA to **exploit more off-the-shelf parameters for rapid adaptation**, as shown in Table 7 in the main paper.\n\nWe will discuss TTA-CDKM [A] in our revised paper. Due to the unavailability of TTA-CDKM [A] source code and the limited time, we did not include empirical comparisons with TTA-CDKM and leave this to the future.\n\n\n> Q3-1. An ablation study on the impact of domain knowledge \"stored from multi-devices\" in multi-device scenarios.\n\nA. We ablate the effects of knowledge \"stored from multi-devices\" in Table A, termed CoLA (NoShare). Here, CoLA (NoShare) only store the knowledge learned on its own device and do not share knowledge with other devices. As in this multi-device setting each device encounters fully different corruptions, SAR/ETA+CoLA (NoShare) achieves the similar performance as SAR/ETA. By leveraging knowledge from other devices, CoLA consistently boosts the performance of SAR and ETA, suggesting the importance of cross-device knowledge sharing.\n\nTable A. Ablation of cross-device knowledge sharing in CoLA. The experiments follow **the same settings as Table 2 in the main paper**. Due to page limits, we will include the *detailed results for each device with each corruption in the revision*.\n\n|Method|Avg. Acc.|\n|-|:-:|\n|NoAdapt|28.6|\n|SAR|58.6|\n|SAR+CoLA(NoShare)|58.6|\n|**SAR+CoLA(Ours)**|**61.7**|\n|ETA|61.3|\n|ETA+CoLA(NoShare)|61.3|\n|**ETA+CoLA(Ours)**|**63.7**|\n\n> Q3-2. An analysis of the proposal's efficiency in terms of memory overload and inference time.\n\nA. Our CoLA is both computation and memory efficient. In Table 5 of the main paper, we have provided latency and memory usage comparisons, in which both CoLA's follower and principal agents boost the performance a lot while maintaining nearly the same efficiency as NoAdapt/ETA baseline.\n\nIn the response, we further illustrate CoLA’s efficiency with an increasing number of domain vectors. From Table B, CoLA efficiently scales to over 10,000 domain vectors, incurring only an additional 11s of runtime and 1,502MB of extra memory, yet remains substantially more efficient than CoTTA. We believe that 10,000 domain vectors should be adequate for handling most real-world applications with proper management.\n\n\n\nTable B. Efficiency comparison on ImageNet-C (Gaussian, level 5). The time is counted for processing 50,000 images on a single A100. $N$ is the number of domain vectors, which we initilize them as random weights in this table. \n|Method|ETA|+CoLA(N=1)|+CoLA(N=100)|+CoLA(N=10,000)|CoTTA|\n|-|:-:|:-:|:-:|:-:|:-:|\n|Time (s)|109|110|112|120|937|\n|Memory (MB)|7,433.2|7,433.7|7,448.2|8,935.5|21,628.6|\n\n\n> Q3-3. An analysis of the ratio of selected domain knowledge for reprogramming/aggregation.\n\n- **On Principal Agents (knowledge reprogramming):** When adapting to a new domain, CoLA favors knowledge that learns from a similar domain for rapid adaptation. However, as it captures more knowledge on the current domain, CoLA adaptively allocates more ratio to diverse domains for complementation (refer to Fig. I in our rebuttal PDF).\n- **On Follower Agents (knowledge aggregation):** With accurate domain similarity measurement, follower agents select relevant knowledge for seen domains. For unseen domains, CoLA assigns ratio more evenly while still prioritizing knowledge from a similar domain (refer to Fig. II in our rebuttal PDF).\n \n\n\n\n> Q3-4. Experimental results on the diverse order of domains for multiple devices.\n\nA. From Table I in our PDF file in Global Response, CoLA consistently outperforms baselines under 3 different domain orders, suggesting CoLA's stability.\n\n> Q4. Experiments under real-world scenarios.\n\nA. The datasets adopted by EATA effectively meet real-world demands. For instance, the ImageNet-A/V2/R datasets in Tables 6-7 are sourced from the real world. Additionally, the corruptions present in ImageNet-C are commonly encountered in real-world scenarios, such as when cameras degrade, weather changes, or cameras move. All these datasets are widely used in the TTA literature. Due to the time constraints of the rebuttal period, we will explore CoLA in more real-world scenarios in the future.\n\n> Q5. Could the proposed methods be applied to the EATA method?\n\nA. Yes, CoLA enhances the accuracy of EATA by 3.2% on ResNet-50. Please refer to Reviewer 1r2f's Q3 for more details.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "XzeAOuCfja", "reviewer": "Reviewer_L5Gu", "summary": "This paper proposes a novel Collaborative Lifelong Adaptation (CoLA) method for test-time adaptation (TTA) across multiple devices. The key idea is to accumulate and share knowledge learned during adaptation across devices, enabling more efficient and effective adaptation to distribution shifts at test time. The authors evaluate CoLA in various scenarios, showcasing the effectiveness of the plug-and-play CoLA method.", "questions": "1. Please explain details on the limitations above.\n1. What are the overheads of the shared domain vectors? Can you provide any numerical example of the growth of shared vectors?\n1. Given that CoLA involves optimizing both $\\alpha$ and $\\Delta \\theta$, how does the method achieve such minimal computational overhead (only a 3-second increase) compared to the baseline ETA, as reported in Table 5?\n\n**Important**: I will raise the ratings if concerns are resolved.", "limitations": "The paper properly discussed the limitations and societal impacts.\nStill, it's worth mentioning the limited applicability and scalability of continuously stacking shared domain vectors, which is discussed by the authors.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "1. The authors propose a new and novel setting of cross-device collaborative TTA.\n1. The authors conducted intensive experiments with transformer-based models on various scenarios.", "weaknesses": "Although the paper proposes an innovative research direction, I have concerns about its positioning and presentation of the results.\n\n1. **Limited comparisons with test-time federated learning [1, 2]**. Collaborative TTA could be considered as a variation of federated learning. Please thoroughly discuss the relation with federated learning. The paper should thoroughly discuss this relationship and provide comparative experimental results if possible.\n1. **Insufficient scalability analysis**. The paper does not address how the method scales with an increasing number of devices and domains, particularly regarding the growth of the shared domain vector set T.\n1. **Vague approach to lifelong adaptation**. While the paper claims to address lifelong adaptation, it doesn't fully explain how the method performs in lifelong scenarios. It's unclear how the proposed methods tackle potential performance or memory issues arising from accumulating domain vectors over time.\n1. **Incomplete experimental details**:\n   - Lack of explanation on the detailed experiment settings, including the details of resource-abundant and resource-limited devices.\n   - The authors only applied CoLA on ETA, SAR, and DeYO. Please discuss why these specific methods were chosen and whether they are applicable to other TTA methods.\n   - The experiment results use different baselines on each table; please elaborate on the baseline selection criteria.\n1. **Lack of statistical significance**. The paper does not report any error bars of the experimental results.\n\n\n\n[1] Bao, Wenxuan, et al. \"Adaptive test-time personalization for federated learning.\" Advances in Neural Information Processing Systems 36 (2024).\n\n[2] Jiang, Liangze, and Tao Lin. \"Test-Time Robust Personalization for Federated Learning.\" The Eleventh International Conference on Learning Representations.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "> Q1. Comparisons with test-time federated learning. \n\nA. Please refer to our General Response.\n\n> Q2. More scalability analysis regarding CoLA.\n\nA. CoLA scales well with an increasing number of devices, i.e., with more shared domain vectors. From Table A, ETA/SAR+CoLA consistently benefits from additional participating devices, e.g., ETA+CoLA achieves an accuracy of 65.2% with 11 devices compared to 61.8% with just one device. This highlights the importance of cross-device collaboration and demonstrates CoLA’s effectiveness under more large-scale multi-device collaborative TTA.\n\n\nTable A. Effectiveness of CoLA with an increasing number of principal devices (with back-propagation capability). Here, each device continuously encounters 15 domains from ImageNet-C (level 5) in different domain orders. \n|#Devices|1|3|5|7|9|11|\n|-|:-:|:-:|:-:|:-:|:-:|:-:|\n|SAR+CoLA|59.7|62.1|62.8|63.6|63.8|63.9|\n|ETA+CoLA|61.8|63.9|64.3|64.9|65.0|65.2|\n\n\n\n> Q3. More clarifications of CoLA for lifelong adaptation and its scalability with accumulating domain vectors.\n\nA. For lifelong adaptation, CoLA automatically detects domain changes and then explicitly saves each domain vector once domian is changed for future reuse. When similar domain data reappears, CoLA can directly reuse the previously encountered knowledge through reprogramming, i.e., learning a weight $\\alpha$ to reweight existing domain vectors while simultaneously learning new domain vectors through gradient optimization, thereby overcoming catastrophic forgetting.\n\nAccording to Q2, Reviewer sr9b's Q6, and Reviewer SsoW's Q3-2, CoLA scales well in terms of both performance and efficiency with accumulating domain vectors. \n\n\n> Q4-1. Lack of explanation on the detailed experiment settings, including the details of resource-abundant and resource-limited devices.\n\nA. We have provided detailed settings for each experiments in Appendix D.2 and do not include them in the main paper due to page limits. We will make this clearer in the main paper of the revision.\n\nFor resource-abundant/limited devices, we term devices with backpropagation capacity, like GPU servers, as resource-abundant. Devices without backpropagation capacity or requiring very low latency, such as smartphones and surveillance cameras, are considered resource-limited. However, in our experiments, we simulate these devices using GPUs, i.e., run all experiments on GPUs, which does not affect the fairness of comparisons.\n\n\n\n\n\n> Q4-2. The authors only applied CoLA on ETA, SAR, and DeYO. Please discuss why these specific methods were chosen and whether they are applicable to other TTA methods.\n\nA. We select ETA [ICML 2022] and SAR [ICLR 2023 Oral] as they are strong baselines on various benchmarks and widely used by existing TTA literature. Moreover, DeYO, accepted as ICLR 2024 spotlight, is one of the most advanced TTA method. \n\nCoLA is applicable but not limited to the above methods. This is because compared to conventional TTA objective, CoLA introduces only an adaptive aggregation term in optimization, $\\sum\\alpha_i\\Delta\\theta_i$, where $\\Delta\\theta_i$ are frozen and only $\\alpha_i$ are learnable. **We do not limit the loss funtion to optimze these $\\alpha_i$.** Thus, CoLA can integrate and benefit from the loss designs of the advanced TTA solutions. \n\nPlease also refer to Reviewer 1r2f's Q3 in which CoLA works well with EATA.\n\n\n> Q4-3. The experiment results use different baselines on each table; please elaborate on the baseline selection criteria.\n\nA. We selected baselines based on whether the method relies on backpropagation, aligning with the computational resources of each device.\n\nIn Tables 1-2 of the main paper, experiments are conducted on principal agents (with backpropagation capability), and thus the baselines are all backpropagation-based. In Table 3 of the main paper, the experiments are conducted on resource-limited devices (follower agents), in which backpropagation-based methods are infeasible. Therefore, all baselines in Table 3 are forward-only methods.\n\n\n> Q5. Error bars of the experimental results.\n\n\nA. We re-run Table 2 in the main paper with 5 different seeds and report the mean and std of each method in Table B. The results show that CoLA performs stably with small stds and it lowers the std of ETA\\&DeYo by alleviating the model collapse issue during a long-term adaptation process, suggesting CoLA's stability.\n\nTable B. Statistical comparison. Experiments follow the settings of Table 2 in the main paper, i.e., multi-device TTA. The accuracy is averaged over all devices and corruptions.\n|Method|Avg. Acc.|\n|-|-|\n|ETA|$60.5\\pm1.8$|\n|**ETA+CoLA(Ours)**|$63.1\\pm0.7$|\n|SAR|$58.6\\pm0.1$|\n|**SAR+CoLA(Ours)**|$61.4\\pm0.3$|\n|DeYO|$55.8\\pm3.9$|\n|**DeYO+CoLA(Ours)**|$62.2\\pm0.3$|\n\n\n> Q6. What are the overheads of the shared domain vectors? Can you provide any numerical example of the growth of shared vectors? How does the method achieve minimal computational overhead (only a 3-second increase) compared to the baseline ETA, as reported in Table 5?\n\nA. Compared to ETA, **CoLA only introduces an extra matrix ADD operation** in computational overhead to aggregate domain vectors before forward propagation. In this process, no extra forward passes is needed even with the increased domain vectors. During backpropagation, the previously learned domain vectors $\\Delta\\theta_i$ are frozen and CoLA only incurs extra update cost for the 1-d $\\alpha$ to aggregate $\\Delta\\theta_i$ adaptively. **We do not alter the dimension of model weights or activations**, thus posing minimal computation and memory overhead in Table 5, i.e., additional 2.7% time cost and 0.08% memory footprint.\n\nPlease also refer to Reviewer SsoW’s Q3-2, which demonstrates that CoLA can efficiently scale to 10,000 domain vectors with only an additional 11 seconds of latency.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We deeply appreciate all reviewers for their valuable feedbacks and constructive comments on improving the quality of our paper. We would like to address your questions below.

>  G1. Differences and advantages of our methods from/over federated TTA [A,B]:

- **Differences on problem settings and learning paradigms:** Federated TTA [A,B] focuses on addressing data silos/privacy issues by exploiting collaborative learning during the federated source training phase **(supervised)**. However, it performs TTA on each device independently during the testing phase. In our paper, we seek to improve test-time model adaptation performance and efficiency by exploiting knowledge from multiple devices in the application environment, which essentially is a new **unsupervised** on-time TTA paradigm.
- **Differences on application flexibilities in deployment:** In federated TTA [A,B], the training phase and test-time adaptation phase are highly correlated, which means it can only use its own trained model for TTA. This makes federated TTA [A,B] restricted for real-world applications. In contrast, our CoLA paradigm can be applied to any pre-trained models, given that feedbacks from multiple devices or single device are available. Thus, it offers much better flexibility in deployment.
- **Differences on TTA paradigms in the testing phase**. In federated TTA [A,B], each device conducts model adaptation using normal TTA strategies independently, which means it inherits the limitations of the used TTA methods. In contrast, our method is able to exploit knowledge from multiple devices to improve both the performance and efficiency of TTA. 
- In fact, **our CoLA can be incorporated into federated TTA with slight efforts to boost its performance.** As shown in Table A, the performance of the federated TTA[A] can be much boosted with our CoLA learning paradigm.



Table A. Performance improvement of  Federated TTA [A] with our CoLA on CIFAR-10-C with ResNet-18. **The source model is trained by federated TTA [A].**
|Method|NoAdapt|Federated TTA [A]|Federated TTA [A]+CoLA(Ours)|
|:-:|:-:|:-:|:-:|
|Acc.|65.2|71.2|**74.3**|


[A] Adaptive test-time personalization for federated learning. NeurIPS 2023.

[B] Test-time robust personalization for federated learning. ICLR 2023.


> G2. Differences and advantages from/over Cloud-Edge TTA (CEMA) [C]:

- **Differences on problem settings**: CEMA [C] is dedicated to cloud-edge collaborative TTA between two devices, *i.e.*, one cloud server and one edge device. While our CoLA focuses on **collaborative TTA** among multiple devices with diverse computational resource budgets.
- **Differences on communication overhead**: CEMA [C] transmits raw samples and model parameters for each learning iteration/batch, leading to relatively high communication overhead. In contrast, our CoLA only transmits domain vectors (affine parameters of norm layers) when a domain shift is detected, resulting in **much lower communication costs**. Taking ViT-Base and ImageNet-C as an example, the communication cost for a single domain is 0.15MB (Ours) vs. 568.85MB (CEMA).
- **Differences on privacy protection**: CEMA [C] directly uploads raw samples from the edge devices to the cloud, and thus suffers privacy concerns. Instead of transmitting raw samples, our CoLA only transmits domain vectors/norm layer parameters, enhancing its availability for privacy-sensitive applications.
- Moreover, our CoLA achieves better adaptation performance than CEMA [C]. In Table C, our CoLA equipped with SAR attains a higher accuracy than CEMA (61.7% *vs.* 59.4%).

**We will include discussions with methods [A,B,C] in the revised paper.**

Table C. Comparison between CEMA and SAR+CoLA(Ours) on ImageNet-C (level 5). The experiments follow the settings of Table 2 in the main paper. Here, we report only the average results across all devices and corruptions. 
| Method             | Avg. Acc. |
| ------------------ | :-------: |
| NoAdapt            |   28.6    |
| Cloud-Edge TTA (CEMA) [C]            |   59.4    |
| SAR                |   58.6    |
| **SAR+CoLA(Ours)** | **61.7**  |


[C] Towards robust and efficient cloud-edge model adaptation via selective entropy distillation. ICLR 2024.

Meta-review: This paper considers test-time adaptation in multi-device scenarios and proposes Collaborative Lifelong Adaptation (CoLA). Domain knowledge vectors, as interactive objects on devices, are maintained and stored by measuring KL-divergence between the current test batch and the online estimated distribution. This paper categorizes different devices into principal agents and follower agents based on computational resources and latency demands, and uses different strategies for online prediction or parameter updates during testing. 

The reviewers were initially mixed with both positive and negative scores. They agreed that the approach is novel and interesting, the paper is well written, the method is of practical importance. The positive reviewers noted that CoLA is simple and efficient, demonstrating good performance in collaborative, lifelong, and single-domain TTA tasks. It also considers the application for devices with different computational resources and latency demands in practice, and  is technically sound. The negative reviewers mostly raised concerns about lack of citations and comparisons to previous work, limited experimental evidence, and various other questions of detail. The authors addresses these concerns in their rebuttal and extensive discussion with the reviewers. In the end, all reviewers were positive towards the paper. The ACs concur that the paper merits publications
Decision: Accept (poster)