SANTA: Source Anchoring Network and Target Alignment for Continual Test Time Adaptation
Authors: Goirik Chakrabarty
Manogna Sreenivas
Soma Biswas
Venue: Accepted by TMLR ()

Abstract: Adapting a trained model to perform satisfactorily on continually changing test environments is an important and challenging task. In this work, we propose a novel framework, SANTA, which aims to satisfy the following characteristics required for online adaptation: 1) can work effectively for different (even small) batch sizes; 2) should continue to work well on the source domain; 3) should have minimal tunable hyperparameters and storage requirements. Given a pre-trained network trained on source domain data, the proposed framework modifies the affine parameters of the batch normalization layers using source anchoring based self-distillation. This ensures that the model incorporates knowledge from the newly encountered domains, without catastrophically forgetting the previously seen domains. We also propose a source-prototype driven contrastive alignment to ensure natural grouping of the target samples, while maintaining the already learnt semantic information. Extensive evaluation on three benchmark datasets under challenging settings justify the effectiveness of SANTA for real-world applications. Code here: https://github.com/goirik-chakrabarty/SANTA

Published in Transactions on Machine Learning Research (11/2023)
SANTA: Source Anchoring Network and Target Alignment for Continual Test Time Adaptation
Goirik Chakrabarty
Indian Institute of Science, Education and Research, Pune
Manogna Sreenivas
munognas@iisc.arr
Indian Institute of Science, Bangalore
Soma Biswas
somabismas wiias...iiisccc...oo
Indian Institute of Science, Bangalore
Reviewed on OpenReview: htppss///ppenevvviee net/ por/formm? id= V7guVYzvE4
Abstract
Adapting trained model to perform satisfactorily continually changing lest environ- ments is an important and challenging task. In this work, we propose a novel framework, SANTA, which aims to satisfy the following characteristies required for online adaptation: 1) can work effectively for different (even small) batch sizes; 2) should continue to work well on the source domain; 3) should have minimal tunable hyperparameters and storage requirements. Given a pre-trained network trained on source domain data, the proposed framework modifies the affine parameters of the batch normalization layers using source anchoring based self-distillation. This ensures that the model incorporates knowledge from the newly encountered domains, without catastrophically forgetting the previously seen do- mains. We also propose a source-prooolype driven contrastive alignment to ensure natural grouping of the target samples, while maintaining the already learnt semantic information. Extensive evaluation on three benchmark datasets under challenging settings justify the effectiveness of SANTA for real-world applications. Code here: https:// https:///gthubbcomm cotttiuu..oomm/ooo /goirii Chakrabarty/SAANT
1
Introduction
Deep learning has achieved phenomenal success several computer vision lasks like classification, object detection, segmentation, etc (Deng et al. 2009 Ren et al. 2015 He al. [2017] Chen et al. 2018 Everingham et al. 2010). But it is well known that these models tend perform poorly when the test data comes from different distribution compared to the training data Ovadia et al. (2019). Unsupervised Domain Adaptation (UDA) techniques (Ajakan et al. 2014 Saito al. (018)) have shown promising results in this scenario, but they require access to unlabeled target data along with the labelled source data, which may be difficult due to privacy concerns, storage constraints, etc. In addition to having a different distribution compared to the training data, the test data distribution may dynamically vary with time. For example, autonomous driving, the model trained using data captured in clear weather can encounter cloudy weather followed by heavy rain, etc. during deployment. Thus, continually adapting the model during test-time is critical for the model to perform well in changing scenarios.
Test-time adaptation of trained models has thus emerged as an important rescarch area, where an off-the- shelf trained model is adapted to the lesting data as and when they are encountered. Majority of successful frameworks for this task, (Wang et al. 2021 Boudiaf et al. 2022), assume that the test data belongs to a single domain, which is a restrictive assumption for practical applications. Very recently, rescarchers have started to look the continual test time adaptation setting (Wang et al. 2022a, Döbler et al. 2023 Brahma & Rai 2023 where the target distribution can change over time.

---

Published in Transactions on Machine Learning Research (11/2023)
In this work, we propose novel framework, termed Source Anchoring Network and Target Alignment (SANTA); for the task of continual test-time domain adaptation. We believe that, for the model to be prac- tically useful in an online setting, it should satisfy the following requirements: 1) For online adaptation, the models should work effectively with different (preferably small) batch sizes, while also being computationally efficien;; 2) The updaled model should continue lo work well on the source domain; 3) The frumework should require less storage and minimal tunable hupprpaarameters, since a validation set usually not available dur- ing test-time. With this motivation, we propose use source anchoring based self-distillation, which ensures that the model robustly adapts to the incoming data, while not forgetting the source domain information. The proposed SANTA framework also utilizes contrastive learning to ensure better model generalizability to unseen domains. Further, we utilize the source prototypes for aligmment of the target features to the corre- sponding source data, which helps preserve the semantic information learnt using the source. We propose to only update the Natch-Normalization (BN) affine parameters like TENT Wang et al. (2021). as it avoids overfitting on the small amount of target data and also reduces the storage requirements. This simple, yet effective framework helps us take a step forward in achieving all the objectives mentioned earlier. Extensive experiments three large-scale benchmark datasets, namely CIFAR- 10C, CIFAR- 100C and ImageNet-C Hendrycks & Dietterich (2019) for different challenging and realistic scenarios justify the effectiveness of the proposed SANTA framework. To summarize, the contributions of this work are as follows:
We propose novel SANTA framework for the task of continual test-time domain adaptation.
The proposed framework takes step forward in overcoming some of the important challenges in a practical test-time adaptation setting.
We show that the proposed source-based anchoring along with contrastive target alignment, guided by source prototypes can be effectively utilized to robustly update the model under dynamically changing lest conditions.
Extensive evaluation several benchmark datasets for challenging scenarios justify its effectiveness.
We now discuss the related work, followed by the proposed method and evaluation.
2
Related works
The proposed approach is inspired by several seminal works in different arcas. Here, provide pointers to some of the related work in literature.
Source Free Domain Adaptation: Source Free Domain Adaptation (SFDA) methods aim to adapt a model trained using source data to a new domain using only the unlabeled target data (Xia et al. 2021 Huang et al., 2021 Yang et al. 2021a Liang et al. 2020 Kundu et al.] 2020 Li al. 2020 Tian et al. 2021| Yang et al. 2021b), Some of these methods, such as SHOT (Liang et al.) 2020), use a escudo-labelingg strategy to maximize information and minimize entropy, while others, such as Li et al. 2202 use generative models to enhance model performance on the target domain by generating target-style images. GSFDA Yang et al. (2021b) aims to activate different channels within a network for different domains while also taking into account the local data structure. A Xia et al. (2021) is another hot that utilizes different classifiers to align the two domains using adversarial training. BUFR Eastwood et al.
(2021) propose Bottom-Up Feature Restoration scheme where they store a lightweight source feature distribution and then adapt the feature-extractor such that the target feature distribution realigns with that saved on the source.
(Continual) Test time adaptation: Test Time Adaptation (TTA) is an online variant of SFDA that adapts the model at test. time using small batches of test data, as and when they become available. Here, the model's parameters or architecture are usually adjusted to handle the differences between the
two domains better, thereby improving its performance on the target domain (Wang et al. 2021 Shen et al. 2022 Boudiaf et al. 2022 He et al. [2021] Schneider et al., (020). Some of these methods focus on modifying the original architecture during the source training like TTT Sun et. al. (2020). TTT++ Liu
2

---

Published in Transactions on Machine Learning Research (11/2023)
et al. 2021 which trains the model on supervised and self-supervised tasks using source data. During lesting, the self-supervised module is fine-tuned on the target data to improve performance. MT3 Bartler et al. (2022) employs meta learning and self supervision to effective adapt a model during test time. After that, mothods such as EATA Niu et al. (2022) use some fraction of source data for identifying the effect of domain shift and update the model accordingly during lest time. Recently, several researchers are focusing on the fully test time adaptation sctting (Wang et al., 2021] Schneider et al.] 2020 Zhang et al.) 2022; Mummaai et a 2021), which does not assume any access source data or the source training process making it more practical. TENT Wang et al. (2021) adopts entropy minimization objective for training the BN layers, while BN Stats Adapt Schneider et al. 2020) adjusts the BN statistics during test time to align the target with the source domain.
A more realistic scenario is handled by the recently proposed continual test-time adaptation protocol Wang ct al. (2022a), where the trained model should continually adapt a dynamic environment, i.c. the test domain can change over time. CoTTA Wang et al.) (2022a) utilizes woight-averagod and augmentation- averaged predictions to reduce error accumulation and also stochastically restores small part of the neurons to the source pre-trained weights during each iteration to avoid catastrophic forgetting. This allows for long- term adaptation of all paramcters in the network while preserving source knowledge. Recent methods (Döbler et al. 2023 Brahma & Rai 2023) which have achieved better performance than CoTTA also use a teacher student framework to update the adapting model. Specifically, RMT Döbler et al. 20023 uses symmetric cross-entropy loss in a teacher-student framework. PETAL Brahma & Rai 2023 use a data-driven parameter restoration technique as opposed stochastic restoration in CoTTA
Recently, several methods (Gan et al., 2022 Gong et al. 2022 Niu et al. 2022 have been developed which can aid the dynamic adaptation to test-data. However, these methods need to be optimised along with the model during training with source. Hence, we do not compare with these methods as, in our protocol, we do not assume access to source data.
Knowledge and Self-distillation: Knowledge distillation is a technique used to transfer knowl- edge from a large, complex model ('teacher' model) to a smaller, simpler model ("student" model) (Hinton et al. 2015 Gou et al. 2021). This is done by training the student model to mimic the predictions of the teacher model, which has already learned useful representations, rather than training the student model on the original labeled data. CoTTA also utilizes distillation method to enhance the adaptation to new domains, which involves the implementation of a teacher model to make accurate predictions based on the student model.
A variant termed as self-distillation or self-knowledge distillation Yuan et al. (2019) or self learning Rusak let al. (2022), involves training model mimic its own predictions. The framework in Yang et al. (2019) shows that using model from previous epoch to train the same model in future epochs can increase the training efficiency and accuracy of the model.
2020 Caron et al. 2020 has shown tremendous improvement in learning visual features for various downstream tasks, Because of its robustness, researchers have used in SFDA (Huang et al.
Contrastive learning: Contrastive learning CCen et al.
2020
Khosla et al.
2021 Xia et al. 2021 Wang et al. 2022b Yang al. 2022: Zhao et al.] 2022], TTA Chen al. (2022). Domain Generalisation (DG) (Kim et al. 2021 Yao al., 2022) and other studies Vaze et al., 2022: Fei al.] 2022) to train/adapt, pre-trained model to a target domain using only unlabeled data. In previous works, contrastive learning has been used to get better feature representation. However, in this work, we align target features guided by source prototypes to get meaningful target features with respect to the source feature space. This results in meaningful clustering and good separation of classes in unseen domains.
3 Problem Setting and Motivation
In continual test-time domain adaptation, we are given a model trained using source domain data D, [(., 9,) P.. Here, (ax, yi) source data and label which is drawn from the source distribution P.. During testing, the source data is usually not available due to privacy concerns storage constraints. At
3

---

Published in Transactions on Machine Learning Research (11/2023)
2
Source Guided Target Alignment
Fropen
LTA
Sourte praceypss
L
P
Images
Adapting
Clasalfer
LSA
dspreelution
a
Source Model
Figure Illustration of the proposed SANTA framework. The original image and its augmentation are passed through the adapting model (only BN affine parameters updated) and the source model. 1 The prediction of the input given by the source model is used as an anchor (a) for the prediction given by the adapting model (p for input image and q for augmented image) for computing the source-anchoring loss CSΛ- The source guided target alignment LtΛ uses the source prototypes to help maintain the semantic information learnt using the source domain and enforces clustering that is meaningful in the feature space.
this stage, the model encounters test data D. = [m P.. In practice the test data can belong to different domain compared to the source, i.c. P. P P.. Further, Pa can change over time such that p(") # pp{2) + Ps leading to continual test time adaptation scenario.
In this work, the goal is to develop practically useful framework for the above continual fully test time scenario, which should have the following characteristies: 1) For online adaptation, to reduce the latency, the model should be able to work seamlessly for different (preferably small) batch sizes, like 25 or 10 as used in (Zhao et al. 2023 Lim et al. 2023). But the current approaches Wang et al. (2022a) have been tested only for much larger batch sizes (like 200); 2) The updated model should continue to work well on the source domain; For example, while adapting to cloudy should not fail for images taken in clear weather (source domain); 3) In test-time adaptation scenario, since validation sets are usually not available for tuning pareee,,rameeers, the approach should require minimal parameearmmeeers: and as practical bonus, storage should be minimal and the framework should have low inference time for ease of deployment. Now, we describe the proposed SANTA framework (Figure 1 and claborate on how some these challenges are addressed.
rainy weather, the framework deployed for autonomous driving
4 Proposed SANTA Framework
Given the model trained using the source data, the goal is to adapt it using the limited amount of test-data encountered in each batch in a dynamic environment, while satisfying the desirable criteria mentioned above. Towards this goal, we propose (i) Source Anchorization Network and (ii) Target Alignment, which we now describe detail.
4.1 Source Anchorization Network
During test-time, the model has access to few test samples in a batch, which may not representative of the corresponding target distribution. Thus, modifying the model parameters completely on the basis of the available target data may result in simultancously overfitting on the few target samples and also catastrophic forgetting of the source information. CoTTA Wang et. al. (022a)) addressed this challenge by
4

---

Published in Transactions on Machine Learning Research (11/2023)
(i) learning teacher model by combining the source and contimuously adapting student model, wherein the teacher changes gradually for robust prediction and also using (ii) stochastic restoration to reset some of the model parameters to the source model after every batch. Though this gives impressive performance, it has two limitations, namely (i) the complete teacher, student and source model needs to be stored and (ii) the hyperparanelers required for computing the teacher model and for stochastic restoration need to be determined, which can have different optimal values for different datasets. To overcome these challenges, in this work, we use self-distillation using the source model as the anchor (Hinton et al. 2015 Yang et al. 2019).
We denote the adapting model as fa. as this is the model which is contimuously updated and is used to predict the larget data. The weights 0 of the adapting model is initialised to the weights 0g of the off-the-shelf source model fe.- Now, consider time instant k, when the model encounters a new batch denoted by Bg. Since adaptating the BN statistics has proven to be effective in capturing the data distribution characteristics (Schneider et al.) 20200] Wang et al., 2022a). the BN statistics of the source model (fo.) and the adapting model (Jo) are changed to the BN stat of the larget. batch at each step. Let. these models be denoted fă. and f respectively. f/t (referred to as target corrected source (TCS) from now) can be thought of as a specialized model that accounts for the domain difference between the source and the specific target batch, On the other hand, the weights O of the adapting model fe are optimized after every batch using the loss function that will be described later. It should also be noted that during optimization, we only update the BN parameters of the adapting model [Wang et al. b021b.
The proposed self-distillation loss is inspired from the knowledge distillation loss formulation used in in- cremental learning (Dong et al. 2021 Kang et al. 2022 to prevent catastrophic forgetting. In this work, self-distillation between the response of the TCS model f² and the adapting model f acts a regular- izer Mobahi et al. (0200), that encourages the adapting model to mimic the TCS model, which is specialized model whose response corresponds to domain invariant features, Thus, our adapting model is reinforced to Iearn domain invariant fcatures Mobahi et al. 2020)) leading to better genoralization, which we empirically show in Table
The loss function is based on the prediction scores of the adapting model and the TCS model for a given batch of test images, Bg : {#1, T2, ......... ENN) }. N being the batch size. Let Pij and aiy denote the jth element of I (a:) (adapting model) and fH, (zx) (TCS model) respectively, which gives the prediction score of the jth class for the tt test image. The source-anchoring loss for given batch is calculated follows:
LSA
Σ Σ Piy log(ais)
(1)
Here, C' is the number of classes, and N is the batch size. Empirically, we observe that using augmentations of the test images make the model more robust. Let qı denote the prediction score of the jth class for the tth augmented test imago, given by the adapting model. The complete source-anchoring loss is given by
N
C
1 N
LSA
(pig log(ass) + 4iz log(a:)))
(2)
This simple, yet effective self- eef-ddstillattin offers multiple advantages as follows: i) Since the target corrected source model used for anchoring, there is no hyper-parameter involved (such as the weight for combining student with source to form the teacher model as in CoTTA Wang et al. (2022a)) ii) The adapting model can be directly used for prediction continuously, without requiring any restoration to the source model; iii) Since only the BN parameters updated, just these parameters of the source need to be stored, resulting in much lesser storage requirements compared to storing two different models.
4.2 Target Alignment
The goal to learn a generalized model as it encounters data from different domains, thereby making the features gradually domain invariant. Here, we additionally use self-supervision (in the form of contrastive
5

---

Published in Transactions on Machine Learning Research (11/2023)
learning) for improved generalization as used in ZZao et al. 2022 adapting model f can be decomposed into a feature extractor, g and the fixed classifier h, i.e.
Wang et al 2022b). Formally, the
S = go
(3)
Suppose, the augmented samples for the test batch { Ni are denoted by { We pass these samples through the feature extractor & and further a projection head Pe and normalized so that the features are mapped to a d-dimensional hyper-sphere (Chen et al. 2020 Khosla et al. 2020
= Pv gz();)
(4)
The following is the contrastive loss
N
exp(2.2N+5/T)
Coo
log
(5)
N
22 kml [iggt exp exp(2i.zk/T)
where > 0 is the temperature hyperparameter. In this work, in Wang al. (0021 the learnable model parameters & comprise of only the BN affine parameters. The classifier is kept fixed throughout the adaptation process to avoid overfitting on the target samples and preserve the discriminative information learnt using the abundant source data. Thus, for correct classification, the target clusters should also align with the original source representations. To achieve this, we use the source prototype guided target alignment loss. To this end, we include a third view which assigns the nearest source prototype as an augmented view for the test time features. Specifically, let the source prototypes for the C classes be denoted as {Ti)C= We compute the nearest source prototype for sample I;
x (x:) ( arg max(cos 95 (x:)))
(6)
For each feature 24, we use two positives for contrastive learning. The first view zN+i is the projection of its augmented sample 2N+i, the second view #2N44 is the projection of its nearest class prototype (I;). The source prototype guided Target Alignment loss is given by:
N
exp(zi.2+1i/T) E [Rg exp(2i.za/7
exp(2x-22N45/T)
LTA
log
+ log
2N
E [K exp(2i-za/T)
By using this additional view, we implicitly pass the class information to result in improved clustering while also keeping intact the semantic information learnt from the source data. The qualitative effect of clustering using prototypes a view be in Figure 2 We further quantitatively ablate its effect in Table
4.3
Final loss
Given any off-the-shelf pre-trained model and the source prototypes, during testing, using the current test batch, we modify the BN parameters & and the projection head is such that the following loss minimized:
CSANTA LSA + CTA
The adapting model is used for predicting the class of all the test samples. is robust enough to be updated continuously without any restoration back to the source model.
5 Experimental Evaluation
Here we describe the experiments performed to evaluate the effectiveness of the proposod framework.
Dataset Details:
Here,
evaluate the proposed framework extensively multiple benchmark

---

Published in Transactions on Machine Learning Research (11/2023)
(a) With Source prototypes
(b) Without source prototypes
Figure 2: t-SNE plot of the feature space using only LTA for adaptation. We observe that the aligmment and clustering is better with source prototypes (a) as compared to not using them (b). The source prototypes are represented using stars.
bright mess
gausatio
deforas
10stion
ehastic
Pianlate
aboot
50OW
Grost
Speg
Dataset
Method
Mean
Source
72.3
65.7
72.9
46.9
54.3
34.8
42.0
25.1
41.3
26.0
9.3
46.7
26.6
58.5
30.3
43.5
CIFAR-10C R-C
BN Stats Adapt
28.1
26.1
36.3
12.8
35.3
14.2
12.1
17.3
17.4
15.3
8.4
12.6
23.8
19.7
27.3
20.4
TENT-contional
21.8
20.6
28.6
31.1
16.3
14.1
19.1
18.6
18.6
12.2
20.3
25.7
20.8
21.9
20.7
CoTTA
213
12.4
18.3
16.2
RMT
24.5
20.0
25.5
13.9
24.6
14.9
13.3
16.0
15.8
15.6
11.1
15.0
18.3
14.6
16.9
17.3
PETAL
23.4
21.1
25.7
11.7
27.2
12.2
10.3
14.8
13.9
12.6
7.4
10.5
18.1
13.4
16.8
15.95
SANTA
23.9
20.1
28.0
11.6
27.4
12.6
10.2
14.1
13.2
12.2
10.3
19.1
13.3
18.5
16.1 + 0.06
Source
73.0
68.0
39.4
54.1
30.8
25.8
39.5
45.8
50.3
55.1
37.2
74.7
46.4
CIFAR-108 10C
BN Stats Adapt
42.1
10.7
12.7
27.6
29.7
27.9
34.9
35.0
41.55
30.3
35.7
32.9
35.4
TENT-continal
37.2
35.8
41.7
37.9
51.2
48.3
18.5
58.
63.7
71.1
70.4
82.3
88.0
88.5
90..1
60.9
COTTA
40.1
37.7
39.7
26.9
38.0
27.9
26.4
32.8
31.8
40.3
24.7
26.9
32.5
28.3
33.5
32.5
RMT
40.5
36.1
36.3
27.7
33.9
28.5
26.4
29.0
29.0
32.5
25.1
27.4
28.2
26.3
20.3
30.4
PETAL
38.3
36.4
38.6
25.44
32.0
30.8
38.7
26.44
31.5
26.9
31.46
SANTA
36.5
33.1
35.1
27.7
25.4
29.5
29.9
33.1
26.77
31.9
27.5
35.2
30.3
IniggeNet C(50k) EmageNNe C(5kk)
Sourve
97.8
97.1
98.2
81.7
80.8
185.2
78.0
$3.5
77.1
75.9
41.3
134.5
82.5
79.3
68.6
82
BN Stats Adapt
85.0
63.7
85.0
84.7
84.3
73.7
61.2
66.0
68.2
52.1
34.9
82.7
55.9
51.3
5818
68.6
TENT-continnal
81.6
74.6
72.7
77.6
73.8
65.5
55.3
61.6
63.0
51.7
38.2
72.1
50.8
47.4
53..3
62.6
CoTTA
81.7
82.1
80.6
79.0
08.6
57.5
60.3
60.5
48.3
66.1
47.2
41.2
46.0
62.7
RMT
80.2
76.1
57.6
57.0
59.1
18.0
60.6
17..3
60.2
SANTA
74.1
72.9
71.6
75.7
74.1
64.2
55.5
35.6
62.9
46.6
36.1
693.9
30.6
44.3
48.5
60.1 0.06
Source
97.8
97.1
98.1
82.1
90.2
85.2
77.5
83.1
76.7
75.6
41.1
94.6
83.0
79.4
68.4
82.0
BN Stats Adapt
81.9
81.0
81.2
85.0
81.7
73.6
61.2
65.6
66.9
52.0
31.7
$3.2
55.8
51.0
60.2
68.5
TENT-contimal
71.5
66.1
69.3
94.9
97.0
98.8
99.3
99.3
99.6
99.4
99.4
91.0
CoTTA
78.4
68.4
64.4
71.8
67.4
71.1
67.0
73.5
69.4
67.1
69.7
RMT
73.6
65.9
61.3
713
72.0
71.0
69.9
70.2
71.9
70.3
66.2
74.7
68.5
67.3
67.9
69.9
SANTA
73.6
75.1
73.2
76.2
76.8
61.1
53.5
55.8
61.7
43.7
34.5
72.7
49.2
43.9
50.2
80.3 0.07
Table 1: Error percentages (lower better) of Source model, BN Stats Adapt Schneider al. (2020), TENT- Continual Wang et al. (2021), CoTTA Waag al.) (2022a), RMT Döbler et al. (2023), PETAL |Brhmma
Rai (2023) for CIF,R-10C, CIFAR-100C and ImageNet-C (5k and 50k samples) for batchsizes of 200, 200 and 64 respectively. For SANTA, the standard deviation is reported over 5 random seeds.
datasets, namely, CIFAR-10C, 10C, CIFAR-100C and ImageNet-C Hendrycks & Dietterich (2019). These datasets have 10, 100 and 1000 classes respectively. All the datasets contain 15 diverse corruptions of the

---

Published in Transactions on Machine Learning Research (11/2023)
Baax Aday
Frice
Dourss
Bisrz
Bax Auay
BraNe
B
BANCA
100
80
60
60
60
40
40
40
20
20
20
10
50
10
150
32
64
(a) CIFAR-10C
(b) CIFAR-100C
(c) ImageNet-C
Figure 3: These figures compare the robustness of the different approaches for different batch sizes. The x-axis is the batch size and y-axis is the error percentage (lower is better). We observe that the proposed SANTA is robust across batch sizes.
Methods
CIFAR- 10C
CIFAR-10CC
IMAGENET-C
200
150
100
50
26
10
200
150
100
50
26
10
64
322
16
(original image)
21.1
20.7
21.7
24.8
33.7
62.7
77.5
SALoss (original sugmented image)
17.1
17.2
17.4
18.0
19.0
22.5
31.1
31.6
32.0
32.8
94.7
39.9
61.3
63.8
88.5
76.8
SANTA (SALoes + TALoss)
16.1
16.2
16.5
17.2
18.3
22.0
30.4
30.5
30.8
31.9
33.9
39.1
60.2
63.2
68.5
78.5
Table 2: Mean error percentage (lower is better) demonstrating the importance of the two proposed compo- nents. Ablation is done all the four datasets for multiple batch sizes.
forms noise, blur, weather, and digital. Each corruption has five levels of severity, applied to the test set of all the three datasets. Following CoTTA Wang et al. (2022a). we do ImageNet-C experiments with 5k samples per corruption. Further, we do a more robust experiment using the complete ImageNet-C dataset consisting of 50k samples for each corruptior]] For all the experiments, unless mentioned otherwise, the test sequence consists of all 15 corruptions at the highest level of severity Wang et al. 2022a)). The goal is adapt an off-the-shelf source model to this dynamically changing environment efficiently during test time.
Implementation Details: For CIFAR-10C, we a pre-trained WideResNet-255 Zagoruyko & Komodakis (2016) model from the Bench benchmark Croce et al. (2021) as in Wang et al. (2022a). The model is updated with one gradient step per batch, and the Adam optimizer with a learning rate of 1c-3 is used. The temperature is set to the default value 0.1. The CIFAR-100C, experiment uses a pre-trained ResNeXt-299 Xie et al. (2017) model, which is one the default models for CIFAR-100 in the RobustBench benchmark
Croce et al. 22021). The same hyperparameters as the CIFAR-10 experiment are used. For the ImageNet-CC
experiment, the standard pre-trained Resnet50 He et al. (2016) model from RobustBench Croce al. (2021)
is used. Here, SGD is used as the optimiser with learning rate of 1c-2 as in Wang al. 2022a
We
conduct all the experiments an NVIDIA GeForce RTX 3090.
6 Research Questions
We perform experiments to address the following research questions:
1) How does SANTA perform on the standard experimental protocol (higher batch size) as used in Wang et al. (2022a)? Table reeports the results on the three benckmark datasets. The batch sizes used for these experiments are 200, 200 and 64 for CIFAR-1OC, CIFAR-100C and ImageNet-C respectively as Wang et al. (2022a). the TENT-contimaal setup, the model continuously adapts and is not reset to the source model after each corruption.
' Previoos works have used only samples per corruption for ImageNet-C due a bug in Robusthench setup.
8

---

Published in Transactions on Machine Learning Research (11/2023)
Avg. Error (%) Source
BN Stats Adapt
TENT-Contimal
CoTTA
RMT
SANTA
ImageNet-C
82.0
68.5
84.1 + 6.38
64.61 + 2.72
63.6 + 4.15
60.18 + 0.13
Table 3: Average error for MetgeNet-C experiments over 10 diverse corruption sequences (severity level 5).
CIFAR-10C
CIFAR-100C
ImageNet-CC
SANTA w/ prototypes
16.1
30.0
60.2
SANTA w/o prototypes 16.2 (-0.1)
31.4 (-1.4)
61.1 (-0.9)
Table 4: Here, we see the performance comparison (error % lower is better) between using protolypes as a view v/s not using them in the target alignment loss. (.) is the relative drop in performance.
We observe, from Table I that for all the datasets, the proposed method SANTA consistently performs at par or surpasses the existing approaches. For ImageNet-C, using 5k samples per corruption, we obtain an error of 60.1%, which is 2.6% better than the seminal work COTTA Wang et al. (2022a) and par with RMT Döbler et al. (2023). Moreover, in the challenging case of ImageNet-C,, using all 50k samples, SANTA significantly outperforms existing baselines. No hyperparameter tuning is done for any methods extending the protocol to 50k samples. This is in congruence with the fact that, in real world CTTA, tuning hyperparameters is out of scope.
2) How does SANTA perform on lower batch sizes, which is more realistic scenario for online test time adaptation?
A practical lest-time adaptation algorithm should work satisfactorily for lower batch sizes, which will decrease the average time for inference of a sample and thus lower the latency of the framework. Although researchers have recently started to address the issue of robustness across batch sizes (Lim et al. 2023 Gong et al. 2022 these are not fully test time adaptation and requires source data for warmmu before online adaptation.
Here, we evaluate the robustness of the proposed SANTA framework for lower batch sizes and compare the results with the ttate-of-the-art. Ssows the results for the three datasets with decreasing batch sizes. Since the results of the other approaches were not reported for other batch sizes, we used the official codes and obtained the results reported in the table. To ensure the best results for the other methods, we tuned the appropriate hyperparametors. Specifically, for TENT and RMT, varied the learning rate. For CoTTA, the restoration probability and model EMA factor was reduced in proportion to the decrease in batch size. Note that for the proposed SANTA, no parameters were changed for dataset across batch sizes. We observe that the improvement provided by our approach becomes clcarer as the batch size decreases. For example, for a batch size of 10, we obtain 22.0% for CIFAR-1OC dataset, which is significantly better compared to the next best of 25.0% obtained by BN Stats Adapt as shown in Figure 3 This experiment justifies the effectiveness of the proposed SANTA for online settings. Also, prior methods RMT and CoTTA perform significantly worse compared to SANTA as the batch size decreases, suggesting that it can be unreliable to use them in real-world settings where more often than not there is both less data and therefore less batch size from a particular domain.
3) Are both the proposed modules important? Here, we analyze the importance of the two losses the proposed SANTA framework. We observe from Table Abllethhat that significant portion of the performance improvement of SANTA can be attributed to the source- anchoring of the (est samples and its augmented version. The clustering and alignment terms further help to improve the performance, thereby achieving state-of-the-art performance for continual test-time domain adaptation under different challenging settings.
4) How does SANTA perform for different corruption orders?
From Table 3, we observe that SANTA demonstrates competitive performance on par with RMT and notably outperforms all previously employed methods. Additionally, it observed that SANTA exhibits low variance across corruptions, compared other tochniques, which a highly sought-after characteristic for ensuring the reliability of CTTA.
9

---

Published in Transactions on Machine Learning Research (11/2023)
Avg. Error (%) Source
BN Stats Adapt
TENT-continual
CoTTA
RMT
SANTA
CIFAR-10C
24.8
13.7
20.4
10.9
10.4
10.7
CIFAR-100C
33.6
29.9
74.8
26.3
26.6
25.6
Table 5: Average error over all corruptions with severity presented in the gradual test time adaptation manner. The order of corruption used here is the same as in Table
batchsize
200
150
100
50
25
10
TENT-Continual
92.0 (-70.9)
96.2 (-75.1)
97.8
(-76.7)
98.2 (-77.1)
98.5 (-77.4)
99.0 (-77.9)
CoTTA
22.8 (-1.7)
23.0 (-1.9)
23.9
(-2.8)
27.2 (-6.1)
35.5 (-14.4)
58.5 (-37.4)
RMT
24.7 (-3.6)
26.3 (-5.2)
29.0
(-8.9)
35.9 (-14.8)
48.0 (-26.9)
76.1 (-55.0)
SANTA
22.4 (-1.3)
22.7 (-1.6)
22.9 (-1.8)
23.6 (-2.5)
25.8 (-4.7)
29.9 (-8.8)
Table 6: Error on CIFAR-100 test set after the model has been adapted to all the corruptions in Table (.) is the degradation in performance on source data compared to the source model, whose error is 21.1%.
This deviation quantifies the amount of catastrophic forgetting while adaptation.
5) What is the effect of source prototypes in Target Alignment? We perform experiments with and without source prototypes to study its effect and report the results in Table | We observe that without the prototypes, the performance drops slightly compared to when using it. Although source prototypes aid clustering as scen in Figurd] our method does not primarily rely on it, suggesting that SANTA can still be reliably used in cases where source prototypes are not accessible.
6) How does SANTA perform when the domain shifts are gradual?
In the standard setup, the corruption types change abruptly with maximum severity levels. A more realistic approach will be to evaluate the performance of the approaches when the severity levels change gradually over sequence of 15 corruption types. Thus we experiment with the gradual setup as also done in Wang et al. 2022a).. The representation below shows the order in which severity is faced for every corruption.
2
1-2
t- and before
corruption type, with changing severity
++1 and after
The results suggest that BN Adapt, CoTTA, RMT and SANTA are more effective than the source and TENT-continul approaches in dealing with corruption types that change gradually over time. For CIFAR- 10C, we observe COTTA, RMT and SANTA perform similarly. For CIFAR-100C; our approach achieves the lowest average error rate of 25.6%, followed by CoTTA with an average error rate of 26.3%. BN Stats Adapt approach also performs well with an average error rate of 29.9%.
7) Does SANTA prevent. catastrophic forgetting by retaining good performance on revisiting the source data?
Although the source trained model is gradually adapted to the changing testing conditions, we also want it to be able to perform well on the original source distribution, which requires the model to not catastrophically forget the original source training information. For example, the original model trained on clear weather conditions should continue to do well for clear weather images, even though has adapted other conditions like rainy, foggy, etc. To achieve these contrasting goals, it is important that the model has the right balance of stability-plasticiyy knowledge and adapting to new information. Table f reports the results of using the model adapted on CIFAR-100C, on a held-out testing set from the source distribution, which is the clean CIFAR-100 dataset in this case. The performance of the original model trained on source gives an error 21.1% on its test set. Thus the difference gives an estimate of the catastrophic forgetting (Table B). We observe that even after adaptation, SANTA is able maintain its performance on the source distribution very well as compared CoTTA, RMT and TENT for all batch sizes.
The plassictyyy trade-off refers to the balance between preserving learned
10

---

Published in Transactions on Machine Learning Research (11/2023)
brightness
contrast
pixelate
clastic
Snow
frost
ipeg
fog
Method
Mean
BN Stats Adapt
35.6
34.9
42.1
26.9
31.0
36.0
33.5
41.7
35.2
CoTTA
31.3
30.5
36.7
25.2
27.8
31.5
29.1
36.4
31.1
RMT
32.1
32.1
38.6
28.1
33.0
30.8
28.7
34.3
32.2
SANTA
31.2
31.2
36.3
25.0
28.7
32.2
28.0
37.2
31.2
Table 7: This experiment on CIFAR-100C demonstrates the generalizability of the learnt model. We observe that both CoTTA and SANTA yield more generalized model after adaptation.
Method
# Parameters
# Trainable
% Trainable
BN-Stats
6,900,132
0
0
TENT
6,900,132
25,216
0.365
CoTTA
20,700,396
6,900,132
33.333
RMT
13,947,976
6,900,132
50.000
SANTA
7,047,972
172,928
2.506
Table 8: Number of (trainable) parameters as a proxy for the storage requirement of the respective algorithms. This table is for CIFAR-100C with ResNeXt-29 as the backbone.
8) How well does the model generalize?
When the source model encounters data from different domains, want it to gradually become more generalized, such that the features become domain invariant. To evaluate whether this happens in practice, we perform an experiment, in which the model is first adapted on the first 7 corruptions (gaussian to zoom). We then freeze the weights and evaluate its performance on the last 8 corruptions (snow to jpee). We compare the performance of this adapted model (using the methods CoTTA and SANTA) to the performance using the optimization free BN Stats Adapt model, where BN statisties are adapted to the corresponding batch statistics. We see from Table that both CoTTA and the proposed framework have indeed learnt generalized features and is therefore performing better than the BN Stats Adapt model. SANTA is however, much lighter framework when compared to CoTTA, which we discuss in detail below.
9) How does the model fare in terms of storage cost, inference time?
The proposed SANTA framework has computational advantages over the existing approaches:
(i) Number of parameters be stored: Table 8 reports the total number of parameters and trainable parameters for TENT, CoTTA, RMT and SANTA. If the source backbone has N parameters, CoTTA stores 3N parameters (source, teacher and student). RMT stores 2N parameters(teachet and student). We only store 1.02N parameters (source, BN parameters of the adapting model which is only 2% of N). Thus the proposed framework is much simpler and computationally efficient, which can be especially important in real-time applications where time and computational resources are limited. Additionally, as we freeze the convolution blocks and only optimize the BN affine parameters in SANTA, it less prone to catast forgetting as is less likely to overfit to the test time data. This important in test-time adaptation scenarios where the model must adapt new data quickly and accurately.
(ii) Inference time for fixed batch size: The inference time of model is an important factor in test-time adaptation because it determines how quickly the model can be applied to new data. In Figure g we have compared the inference time (in sec) per batch (of 200 samples) for RMT, CoTTA and SANTA. SANTA does forward and 1 backward pass per batch which takes 0.224 bec/batch. RMT does 2 forward, backward pass and model EMA update, which takes 0.26 sce/batch. On the other hand, CoTTA takes 2.2 sec soe/bbtae /bttc on average as it does 32 forward passes in the worst case and 1 backward pass, implying that SANTA is ~00 faster than CoTTA This can be attributed to the fact that COTTA uses 32 forward passes in the worst case
11

---

Domain- Published in Transactions on Machine Learning Research (11/2023)
SANTA
RMT
COTTA
0.5
2.5
Forward Pass
Backward optimisation
Post Backward
Figure Comparison of inference time of the proposed SANTA framework with RMT and CoTTA. The x-axis is time (wall clock time) of inference per batch (sec/batch). These experiments were done CIFAR- 100C using NVIDIA GeForce RTX 3090.
for its prediction and also updates the teacher model after every step. Although RMT and SANTA are both computationally efficient compared to COTTA, our method has clear advantage in low batch size setting as it significantly outperforms RMT, and hence is more effective in varied settings.
7 Conclusion
In this paper, proposed novel SANTA framework for the challenging task of continual test-time domain adaplation. The proposed approach modifies the batch-norm afline parameters using source anchoring- based delf-distillation to ensure the model incorporates knowledge of newly encountered domains while avoiding catastrophic forgetting. Additionally, source prototype guided target alignment proposed to maintain the already learned semantic information while grouping targel samples naturally. The approach is quile robust decreasing batch sizes, justifying effectiveness for online application. The SANTA framework offers additional advantages such as retaining performance on the source domain, having minimal tunable hyper- parameters and storage requirements, in addition to achieving state-of-thes the---rttt results on all the benchmark datasets.
8 Acknowledgements
This work is partly supported through research grant from SERB, Department of Science and Technology, Govt. of India (SPF/ OPFF2201/0001188) The authors would like to thank Jayateja Kalla. for helpful discussions throughout this work.
References
Hana Ajakan, Pascal Germain, Hugo Larochelle, François Laviolette, and Mario Marchand. adversarial neural networks. arXiv preprint arXim:1412.4444 2014.
Alexander Bartler, Andre Bühler, Felix Wiewel, Mario Döbler, and Bin Yang. Mt3: Meta test-time training for self-supervised lest-time adaption. In International Conference on Artificial Intelligence and Statislics, pp. 3080-30900 PMLR, 2022.
Malik Boudiaf, Romain Mueller, Ismail Ben Ayed, and Luca Bertinetto. Parameter-free online test-time adaptation. In CVPR, pp. 8344-8353, 2022.
Dhanajit Brahma and Piyush Rai. A probabilistic framework for lifelong test-time adaptation. In CVPR, pp. 3582-3591, 2023.
Mathilde Caron, Ishan Misra, Julien Mairal, Priya Goyal, Piotr Bojanowski, and Armand Joulin. Unsuper- vised learning of visual features by contrasting cluster assignments. NewrIPS, 33:9912 2299924, 2020.
12

---

Published in Transactions on Machine Learning Research (11/2023)
Dian Chen, Dequan Wang, Trevor Darrell, and Sayna Ebrahimi. pp. 295-305, 2022.
Contrastive test-time adaptation. In CVPR,
Liang-Chiele Chen, George Papandreou, Tasonas Kokkinos, Kevin Murphy, and Alan L. Yuille. Deeplab: Semantic image segmentation with deep convolutional nets, atrous convolution, and fully connected crfs. IEEE Transactions on Pattern Analysis and Machine Intelligence, 40(4):8344 848, 2018
Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. pp. 1597-1607. PMIR, 2020.
Francesco Croce, Maksym Andriushchenko, Vikash Sehwag, Edoardo Debenedetti, Nicolas Flammarion, Mung Chiang, Prateek Mittal, and Matthias Hein. Robustbench: a standardized adversarial robustness benchmark. In NewrIPS, 2021.
J. Deng, W. Dong, R. Socher, L. J. Li, K. Li, and L. Fei-Fei. database. In CVPR, 2009.
Imagenet: A large-scale hierarchical image
Mario Döbler, Robert Marsden, and Bin Yang. Robust mean teacher for continual and gradual test-time adaptation. In CVPR, pp. 7704-7714, 2023.
Songlin Dong, Xiaopeng Hong, Xiaoyu Tao, Xinyuan Chang, Xing Wei, and Yihong Gong. Few-shot class- incremental learning via relation knowledge distillation. In AAAI, volume 35, pp. 1255 1263, 2021.
Cian Eastwood, Ian Mason, Christopher KI Williams, and Bernhard Schölkopf. Source-free adaptation to measurement shift via bottom-up feature restoration. arXiv preprint vrXic:2107.05440 2021.
Mark Everingham, Luc Gool, Christopher K. Williams, John Winn, and Andrew Zisserman. The pascal visual object classes (voc) challenge. IJC,, 2010.
Yixin Fei, Zhongkai Zhao, Siwei Yang, and Bingchen Zhao. Xcon: Learning with experts for fino-grained category discovery. arXiv preprinl ar Xiv:2220 01898, 2022.
Yulu Gan, Xianzheng Ma, Yihang Lou, Yan Bai, Renrui Zhang, Nian Shi, and Lin Luo. Decorate the newcomers: Visual domain prompt for continual test time adaptation. orXiv preprint orXiv:2212.04145, 2022.
Tacsik Gong, Jongheon Jeong, Taewon Kim, Yewon Kim, Jinwoo Shin, and Sung-Ju Lec. NOTE: Robust continual test-time adaptation against temporal correlation. In Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho (eds.), NewrIPS, 2022.
Jianping Gou, Baosheng Yu, Stephen J Maybank, and Dacheng Tao. Knowledge distillation: A survey. IJV,, 129:1789 1819, 2021.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In CVPR, pp. 770-778, 2016.
Kaiming He, Georgia Gkioxari, Piotr Dollár, and Ross Girshick. Mask R-CNN. In ICCV, 2017.
Yufan He, Aaron Carass, Lianrui Zuo, Blake E Dewey, and Jerry Prince. Autoencoder based self-supervised test-time adaptation for medical image analysis. Medical image analysis, 72:102136, 2021.
Dan Hendryeks and Thomas Dietterich. Benchmarking neural network robustness common corruptions and perturbations. arXiv preprint arXiv:1903.122: 2019.
Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the knowledge in a neural network. arxiv preprint arXiv:1503.02555 2015.
Jiaxing Huang, Dayan Guan, Aoran Xiao, and Shijian Lu. Model adaptation: Historical contrastive learning for unsupervised domain adaptation without source data. NewrIPS, 34:3635-3044 2021.
13

---

Published in Transactions on Machine Learning Research (11/2023)
Minsoo Kang, Jacyoo Park, and Bohyung Han. Class-incremental learning by knowledge distillation with adaptive feature consolidation. In CVPR, pp. 16071-16080, 2022.
Prannay Khosla, Piotr Teterwak, Chen Wang, Aaron Sarna, Yonglong Tian, Phillip Isola, Aaron Maschinot, Ce Liu,, and Dilip Krishnan. Supervised contrastive learning. NeurIPS, 33:18661-18677 2020.
Daehee Kim, Youngjun Yoo; Seunghyun Park, Jinkyu Kim, and Jaekoo Lee. Selfreg: Self-supervised con- trastive regularization for domain generalization. In ICCV, pp.
2021.
Jogendra Nath Kundu, Naveen Venkat, R Venkatesh Babu, et al. Universal source-free domain adaptation. In CVPR, pp. 4554,4553, 2020.
Rui Li, Qianfen Jiao, Wenming Cao, Hau-San Wong, and Wu. Model adaptation: Unsupervised domain adaptation without source data. In CVPR, pp. 9641 9650, 2020.
Jian Liang, Dapeng Hu, and Jiashi Feng. Do really need to access the source data? source hypothesis transfer for unsupervised domain adaptation. In ICML, pp. 6028-6039. PMLR, 2020.
Hyesu Lim, Byeonggeun Kim, Jaegul Choo, and Sungha Choi. TTN: A domain-shift aware batch normal- ization in test-time adaptation. In ICLR, 2023.
Yuejiang Liu, Parth Kothari, Bastien Germain van Delft, Baptiste Bellot-Gurlet, Taylor Mordan, and Alexan- dre Alahi. Ttt++: When does self-supervised test-time training fail thrive? In NewrIPS, 2021.
Hossein Mobahi, Mehrdad Farajtabar, and Peter Bartlett. Self-distillation amplifies regularization in hilbert space. NeurIPS, 33:3351-3361, 2020.
Chaithanya Kumar Mummadi, Robin Hutmacher, Kilian Rambach, Evgeny Levinkov, Thomas Brox, and Jan Hendrik Metzen. Test-time adaptation to distribution shift by confidence maximization and input transformation. arXiv preprint arXiv:2106. 149999 2021.
Shuaicheng Niu, Jiaxiang Wu, Yifan Zhang, Yaofo Chen, Shijian Zheng, Peilin Zhao, and Mingkui Tan. Efficient test-time model adaptation without forgetting. In ICML, pp. 16888 16905. PMLR, 2022.
Yaniv Ovadia, Emily Fertig, Jie Ren, Zachary Nado, David Sculley, Sebastian Nowozin, Joshua Dillon, Balaji Lakshminarayanan, and Jasper Snoek. Can you trust your model's uncertainty? evaluating predictive uncertainty under dataset shift. NewrIPS, 32, 2019.
Ori Press, Steffen Schneider, Matthias Kümmerer, and Matthias Bethge. Rdumb: A simple approach that questions our progress in continual lest-time adaptation. Xix preprint Xiv::306.00540, 2023.
Shaoqing Ren, Kaiming He, Ross Girshick, and Jian Sun. Faster r-cmn: Towards real-time object detection with region proposal networks. NewrIPS, 2015.
Evgenia Rusak, Steffen Schneider, George Pachitariu, Luisa Eck, Peter Vincent Gehler, Oliver Bringmann, Wicland Brendel, and Matthias Bethge. If your data distribution shifts, use self-lcarning. Transactions on Machine Learning Research, 2022.
Kuniaki Saito, Kohei Watanabe, Yoshitaka Ushiku, and Tatsuya Harada. Maximum classifier discrepancy for unsupervised domain adaptation. Tn CVPR, pp. 3723 3732, 2018.
Steffen Schneider, Evgenia Rusak, Taisa Eck, Oliver Bringmann, Wieland Brendel, and Matthias Bethge. Improving robustness against common corruptions by covariate shift adaptation. NewrIPS, 2020.
Maohao Shen, Yuheng Bu, and Gregory Wornell. On the benefits of selectivity in pseudo-labeling for unsupervised multi-source-fice domain adaptation. arXie preprint arXiv:2202.000 007966 2022.
Y. Sun, X. Wang, Z. Liu, J. Miller, A. A. Efros, and M. Hardt. Test-time training with supersupen for generalization under distríbution shifts. Tn TCML, 2020.
14

---

Published in Transactions on Machine Learning Research (11/2023)
Jiayi Tian, Jing Zhang, Wen Li, and Dong Xu. Vdm-da: Virtual domain modeling for source data-free domain adaptation. IEEE Transactions on Circuils and Systems for Video Technology, 32:3749-377 2021.
Sagar Vaze, Kai Han, Andrea Vedaldi, and Andrew Zisserman. Generalized category discovery. Tn CVPR, pp. 7492-7501, 2022.
Dequan Wang, Evan Shelhamer, Shaoteng Liu, Bruno Olshausen, and Trevor Darrell. Tent: adaptation by entropy minimization. In ICLR, 2021.
Fully test-time
Qin Wang, Olga Fink, Luc Van Gool, and Dengxin Dai. Continual test-time domain adaptation. In CVPR, pp. 7201 1211,, 2022a.
Rui Wang, Zuxuan Wu, Zejia Weng, Jingjing Chen, Guo-Jun Qi, and Yu-Gang Jiang. Cross-domain con- trastive learning for unsupervised domain adaptation. IEEE Transactions on Mullimedia, 2022b.
Haifeng Xia, Handong Zhao, and Zhengming Ding. Adaptive adversarial network for source-Iree domain adaptation. In ICCV, pp. 9010 9019, 2021.
Saining Xie, Ross Girshick, Piotr Dollár, Zhuowen Tu, and Kaiming He. Aggregated residual transformations for deep neural networks. In CVPR, pp. 1492 1500, 2017.
Chenglin Yang, Lingxi Xie, Chi Su, and Alan L Yuille. Snapshot distillation: Teacher-student optimization in one generation. In CVPR, pp. 2859-2868, 2019.
Shiqi Yang, Joost van de Weijer, Luis Herranz, Shangling Jui, al. Exploiting the intrinsic neighborhood structure for sourco-free domain adaptation. NewrIPS, 0929393-29405 2021a.
Shiqi Yang, Yaxing Wang, Joost Van De Weijer, Luis Herranz, and Shangling Jui. Generalized sourco-free domain adaptation. In ICCV, pp. 8978 8987, 2021b.
Shiqi Yang, Yaxing Wang, Kai Wang, Shangling Jui, et al. Attracting and dispersing: A simple approach for source-free domain adaptation. In NeurIPS, 2022.
Xufeng Yao, Yang Bai, Xinyun Zhang, Yuechen Zhang, Qi Sun, Ran Chen, Ruiyu Li, and Bei Yu. Pel: Proxy-based contrastive learning for domain generalization. In CVPR, pp. 7097-7107, 2022.
Li Yuan, Francis EH Tay, Guilin Li, Tao Wang, and Jiashi Feng. Revisit knowledge distillation: a teacher-free framework. 2019.
Sergey Zagoruyko and Nikos Komodakis. Wide residual networks. In BMVC, pp. 07.1-87.12, 2016.
Marvin Zhang, Sergey Levine, and Chelsea Finn. Memo: Test time robustness via adaptation and augmen- tation. NewrIPS, 35:38629-38644 2022.
Bowen Zhao, Chen Chen, and Shu-Tao Xia. Delta: degradation-fies fully test-time adaptation. arXiv preprint arXiv:2301.1300 13018, 2023.
Xuejun Zhao, Rafal Stanislawski, Paolo Gardoni, Maciej Sulowicz, Adam Glowacz, Grzegorz Krolczyk, and Zhixiong Li. Adaptive contrastive learning with label consistency for source data free unsupervised domain adaptation. Sensors, 4211) 2022.
15

---

Published in Transactions on Machine Learning Research (11/2023)
Appendix
A Experiment on CCC
This benchmark was pointed out during the review process. The Continuously Changing Corruptions (CCC) Press et al. (2023) benchmark is generated from ImageNet-CC by interpolating between domains in ImageNet- C to produce any number of samples. By default, CCC consists of 7.5 million samples which is 10x more than the samples in ImageNet-C. This ereates benchmark which tracks the long term performance of CTTA frameworks. The CCC benchmark shows that prior CTTA methods fail across long adaptation lime-soales. Even in this challenging long term setting, we observe that SANTA performs consistently better than prior methods as shown in Table g
Method
CCC-Medium
CCC-Hard
BNStats
27.9 #0.74
6.00 00.31
CoTTA
7.70 #0.43
1.10 =0.16
ETA
1.1 +0.43
0.23 =0.05
EATA
35.4 +1.0
8.70 =0.80
RDumb
38.9 #1.4
9.60 1.60
SANTA
32.7 #0.8
9.10 =0.60
Table 9: Percentage accuracy of CTTA methods (higher is better) on CCC benchmark.
B Ablation on Temperature
0.005
0.01
0.05
0.1
0.5
1.0
5.0
CIFARIO-C
16.1
16.1
16.1
16.1
16.3
16.5
16.7
CIFAR100-C
30.3
30.3
30.3
30.3
30.8
31.2
31.3
ImageNet-C
60.6
60.4
60.3
60.2
61.0
61.1
61.2
Table 10: Ablation showing the sensitivity of our method on all considered datasets. The above numbers are error (lower better)
Table 10 shows the performance of the model for different temperature values. The best performance is approximately achieved for a temperature value of 0.1, indicating that a soft probability distribution is more suitable for this model. It also worth noting that the model fairly resilient large range of temperature values, implying that the model's performance does deteriorate significantly for different temperature values. This suggests that the model is robust to variations in temperature hyperparameter and performs reasonably well for wide range of values.
C Predict and Adapt Code
Here,
we present the code for our method. This code is in the format of code provided by the CoTTA Wang
al.
2022a) and TENT TEng et al. 2021 The code for contrastive learning loss can be found here Khosla
et al.
2020
def forward and add__ddpt(x) imgs lest = x [0]
optimizer zero grad
# forward original test data
16

---

Published in Transactions on Machine Learning Research (11/2023)
features_ test feature_extraaee (ings_tesisssss outputs test classifier caaeiifferreaattii (features_ test)
# forward augmented test data
features aug Lest feature extractor (it transform ((imgs test ) outputs auiputs_aug_test aug_test
classifier
_ariees__uu___ aug_ test)
# forward original test data through the source model outputs_sre src = model_sre (imgs_test)
with torch.m. no grad ():
# disti: il contains the distance from # every source sample to one test sample dist = cosine similarity
prototypes_ repeat)
1, features test shape [0],
)
features test... unsqueeze (0). repeat ( catrssssr.sshhppppes shape[0]. 1,
prototypes src
) dim=1
# for every test feature, get the nearest # source prototype and derive the label indices = d t1, largest=True, dim=0) indices = indices squecze
# Here we see the 3-views that are passed to contrastive loss features torch. cat ([prototypes_sre [indcess]
features test unsqueeze (1), features_ aug test. unsqueeze (1)], dim=1)
# Adapted from: /ttpoobiloooooSSonooorra///////d # hitps: // coattsstteeeeveee contrastive_los (features=features)
loss_ entropy symmetric_entropp (x-rroppt=oouuttt (x=outputs. test
x aug-outputs aug test
x_src=outputs_sre mean (0)
losll_sncroppp entropy + loss contrastsee loss trg backward ()
trg loss
optimizer. step
return outputs_tess test
We finally take the arg max(output. test) across the sample dimension to get our final prediction for the target batch.
D Augmentation
We the same augmentations used in CoTTA here, we explicitly report the series of random augmen- fations along with their ranges for better reprodueibility.
17

---

Published in Transactions on Machine Learning Research (11/2023)
n_pixels shape [0] clip min, clip max 0.0, 1.0
P hflip = 0.5
gaussian std = 0.005
transforms = transforms. Compose (1
Clip (0.0, 1.0).
ColorJitterProl
brightness=(0.6 [0.6, 1.4],
contrast=(0.7 1.3],
Ialuratiion(0..11 1.5],
-0.06, 0.06],
gamma=[0.7, 1.3]
transforms Pad padding-intt (n_ppinnttttiellllll / 2),
padding mode="edde ))
transforms Random
degrees=[-15, 15), mansslate((/I1)) 1/16), scale=(0.9, 1.1), shear=None,
resample-PIL. Image.BBI.INEE..... BILINEAR, fillcolor-None
) transforms. GaussianBlur (kernel_ssee (
t
[0.001, 0.5]),
transforms. CenterCroo(sizeeee pixels);
transforms. Randommlorizontaallip (p=p_hflip)
GaussianNoise (0, gaussian_std) Clip /cipp_min, maxp
])
return tta transforms
18


Reviews:
[{"review_id": "41EZi9y8LT", "reviewer": "Reviewer_aE44", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Pros:\n1.  The motivation and organization are clear and easy to follow. \n2. The proposed method achieves on-bar or slightly better performance than the comparing methods.\n3. Experiments have sufficient ablation study including the discussion of batch size, inference time/cost, different propose module and etc. \n\nCons:\n1. Novelty\nMy major concern lies in the novelty of this work. First, batch-normalization optimization in CTTA has been widely explored such as TENT, [1,2]. Methods along this direction naturally have fewer parameters and better inference time.  Second, the self-distillation loss has been explored in [2]. Contrastive learning or pseudo labels with neighbor information has been discussed in [3,4]. In short, I think the paper integrates several effective techniques for relating DA topics to the CTTA task. The novelty is a bit incremental. \n\n2. Motivation\nThe author states that a good TTA model 1) can work effectively for different (even small) batch sizes; 2) should continue to work well on the source domain; and 3) should have minimal tunable hyperparameters and storage requirements. As I mentioned, the batch normalization optimization method can naturally achieve (3) based on TENT, the student-teacher model can help (2) from RMT and CoTTA. Yet, the paper hasn't proposed any specific method to handle (1). The proposed method has comparable performance with CoTTA and RMT over the different sizes of batches in Figure 3. There is a lack of justification on why the proposed method can help on this feature. \n\n\n[1] TTN: A DOMAIN-SHIFT AWARE BATCH NORMALIZATION IN TEST-TIME ADAPTATION, ICLR23\n[2] EcoTTA: Memory-Efficient Continual Test-time Adaptation via Self-distilled Regularization, CVPR23\n[3] Discovering Informative and Robust Positives for Video Domain Adaptation, ICLR23\n[4] Domain Adaptation with Auxiliary Target Domain-Oriented Classifier, CVPR21", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper tackles the problem of continuous test-time adaptation (CTTA) problem. The paper states the model should handle flexible batch size, forgetting from source and minimum tunable parameters. The authors propose to optimize batch normalization layers with source anchoring distillation loss and source-driven contrastive loss. Experiments show improvement over the comparing method and sufficient ab lation study has been done.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "In general, I think the proposed method is incremental w.r.t the existing works and also lack of justification on why it can help various batch size. I suggest the authors strength this part.", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "Ahz0b0j53A", "reviewer": "Reviewer_qTGo", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n\nImportant research problem: The paper focuses on a crucial problem related to continual test time adaptation. Continual learning and adaptation are essential in dynamic environments, where the distribution of data may change over time. The relevance of the problem is a major strength, as it contributes to enhancing the practical applicability of machine learning models.\n\nComprehensive framework: The proposed framework, involving the Source Anchoring Network and Target Alignment, demonstrates a holistic approach to addressing the challenges of continual test time adaptation. This comprehensive methodology adds value to the paper by providing a complete solution that tackles multiple aspects of the problem.\n\nExperimental validation: The paper supports its claims with experiments on multiple datasets. The inclusion of diverse datasets enhances the generalizability of the proposed approach and demonstrates its effectiveness across various scenarios. This empirical validation adds credibility to the practical viability of the framework.\n\nWeaknesses:\n\nLack of novelty: One notable weakness is the perceived lack of novelty in the proposed framework. The components, such as the Source Anchoring Network and Target Alignment, may not significantly deviate from existing approaches in the field. Providing a clearer differentiation between the proposed method and previous works, along with innovative elements, would enhance the paper's contribution. A meirt is that the entire method framework can address many challenges present in practical scenarios (e.g. small batch size, hyperparameter tuning, etc).\n\nInsufficient theoretical analysis: The paper could benefit from a more comprehensive theoretical analysis. Explaining the underlying principles, mechanisms, and theoretical foundations of the proposed framework would contribute to a deeper understanding of its functioning. This analysis would elevate the scientific rigor of the paper and provide insights into why certain design choices were made.\n\nReadability improvement: While the paper covers a complex topic, its readability could be improved. The organization of the content, clarity in explanations, and effective use of visual aids (if applicable) can significantly enhance the paper's accessibility to a wider audience. Ensuring that the paper is easy to understand for both experts and those less familiar with the topic is important for its impact.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper introduces the challenge of adapting models to changing test environments and presents the SANTA framework. SANTA adapts online with key features like effectiveness for different batch sizes, maintaining source domain performance, and minimal parameter requirements. It achieves this by adjusting batch normalization parameters using source anchoring self-distillation. Additionally, a source-prototype contrastive alignment is proposed to group target samples naturally while retaining learned semantics. Extensive evaluations on tough benchmarks validate SANTA's applicability for real-world scenarios.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Here are some suggested modifications: emphasize the differences between the proposed method and existing ones, enhance readability, and increase theoretical explanations or interpretability", "broader_impact_concerns": "No", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "p4tVzaR8mI", "reviewer": "Reviewer_noN9", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "## Strengths:\n\nThe problem of continuous test-time adaptation is important.\n\nThe paper is generally well-written and easy to follow.\n\nThe approach is fairly clear and well-motivated.\n\nThe authors compare their method to several baselines, although they did miss some.\n\nThe Tables and Figures are well designed and easy to read.\n\n## Weaknesses:\nI have some questions with regards to the experiments which I formulated in the Requested Changes field more explicitly:\n\n1)\tIt is not clear to me whether the authors used 10 corruption orderings as has been done in CoTTA.\n2)\tI wonder whether the evaluation might suffer from a known bug that robustbench only gives you 5k examples when loading ImageNet-C.\n3)\tIt is not clear to me whether their method would collapse (just like CoTTA) does when evaluated on a longer time-scale.\n4)\tSome related works have been missed and should be included, see list below.\n5)\tSome interpretations of the results are wrong: The authors write that their method outperforms all other methods which is not true looking at the numbers in the Table.\n\nIn general, the results in this paper are very close to the baseline numbers the authors compare to. Their approach is much more computationally efficient which is nice. Thus, even if the results are on par with other SotA approaches, but the method is much more efficient, this would be fine. But then, the authors should change their wording, and not claim that their method “outperforms all other methods”. The wording that SANTA is an effective TTA method (used in the abstract + contributions) is fine.\n\n\n### Related work:\nThe authors should include the following works in their TTA section:\n\n- [A] Liu et al. “Ttt++: When does self-supervised test-time training fail or thrive?”\n- [B] Eastwood et al. “Source-free adaptation to measurement shift via bottom-up feature restoration”\n- [C] Mummadi et al. “Test-time adaptation to distribution shift by confidence maximization and input transformation”\n- [D] Rusak et al. “If your data distribution shifts, use self-learning”\n- [E] Bartler et al. “Mt3: Meta test-time training for self-supervised test-time adaption”\n- [G] Zhang et al. \"MEMO: Test Time Robustness via Adaptation and Augmentation\"\n\nI don’t think that any of these works consider continuous TTA, so it’s not necessary to include them as baselines.\n\n### General:\nPage 2: “The proposed SANTA framework also utilizes contrastive learning to ensure better model generalizability to unseen domains.” -> It is not clear how contrastive learning would help here. Not enough information for this claim, not well motivated. Please either provide references or expand upon this thought.\n\nPage 2: “SFDA” has not been introduced as an abbreviation, please fix.\n\nTable 1: I would include citations next to the methods to make it easier for the reader to jump to the references section.\n\n[**major**]: “We observe that for all the datasets, the proposed SANTA outperforms all the other existing\napproaches.” -> This is not true: PETAL outperforms SANTA on CIFAR10-C, and SANTA performs within 1 p.p. better than RMT on CIFAR100-C and ImageNet-C. I don’t think this is even significant.\n\nFigure 3: I would add “Error rate” as the y-label to the left of the first subplot.\n\n[**major**] Figure 3: The BN adapt numbers look strange. Schneider et al. show the performance of a RN50 depending on the batch size, and find that with a batch size of 8, they get more or less the baseline performance and with a batch size of 64, they get close to optimal performance. The difference in error rate should be something around 10-15 p.p., while in Fig. 3c, the performance looks fairly constant for the orange curve across the batch sizes.\n\n### Typos:\n- Page 1: “unlabeled”\n- Page 1: “For example, in autonomous driving, the model trained using data captured in clear weather, can encounter cloudy weather followed by heavy rain, etc. during deployment.” -> remove comma before “can”\n- Page 3: “Recent methodsDöbler” space missing\n- Page 3: “Sppecifically,”\n- Multiple occasions: “a” missing, for example: “Recent methodsDöbler et al. (2023); Brahma & Rai (2023) which have achieved better performance than CoTTA also use teacher student framework to update the adapting model.” -> “a” teacher-student framework\n- Commas are missing when multiple references are cited.\nThe authors should use \\citet and \\citep instead of \\cite to improve readability. \n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The authors study the task of continuous test-time adaptation when the test distribution continuously changes over time. They propose a new test-time adaptation loss which consists of two parts. The first part is a source anchoring network which is supposed to prevent the model to divert too strongly from its initial condition. The second part of the loss is meant to increase the target alignment of the adapted model with a contrastive loss. The authors show results on CIFAR10-C, CIFAR100-C and ImageNet-C.", "claims_and_evidence_yn": "No", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "In CoTTA, the authors average the results across 10 corruption orders. Has this been done in this paper as well? If not, the experiments should be rerun. \n\nThere is a known bug if the CoTTA evaluation code is used, such that only 5k images are sampled instead of 50k: https://github.com/qinenergy/cotta/blob/main/imagenet/conf.py, line 55. The authors of robustbench are aware of this issue: https://github.com/RobustBench/robustbench/issues/92. Could the authors make sure that their evaluation does not suffer from this bug?\n\nPlease include ETA as baseline in your method. ETA does not include source samples as part of their method, so the reasoning that the authors cannot use this method as comparison does not hold. The authors will need to run this experiment as no ETA (lifelong) results were included in the ETA paper.\n\nThe recently released Continuously Changing Corruptions [F] benchmark has been proposed to examine the continuous test-time adaptation over long time scales. They find that all SotA TTA techniques eventually collapse to chance level or at least baseline performance. Since this approach is quite similar to CoTTA and CoTTA does collapse over time, could the authors please test on the CCC benchmark to test whether SANTA maintains performance over a longer time-scale? If the authors evaluate on CCC, comparing to RDumb, the method which was proposed in CCC would also be interesting.\n\n[F] Press et al. “RDumb: A simple approach that questions our progress in continual test-time adaptation”\n\nPlease include the missing related works and respond to / change the text following my remarks in the Strengths and Weaknesses section above. Especially the paragraphs denoted by **major** are important. Please do a round of proof-reading to eliminate all typos.\n\nThe paper would benefit from a \"Limitations\" section.\n", "broader_impact_concerns": "The authors did not include a section on ethical implications of this work. They use publicly available datasets and models, so I do not have any specific ethical concerns for this work.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper proposes a new method for tackling an important research problem: continual test-time adaptation. Their method is efficient, simpler than previous approaches, entirely source-free and works well (equally well or better) than a wide range of previous baselines on various challenging datasets. It is also effective for smaller batch sizes which is important for online adaptation. The authors conduct a thorough empirical study to support their claims. The reviewers also pointed out that the paper is well-written and well-organized.

I am not concerned about reviewer comments on novelty, since novelty is not a criterion for TMLR. In addition, I agree with the authors' response to those reviewers that assembling existing components in a new way is not obvious, and neither of the existing components alone possessed all desirable components that the proposed approach does.
Decision: Accept (poster)