Continual Momentum Filtering on Parameter Space for Online Test-time Adaptation
Authors: Jae-Hong Lee
Joon-Hyuk Chang
Venue: ICLR 2024 poster (2024)
Keywords: Online Test-time Adaptation
Catastrophic Forgetting
Kalman Filter
Primary area: transfer learning, meta learning, and lifelong learning
TL;DR: We propose continual momentum filtering framework, which a novel approach to bolster the online test-time adaptation methodology. This was achieved by deducing a refined source model through target model denoising by leveraging the Kalman filtering.

Abstract: Deep neural networks (DNNs) have revolutionized tasks such as image classification and speech recognition but often falter when training and test data diverge in distribution. External factors, from weather effects on images to varied speech environments, can cause this discrepancy, compromising DNN performance. Online test-time adaptation (OTTA) methods present a promising solution, recalibrating models in real-time during the test stage without requiring historical data. However, the OTTA paradigm is imperfect, often falling prey to issues such as catastrophic forgetting due to its reliance on noisy, self-trained predictions. Although some contemporary strategies mitigate this by tying adaptations to the static source model, this restricts model flexibility. This paper introduces a continual momentum filtering (CMF) framework, leveraging the Kalman filter (KF) to strike a balance between model adaptability and information retention. The CMF intertwines optimization via stochastic gradient descent with a KF-based inference process. This methodology not only aids in averting catastrophic forgetting but also provides high adaptability to shifting data distributions. We validate our framework on various OTTA scenarios and real-world situations regarding covariate and label shifts, and the CMF consistently shows superior performance compared to state-of-the-art methods.

Published as a conference paper at ICLR 2024
CONTINUAL MOMENTUM FILTERING ON PARAMETER SPACE FOR ONLINE TEST-TIME ADAPTATION
Jae-Hong Lee . Joon-Hyuk Chang
Department of Electronic Engineering
Hanyang University
Seoul, Republic of Korea
{1jh931jh, jehang jchanggJhanyaaaeeeeee @hanyang ac.
ABSTRACT
Deep neural networks (DNNss) have revolutionized tasks such as image classifi- cation and speech recogmition but often falter when training and test data diverge in distribution. External factors, from weather effects on images to varied speech environments, can cause this discrepancy, compromising DNN performance. On- line test-time adaptation (OTTA) methods present a promising solution, recali- brating models in real-time during the test stage without requiring historical data. However, the OTTA paradigm is imperfect, often falling prey to issues such as catastrophic forgetting due its reliance noisy, self-trained predictions. Al- though some contemporary strategies mitigate this by tying adaptations to the static source model, this restricts model flexibility. This paper introduces con- tinual momentum filtering (CMF) framework, leveraging the Kalman filter (KF) strike a balance between model adaptability and information retention. The CMF intertwines optimization via stochastic gradient descent with a KF-based inference process. This methodology not only aids in averting catastrophic forget- ting but also provides high adaptability to shifting data distributions. We validate our framework on various OTTA scenarios and real-world situations regarding covariate and label shifts, and the CMF consistently shows superior performance compared to state-of-the-art methods.
INTRODUCTION
Deep neural networks (DNNs) have been successfully applied to challenging tasks such as image classification (Krizhevsky et al. 2012|Simonyan & Zisserman) 2014 He et al. (016b) and speech Graves et al., 2013] Jelinek] 1997). The success of DNNs stems from the assumption that training and test data are drawn from the same distribution (Goodfel- low et al. 2016, Murphy, 2023). However, this assumption is difficult to maintain in real-world environments owing to external factors (Hendrycks & Dietterich 2019b Koh et al. 2021). For example, images may be damaged due to weather changes or sensor degradation during the image classification tasks. Similarly, in speech recognition tasks, discrepancies arise from differences in the speaking environment the frequency of words used by speakers compared with the source data. These distribution shifts lead to a significant performance degradation of the DNN models Quinonero-Candel et al.] 2008: [un et al., 2017). To address these distributional discrepancies, there is growing interest in online test-time adaptation (OTTA) methodologies (Wang et al.) 2020 Zhang et al. 2022). These innovative frameworks are equipped to recalibrate models on-the-fly during the test stage. Impressively, they achieved this without delving back into historical data, leaning exclusively from the knowledge gleaned from the pre-trained source model. This real-time adaptation bypasses many challenges associated with data storage, retrieval, and potential privacy infringement.
recognition (Hinton et al. 2012
Nevertheless, the journey of the OTTA is with roadblocks. Rooted in a self-training paradigm, it habitually employs its own noisy predictions as training targets during adaptation. This recursive feedback induce series of complications, most notably catastrophic forgetting, in which
Corresponding author.

---

Published as a conference paper at ICLR 2024
Inference Process
1p 10000,90000 (01)
63 (2
Optimization Prones
Middde
1gightllig إoft Extasion petf! gf
Figure 1: Illustration of the CMF framework. The optimization process draws the target model parameter g(t) by minimizing the target generalization error Ey using SGD the target data D.g The inference process derives 0(++1) with the refined source model's parameter (hidden) ((()) and the target model's parameters (observation) (1f) thorough the parameter ensemble method.
Furthermore, there looming threat of mode collapse, where models disproportionately amplify the probabilities of specific classes at the expense of others (Boudiaf et al. 2022). Contemporary solutions, ranging from calibrating particular neural layers (Yang et al. 2022; Zhao et al., 2023; Hong
models inadvertently overwrite or discard previously assimilated information.
et al.] 2022) to tethering adaptations to static source model information, have been proposed (Wang et al. 2022 Niu et al. 2022 2023; Marsden et al., (023). SAR Niu et al. (2023) stochasti- cally restores the target model parameters to those of the source model during adaptation. Similarly, ROID (Marsden et al., 2023) continuously ensembles the parameters of the source and target models to prevent loss of past information. These strategies regularize the target model to prevent it
from diverging too far from the source model, thereby preventing catastrophic forgetting.
However, the continuous use of frozen source model information restricts the flexibility of the target model, making it difficult to adapt to distribution shifts. For more flexibility, we need to update the source model using noisy information from the target model; however, this carries the risk of catastrophic forgetting. The Kalman filter (KF) (Särkkä & Svensson, 2023) can emerge as beacon in the quest for harmonious balance between adaptability and information retention. The KF an exact Bayesian filtering algorithm that accumulates past observations (i.e., parameters of the target model) along with denoising observations. The linear Gaussian state-space model (I. (IGG-SM),, which forms the basis of the KF, consists of three types of random variables: additional input, hidden, and observation. These random variables comprise two types of linear Gaussian models: the transition model, which models the evolution of the hidden variable over the adaptation time steps, and the emission model, which predicts the observations from the hidden variable.
In this paper, we propose continual momentum filtering (CMF) framework that utilizes the KF algorithm to find a source model that is robust against catastrophic forgetting while maintaining high flexibility. The CMF alternates between descent (SGD) (Ruder 2016) and an inference process based on the KF. During the optimization process, the target model is obtained by minimizing the target generalization error using an unsu- pervised loss function. For the inference process, the CMF conjugates the source model with the transition model continuously preserve the information in the hidden variable (i.e., parameters of a refined source model) and an emission model to update the hidden variables by denoising the pa- rameters of the target model. Finally, the refined source and target models are ensembled to perform the following optimization process. Through CMF, we utilize the information of the noisy target model to obtain a refined source model and use for adaptation, thus narrowing the gap between adaptability and robustness and providing new perspective on OTTA.
optimization process based stochastic gradient
We validate the proposed framework in various scenarios used in existing OTTA methods (Niu et al., 2022 2023 Marsden et al. 2023). These scenarios are broadly divided into covariate and label shifts and further classified based on the degree of time correlation of each input or label datum. For the covariate shifts, we experiment with scenarios in which the imput data drawn from different domains are mixed or appear together. For the label shifts, we gradually adjust the degree to which the same labels simultaneously appear over time. We also validate our framework speech recognition task
2

---

Published as a conference paper at ICLR 2024
for streaming scenarios. The CMF demonstrates significant performance improvements in these wide ranges of scenarios compared to state-of-the-ar methods.
BACKGROUND
This section elaborates on the OTTA problem and regularization methods for preventing catastrophic forgetting. First, outline the setup of the problem (Section 2.1) and subsequently introduce strategy of existing studies to mitigate catastrophic forgetting using the source model (Section 2.2).
2.1 TARGET GENERALIZZAION ERROR FOR ONLINE TEST-TIME ADAPTATION
Let the labeled data from source distribution ps be Ds (().. ps 1: NN and the unlabeled data from target distribution Py at each time step t E f1,2...., , be D fx, ~
(t)
PT n 3 where ps + p The main objective is to minimize the generalization error on the target distribution p which can be calculated
( 00 PT
E ppp [(([(x,; (()))))
where f(:; represents the DNN pre-trained on Ds and E(.) is the unsupervised loss function. The target generalization error can be approximated empirically as follows:
(T( )
Σ (((());(())) EDD
NT
OTTA literature frequently utilizes entropy-based loss functions because the target model is adapted unlabeled target data (Wang al., 2020 Zhang al., 2022 Chen al. 2022). In particular, TENT (Wang et al. (020)) demonstrated the effectiveness of entropy-based loss in a self-training manner on single domain; however, recent research has revealed that methods solely based on self-training often fail in more complex scenarios, such as multiple domains (Boudiaf et al., 2022
Gong al. 2022 Niu et al. 2023).
2.2 PREVENTING CATASTROPHIC FORGETTING USING SOURCE MODEL
causes the adapted target model to experience catas- trophic forgetting. This results in a mode collapse problem, in which the probabilities assigned to certain classes become excessively large, causing significant degradation in performance (Niu al., 2023 Marsden et al. 2023). A popular approach for mitigating this issue involves minimizing the empirical generalization error alongside the regularization loss that inhibits the target model from diverging far from the source model BBudiif et al. (2022). The target model parameter ((+1) at adaptation time step / ++ 1 can be found as follows:
Minimizing the empirical generalization
θ (++1) arg min (T((() D(+1)) (
(3)
where (()( (())) represents the distance between the source and target models, and À is the regularization coefficient. Strategies that adapt only subset of the source model parameters, ((0) c ())), essentially perform a function to the regularization loss because the majority of the adapted parameters are identical those of the source model. In particular, methods that calibrate only batch normalization (BN) layers (loffe & Szegedy 2015 Li et al.) 2018 Mancini et al., 2018 Yang al. 2022 Zhao et al. 2023 Hong et al. | 2022) have been widely recognized for their effectiveness preventing catastrophic forgetting. However, these methods exhibit structural limitations when applicd to models that employ layer normalization (LN) layers (Ba et al. (016) because they alter
the functionality of BN (Lin et al. (022).
Recently, strategies that continuously transfer the information acquired from the source model to the target model (Wang et al., 2022 Niu et al.) 2023 2022. Marsden et al.) (023) have overcome these limitations, thereby alleviating the catastrophic forgetting problem across broader range of model structures. For instance, EATA (Niu et al. 2022) mitigates catastrophic forgetting by utilizing the
3

---

Published as a conference paper at ICLR 2024
Algorithm Continual Momentum Filtering
INPUT:
Input data stream {D() D Source model f(.; f(.:8%) Number of updates 1,
Hyperparameter (,, q, r; 7), Initialization ((0) <- ((0), Mojo t- g(o), Eojo +- 0
for / = 1,...,77 do
for = 11=1,-.. 1,..,, do
OPTIMIZATION PROCESS:
((1) = arg mingre Ey ()(((=1)),
b Eq. (4)
INFERENCE PROCESS:
// Predict Step:
Majt-1 Moments( 110)
D Eq. (15)
Edj-- = a a² 1|t-1 +
b Eq. (18)
// Update Step:
B: r /// +
b Eq. (19)
Mal: Moments (ee 1-1 ,8)
Eq. (16)
Exjt -e-
b Eq. (20)
// Parameter Ensemble:
(()) = Moments Peje:
b Eq. (17)
end for end for
Fisher information matrix computed from the source model and elastic weight consolidation loss (Kirkpatrick et al., (017). In contrast, SAR (Niu et al. 2023) does not employ additive loss but in- stead accesses the parameters directly, taking the strategy of averaging the loss values over time and reverting to the source model when they exceed a predetermined threshold. Similarly, ROID (Mars- den et al. 2023) continuously integrates the source model into the target model during adaptation to preserve past information using the parameter ensemble method (Wortsman et al. 2022 Rame al. 2022). These approaches maintain the functionality of specific parameter subsets correspond- ing to layers without altering them, making them model-agnostic and demonstrating state-of-the-aar performance. However, they impose constraints on the flexibility of the target model by utilizing frozen source model.
3 METHODOLOGY
This section elucidates our methodology for constructing a flexible source model while preserving past information. The KF algorithm is employed to denoise the target model parameters, combining them with the source model. First, we outline the CMF that combines the optimization process with KF-based inference (Section 3.1). We then delve into parameterization of linear Gaussian models to thwart catastrophic forgetting (Section 3.2). Subsequently, the KF algorithm for our linear Gaussian modeling explained (Section 3.3). simplified version of the inference process is then presented to reduce the KF parameter count, ensuring the feasibility of the CMF (Section 3.4).
3.1 OVERALL PROCESS
The adaptation strategy alternates between SGD-based optimization and KF-based inference. A subset of the source model parameter ((0) IRd, where is the parameter dimension, designated for adaptation following existing studies. At the outset, the parameter of the hidden model (ft)) Rd
initialized to ((0), The optimization initially minimizes the target generalization error using SGD-optimizer on real-time target data Di++ resulting in the observation ((++1) as follows:
((++1) arg min (0(1), (0(0),(((+)) D. ((+)
The smooth-target model parameter ((t) Rdd which is initialized ((0) is derived recursively. Subsequently, the inference process computes the posterior distribution p()(1))00 (())). The posterior mean is then combined with the target model to obtain g()) via the parameter ensemble

---

Published as a conference paper at ICLR 2024
method. Algorithm [ displays step-by-step description of our framework. Relying solely on the optimization process results the model being recursively adapted a self-training manner. Con- sequently, the target model parameters can become noisy, thereby increasing the risk of catastrophic forgetting. The inference process mitigates this issue by denoising these parameters, updating the hidden model, and re-optimizing with the smooth-target model parameter. However, a general lin- ear Gaussian model for the inference process faces the risk of forgetting the source model during adaptation. The subsequent section introduces countermeasures.
3.2 PARAMETERIZATION
The LG-SSM for the inference process comprises parameterized transition and emission models. We assume the distributions that (g(()))((()) and p( (o()) are both Gaussian distribution. Let the latter as the conjugate prior to the mean of former. We can compute the posterior of o(t) through the Bayesian inference. The mean of this posterior is a convex combination (' and d (See Appendix A.I for further detail). Therefore, the transition model is directly parameterized as follows:
p( o(0) (( + (1- ()) Q),
(5)
where A E Raxd and E Rdxd are assumed to be lime-indppendet. This choice stabilizes the inference process (Murphy 2023). For the emission model, we choose a simple linear Gaussian model follows:
p(0())000)) = (( (1)
(6)
where H E Rdxd and R E IRdd are static, as in the transition model. The source-conjugated transition model aids in persistently infusing source information into the hidden model, thereby ensuring the retention of prior knowledge.
3.3 INFERENCE PROCESS
We now introduce the KF algorithm tailored for the parameterized LG-SSM. The purpose of the algorithm is to recursively determine the posterior distribution p(ot()) (0(1:)) ((0) when the previous is given. By using the posterior from the previous step and Eq. (5), the joint predictive distribution of di(t) and (s(t-1)
step's posterior p(((t=1))@((tt)) (((0) -N(ttt 11--f 1/-1|/t11]
computed and subsequently marginalized over (t(+1)) The one-step-ahead predictive distribution for the hidden variables is then given by
10 ((0)) = =N(6 /46/0/ 11j-11
M - A + (1 (A)
(8)
- +QQ
(9)
This phase, called the predict step, estimates the conditional distribution of (+) based on the past target model parameters (See Appendix A.. for details). Next, given target model information of the current time step, the conditional distribution of ((()) is obtained from the joint distribution of (i(t)) and g by using Eq. (6) and Eq. (7). This phase, called the update step, updates the posterior
distribution (detailed in Appendix A.3) The posterior distribution is obtained as follows:
p( (0) (1(1::))) (00) = =N(6l9) Maje Egje),
(10)
K, H H (HE
R)-1
(11)
M4 MM + (0(1)) Hptp--))
(12)
Enjt = Eepr- K,
(13)
The Kalman gain K, determines the extent to which the target model parameters are updated from the hidden model during the prediction step. For the optimization process of next time step, Ø(t) is derived through the parameter ensemble method of Male and ((() as follows:
ō(() = (1-1))) r'
(14)
where the ensemble hyperparameter I' Bdd is specified be between and This phase allows for the transfer of information from the refined source model to the target model.
We denote the mean and covariance of the posterior distribution by Paje and Enjo and the mean and covariance of the one-step-ahead predictive distribution by Myp-1 and E E-p-1--
5

---

Published as a conference paper at ICLR 2024
3.4 SIMPLIFIED INFERENCE PROCESS
The KF algorithm requires a high computational cost because the parameter dimension of LG-SSM (A, Q, H. R. r) is de where the parameter dimension d of DNNs is typically large. We use scalar parameters for the LG-SSM to reduce the computational complexity, denoted as (a, 7). In particular, we opt 1) for the scalar value 1, and then Eqs. (8), (12), and (14) are can be simplified as follows:
P Moments(4t-1lt-- (1)))
(15)
Mzj = (4 ((0) Br),
(16)
((t) Moments Mele: (1((:)7) 7),
(17)
where Mo = azı + (1 22
-||r-1 a a²²5, 1 4:
(18)
3, = r/ (E +r):
(19)
Eeje = B
(20)
With these simplifications, the inference process becomes computationally efficient. Also, the im- plementation can be streamlined, because it primarily involves repeated applications of Moments
(See Appendix AA.4] for futher details). Next, we detail the rationale behind our choices of each KF parameter. 1) The initialization of (ō("), Pojo, Enjo is (Ol0) ((0), 0). 2) We opt for the same values for 0 v a < and S > < 1 to equalize the degree to which the source and refined source model information are obtained. 3) The properties of the simplified process are predominantly changed by 3, because it is 1 K2. To determine this value, we consider the constraint q + r 1 for variances < q and < r. to satisfy 3. I 1. As q increases, the information in the target model is more reflected in the refined source model, and conversely, as q decreases, the strength of the denoising decreases; in specific case where q = O, our framework is simplified to the parameter ensem- ble method. In summary, only two hyperparameters (ax, q) require modification in the proposed framework.
EXPERIMENTS
We conducted all the experiments on four random seeds using PyTorch (Paszke et al. 2019 toolkits for image classification (Marsden & Döbler 2022) and speech recognition (Ott et al. 2019). The additional implementation and scenario details Is are provided Appendix
Datasets and Metric We relied on multiple datasets to comprehensively evaluate the domain shifts, including both corruptions and natural shifts. For image classification, guided by the benchmark of Marsden et al. (2023), , we selected ImageNet-C (Hendrycks & Dietterich, 2019a). ImageNet-D10% (D109) (Marsden et al.] 2023), ImageNet-R (Rendition) (Hendrycks et al., (02T), and ImageNet- Sketch (Sketch) /Wang et al.] 2019). ImageNet-C comprises 15 corruptions applied to ImageNet (Deng et al. 2009) validation and test images across five severity levels. D109, rooted in Domain- Net (Peng et al. 2019), includes images corresponding to 109 overlapping classes with ImageNet, showcasing six domain shifts (Rusak et af.) 2022; Marsden et al., 2023). Rendition features 30,000 images demonstrates various renderings across 200 ImageNet classes. Sketch consists of 50 sketches for each of the 1,000 ImageNet classes. For speech recognition, we used LibriSpeech (Panayotov al. 2013). TEDLIUMB (TED) (Hernandez et al. 2018), and Common Voice (CV) Ardila et al. (2019) .. LibriSpeech, which includes audio recordings of speakers reading excerpts from Project Gutenberg e-books, serves as source data. TED offers a test dataset of 0.24 hours, representing professional corpus of topical lectures. CV, crowdsourcing project, comprises approximately 25 of utterances recorded by volunteers reading Wikipedia sentences. We used the average error rate for image classification and the word error rate (WER) for speech recognition for performance evaluation.
Scenarios The primary goal was to evaluate the performance of the universal OTTA methods across various scenarios. Guided by previous benchmarks (Gong et al. 2022 Niu et al. 2022 Marsden et al. 2023; Marsden & Döbler 2022). our scenarios delved into covariate shifts (where the input distribution changes) and label shifts (where the label distribution undergoes alterations). We initi- ated the testing of data from multiple domains that underwent covariate shifts. This setting involved

---

Published as a conference paper at ICLR 2024
Table 1:
Average error rates (%) and their corresponding standard deviations in the scenario of CS. Red fonts indicate performance degradation.
ImageNet-C
D109
Method
ResNet-50
VIT
Swin
D2V
ResNet-50
VIT
Swin
D2V
Source
$2.0
60.2
64.0
51.8
58.8
53.6
51.4
48,0
TENT
85.7+0.95
55.1+0.08
62.640.18
50.5=0.06
55.420..8
76.840.36
61.540.41
57.9=0.42
82.0s0.08
59.640002
63.9s0.01
51,2+0.02
55.340.04
53.340.04
51.220.03
47.8:0.01
ReTTA
79.5+0.10
58.740.04
62.940.03
51,3=0.03
5488+0.04
50,960.05
48 48.6+0005
46.8=0.03
SAR
79.6:10.68
52.360.12
60.5a1.04
50.7:0.07
53 63.6600.77
61.250.36
53 53.9:00.8
48.1:0.08
EATA
72.5+1.44
51 51.840.14
56.2+0:29
76.2+20.21
53.1+0.00
48.5+0.11
4M 48.8-1.12
46.2+0.05
ROID 69.520.13
50.710.08
55.020.26
47,4:0.08
50.920.04
46.910.02
17.220.07
45.020.01
CMF (ours)
67.6a10.20
49.0sa.10
52.1n00.1
45.7a40.33
49.4m19.21
44.5s0.08
44.8a00.04
42.8s00.05
Table Average error rates (%) and their corresponding standard deviations in the scenario of TC- CS. Red fonts indicate performance degradation with respect to Source.
ImtgeNet-CC
D109
Rendition
Sketch
Methad
Swin
DDV
VIT
DIV
Swin
D2V
WI
DDV
Sauma:
60.2
64.0
$1.8
$1.6
51.4
48.0
56.0
54.2
46.6
20.6
68.4
60.4
TENT
54.5+0.04
64.0-0.14
51 51.0-0.00
83. 53.3-0.13
66.4-0.33 33
62. 62.9+0.21
53. 8...00.9
53.8+0-38
46.0-0.0
70:8-1.12
68.7-0.23
40 3-0.00
G-TTA
60.4:0:00
64.2-0.01
51.7:0.00
53.3e0.00
51.2.0.01
47.8.0.02
53.6e00.33
54. 1=0.02
46 46-4+1101
70.600.00
68. 38.3=0.02
60 60.3+0..1
RoTTA
59.1it100
63.420.01
51.3-0.01
51.420.08
49.1-0.03
47.250.03
54.8a0.04
53.55005
46.5a11:02
69.3:51.03
67.3-0.03
60.1:10.3
SAR
51.7ell344
65.921.27
51.0a0.12
57.328041
53.5e1.05
48.520.10 10
48.5a0.21
53.72278
459211005
70.5:1.21
73.c11.1
60.2640.07
EATA
4919211.06
$2.920.25
64.4215344
47.2200.10
47.A20:18
45.820006
49.0200.0
29.920.35
450021108
59.220.1
00.620.26
78.3217.00
ROID
45.020.00
47.020.26
44.3:0.01
45.020.04
45.120.10
44.220.05
44.260.13
46.020.10 1O
41.880.11
58.220.4
58.9-0.11
56 26.240.05
CMP (ours)
43.8.0.12
46.429.12
43.520.04
43.429.07
43.620.12
42.369.11
42.750.20
44.110.34
400000066
57.029.08
56.729.13
53.920.00
the analysis of both temporally uncorrelated (CS) and temporally-correlae shifts (TC-CS) during streaming. In CS, test data from all domains are randomly shuffled before adaptation, whereas in TC-CS, consecutive test samples are likely to arise from the same domain. Further, we delved into scenarios where temporal correlated label shifts (TC-LS) occurred over CS or TC-CS by adjusting the Dirichlet distribution's 5 (Gong et al. 2022). As ô increases, the label distribution at time step / begins to approach uniformity; a reduced e sigmifies skewed distribution towards specific classes. These scenarios are illustrated in Fig. 3 In all vision scenarios for ImageNet-C. the domains com- prised 15 corruptions, each encountered at the highest severity level of 5. For D109, the domains comprised five types (clipart, infograph, painting, real and sketch). Our last scenario aimed to test the OTTA methods for speech recognition over real-world streaming data.
Source Model and Baseline For image classification, we employed various of models including ResNet-50 (He et al. (0166a), VisionTrransfrmer (ViT) (Dosovitskiy et al. (2200), SwinTransformrr (Swin) (Liu et al.) (021), and data2vec-vision (D2V) (Baevski et al., 2022b). which were all pre- trained on ImageNet. These modes were suited for wild world (Niu et al. (203)). The Base models for ViT. Swin, and D2V were chosen to balance complexity and performance. For speech recogni- tion, the choices were data2vec Base model (D2V-Libri) and the data2vec Large model (D2V-Vox) The former is pre-trained on a LibriSpeech, while the latter is pre-trained on the entire Vox (Kahn et al., 2020). We compared our approach with other OTTA methods that use arbitrary off-the- shelf pre-trained models. For images, we compared TENT, LAME (Boudiaf et al., 2022), COTTA (Wang et al., 2022), RoTTA (Yuan et al. 2023), EATA, and SAR. For speech, compared our framework with SUTA (Lin et al. 2022). which is state-of-the test-time adaptation method for speech recognition tasks.
Implementation Details We offer implementations for both image classification and speech recog- nition tasks. Image Classification: Our approach largely adheres to the hyperparameter guidelines put forth in Marsden & Döbler (2022) to ensure robustness and reproducibility. We designated the parameter subset (1U) as the normalization layer of each model, inspired by Niu et al. (023). Our chosen loss function is the iiversity-weighhee
soft likelihood ratio loss (DW-SLR) from Marsden et al. (2023), augmented with a consistency loss rooted in symmetric cross entropy (SCE). Unless specified otherwise, the hyperparameters (a, q) for CMF were set to (0.99, 0.005). We employ prior correction for post-processing, as detailed in (Royer & Lampert 2015 Marsden et al. 2023). Speech Recognition: Our approach leverages the Bitfit method from Zaken et al. (2021), which exclusively amends the self-attention bias of the transformer for parameter subset. This methodology was selected because the correction of the normalization layer does not considerably alter performance as noted in (Lin et al. 2202]. For the CMF, the hyperparameters (a. are fixed at (0.8, 0.005). The post-processing step involves greedy decoding, as prescribed in (Pratap et al. 2019). and is devoid of any language model.
7

---

Published as a conference paper at ICLR 2024
Table Average error rates (%) and their corresponding standard deviations in the scenario of TC- LS over TC-CS.
ImageNet-CC
D109
8
Model
LAME
SAR
EATA
ROID
CMF (ours)
LAME
SAR
EATA
ROID
CMF (ours)
VIT
44.120.02
18, 18.24
71.8 22
16.240.06
15.920.04
35.240.55
58, 18.5.00.0
58.611 .4
31/140.07
31.020.10
0.0
Swin
47.120.09
60.110.74
72.740.67
18.1:0.03
16.7a0.10
30.120.16
55.420.17
54.210.99
30.3s0.25
29.620.21
D2V
38.9=0.07
48.340.15
58.242.21
17.4a0.21
14.4:0.24
29.7a0.11
49.5a0.04
46.120.37
29.3s0.03
27.8.0.12
VIT
83.220.23
48.740.29
47.740.12
36.340.08
35.0:0.04
44.840.69
58.621..0
50.741 20
32.220.10
31880.10
0.01
Swin
84.720.12
58.110.86
50,010.35
37.2:0.06
35.1:0.16
39.910,77
53.710.55
19.610.41
31.120.11
30.320.24
D2V
79.5s0.20
47.9:00.5
08:1588
35.920.08
32.7a0.04
39.9:0.55
49.120.11
47.1s1
30.720.09
28.620.11
VIT
79. 79.9:0..6
4+0.30
46.1:0.17
41.3=0.005
39.6200.03
68. 68.921
$7.7+13.55 56
A:1 16
37. 37.3 12
36.1=0.11
0.1
Swin
84.520.09
58.120.75
48.340.09
42.140.04
39.6=0.02
64.610.25
53.420.70
47.410.21
36.9g0.11
35.020.05
70.120.04
18.010.:4
65.5119.11
41.320.03
38.2:0000
61.6210.25
45.710.08
36.320.06
34.120.13
VIT 80.0g0.03
48 3:
45.7.0.15
41.220.033
39, 42..00
90.010.09
57. 12
47.210.04
42 42.920.03
41.3200.66
10
Swin 84.620.06
58.540.41
47.400.39
41.9:0.03
39.4=0.11
86.920.24
54.5a0.65 68
47.410.10
43.020.06
41.3=0.04
D2V
70.220.07
17.910.09
57.0118/44
41.220.01
38.120.03
88.310.13
18.5100.99
15.710.04
42.220.04
40.120.10
VIT 80.220.09
55.5112.62
45.610.17 17
41.320.03
39.5200.0
93.3.0.17
57 17.310..2
47.210.08
43,920.09
42.5200.08
5.0
Swin
04.4
59.2:0.68
47.610.25
41.9200.33
39.4
00.6210.23
54.0:0:72
47.3:0.05
44.120.06
42.5xt0.07
D2V
70.520.12
47.920.08
65.9x18.92
41.2+0.03
38.0=0.05
92.840.16
48.420.12
45.740.06
43.2s0.04
41.120.06
4.1
RESULTS
Covariate shifts Table lists the average eπтоr rates for each OTTA method in the scenario of CS. TENT exhibited a performance dip compared to the source model for both ImageNet-C and D109. SAR in D109 and EATA in ImageNet-C also exhibited degraded performance. RoTTA, CoTTA, and ROID, on the other hand, demonstrated relatively consistent results, with ROID standing out as the top performer. Among all the methods, our CMF framework exhibited the lowest average error rate. In Table 2 where the error rates under the TC-CS conditions are detailed, TENT exhibited substantial decline in performance. CoTTA and RoTTA, which were previously robust, also dis- played performance setbacks. EATA held steady performance in all except for D2V, yet it could not surpass ROID's performance. CMF continued its trend by recording the lowest average error rate, even outperforming ROID.
Covariate and Label shifts Table B presents the average error rates for each OTTA method in the context of TC-LS superimposed on TC-CS, in- dicating that the labels are becoming temporally correlated. In instances with the strongest temporal correla- tion (i.e.. 5 = 0.0), all methods bar- ring LAME and ROID revealed in- consistent outcomes with performance degradation. The latter exhibited superior results between LAME and ROID, except for Swin. As the temporal correlation diminished, the performance of LAME decreased noticeably, but ROID showed robust performance. The CMF still managed to cclipse ROID and LAME. hiiheggisghlights TC-LS under CS following Marsden & Döbler (2022) for ImageNet-C and D109 with ò values of 0.01 and 0.1, respectively. CMF maintained its superi- ority across all datasets and models. Thus, the CMF exhibited unwavering robustness, even in the TC-LS scenarios.
Table 4: Average error rates (%) and their corresponding standard deviations in the scenario of TC-LS over CS.
ImageNet-C
D109
Method
VIT
Swin
D2V
VIT
Swin
D2V
LAME
36.1:0.09
37.420.12 12
36.320
29.920.18
28.6220:33
29.120.19
SAR
51.110.40
65.410.33
47,210.08
51.010.51
53.610.21
48.610035
EATA
70.510.67
17.1.0.9
85.8118.90 90
$2.912.96
50.310.22
45.9100.3
ROID
23.66:005
28.620.16
18.850.01
29.120.09
28.250.05
26.350.07
CMF (ours)
23.2:0.05
27.1:0.08
17.150.09
28.710.19
27.350.05
24.910.10
Distribution shifts on real-world streaming data Table presents the OTTA outcomes for the real-world streaming datasets. Both TED and CV deviate from the source domain, LibriSpeech, in terms of the record- ing environment and vocabulary do- mains. This results in the simulta- neous presence of covariate and label
Table 5: Average WERs (%) and their corresponding stan- dard deviations in real-world streaming scenario.
TED
CV
Method
Livi
D2V-VOX
Libr
D2X-VOX
Source
12.2
8.5
33.4
20.6
SUTA-cont.
67.7+1.70
66.120.36
120.89+4:03
130.3=1.88
SUTA episodic
12.0A0.03
8.040.03
30.340.01
18.910.01
CMF (ours)
11.8+0.05
7.940.02
29.6:4.02
18.740.03
shifts. The leading OTTA method for speech recognition tasks, SUTA, employs test-time adaptation for individual utterances using an episodic strategy (SUTA-cpisodie). However, sharp decrease in performance was observed when this strategy was applied in a continual setting (SUTA-contt). In these continuous settings, the CMF not only thwarted catastrophic forgetting but also displayed
8

---

Published as a conference paper at ICLR 2024
enhanced performance compared to SUTA-episodie. Our framework showed robustness against the intricate distribution shifts found in real-world contexts.
4.2
ABLATION STUDY
We now delve into the essential components of the CMF. Detailed discussions, such as diversity and computational cost, can be found in Appendix C
Effectiveness of leveraging the noisy target model Table 6 lists the average error rates of the CMP CS and TC-CS scenarios for both ImageNet-C and D109 datasets. We exam- ined the performance variations with respect to the value of q. which modulates the infor- mation from the noisy target model. Using the entropy loss of TENT, the CMF demon- strated superior average performance compared to TENT alone. However, this performance gain was only evident for D109, which suf- fered markedly from catastrophic forgetting, as
Table 6: Average error rates (%) for q 0.000 and 0.005 in the CS and TC-CS scenarios.
Mathod
ImagNas-C
AVG
CS
TC-C8
CS
TC-CS
TENT
Entropy
55.1
83.3
CMP
Entropy
0.000
58.6
57.8
53.6
51.5
56.0
0.055
57.1
56.01
57.3
60.9
57.8
DW-SLR
0.000
52.7
47.8
47.1
46.11
48.4
0.005
51.3
47.1
45.3
44.4
47.0
SCE
0.000
50.7
45.0
45.0
46.9
0.005
48.9
43.4
45.4
TENT
Entropy
CMF
Entropy
0.000
64.4
64.2
51.5
51.6
57.9
Swim
0.005
64.3
64.11
51.6
52.5
58.1
shown in Tables [ and 2 When comparing the outcomes for q = 0.000 and q = 0.005 on ImageNet-C, our model recorded a higher average error rate without utilizing the target model (i.e., q 0.000), trend reversed for D109. This highlights the limitation of relying solely on the entropy loss (Nevertheless, the en- tropy version of CMF significantly outperforms TENT when adjusting for diversity, which is detailed in Appendix C.2). This inconsistency was addressed by incorporating DW-SLR (Mars- den et al. 2023), an entropy-based loss function that consider diversity. For both datasets, the CMF coupled with DW-SLR exhibited superior performance improvements when 0.005 compared to = 0.000. Similarly, SCE loss, which modulated diversity via augmentation, further enhanced performance. These results were consistent with the findings that a convex combination of param- eters (i.e., Moments) from out-of-domain can escalate the generalization error if the diversity is not moderate (Wortsman al., 2022 Rame al. 2022). In essence, the CMF consistently boosts performance when diversity is appropriately managed.
DW-SLR
0.000
57.5
51.1
49.4
46.3
51.1
0.005
57.5
48.0
45.9
44.5
49.0
SCE
0.000
55.0
42.0
48.6
0.005
52.1
46.8
TENT Entropy
50.5
CMF
Entropy
0.000
52.3
$2.9
48.1
48.5
50.4
D2V
0.005
51.8
52.7
48.9
50.6
51.0
DW-SLR
0.000
49.0
45.9
46.9
45.4
46.8
0.005
67.2
45.4
43.7
43.3
46.9
SCE
0.000
47.4
44.2
45.4
0.005
42.3
Effectiveness of the source-conjugated transition model FigureZi shows the average error rates for each model con- tingent on variations in a, which determine the sway of the source model in the source-conjugated transition model. A comparison of the error rates for a = 1 (devoid of source model influence) and a = 0.99 (factoring in the source model influence) revealed enhanced performance across all models. However, as increased, the performance metrics for both Swin and D2V models exhibited a downward tra- jectory. These findings underscore the significance of the moderated influence of the source model.
13 VIT Swin D2V
02.
Averaage Error Rate (56)
47
495
48.
45
88
Figure 2: Average error rates (%) as decreasing where ? 0.99 in the scenarios of TC-CS on ImageNet-C.
5 CONCLUSION
We proposed the CMF, which novel approach to bolster the OTTA methodology. This was achieved by deducing a refined source model through target model denoising by leveraging the KF. By stream- lining the KF algorithm, the computational overheads were minimized, underpinning the pragmatic nature of the CMF. Our strategy withstood a rigorous evaluation across a spectrum of scenarios previously subjected to state-of-the-an methods, consistently demonstrating marked performance advancements. Its efficacy in real-time streaming contexts, such as speech recognition tasks, sug- gests its potential as integral facet of established frameworks across diverse applications.

---

Published as a conference paper at ICLR 2024
REPRODUCIBILITY STATEMENT
In this paper, we conducted experiments based on the official GitHub code of the toolkits men- tioned in Section for image classification and speech recognition. Appendix B lists the download URLs each toolkit, the adopted datasets, and the pre-trained source models and mentions more experimental details. The code is available at uttttps://gittnni com/ wow.:oooog/CCee
ACKNOWLEDGEMENTS
This work was supported by the National Research Foundation of Korea(NRF) grant funded by the Korea govermmen(MMSTT) (No. RS-2023-00302424) This work was supported by Institute of formation & communications Technology Planning & Evaluation (IITP) grant funded by the Korea govenieen(MMSIT) (No.2021-0-00433 Development Ultra-high Speech Quality Technology for Remote Multi-speaker Conference System).
REFERENCES
Rosana Ardila, Megan Branson, Kelly Davis, Michael Henretty, Michael Kohler, Josh Meyer, Reuben Morais, Lindsay Saunders, Francis Tyers, and Gregor Weber. Common voice: A speech corpus. arXiv preprint arXiv:1912.066700 2019.
Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E Hinton. Layer normalization. arXiv preprint 167.006007. 2016.
Alexei Baevski, Steffen Sehneider, and Michael Auli. vq-wav2vee: Self-supervised learning of discrete speech representations. arXiv preprint arXiv:1910.05453, 2019.
Alexei Baevski, Yuhao Zhou, Abdelrahman Mohamed, and Michael Auli, wav2vec 2.0: A frame- work for self-supervised learning of speech representations. In Proc. Advances in Neural Infor- mation Processing Systems (NeurIPS), pp. 12449-12460, 2020.
Alexei Baevski, Arun Babu, Wei-Ning Hsu, and Michael Auli. Efficient self-supervised learn- ing with contextualized target representations for vision, speech and language. arXiv preprint arXiv:2212.07525. 2022a.
Alexei Baevski, Wei-Ning Hsu, Qiantong Xu, Arun Babu, Jiatao Gu, and Michael Auli. data2vec: A general framework for self-supervised learning in speech, vision and language. In Proc. Inter- national Conference on Machine Learning (ICML), pp. 1298-1312, 2022b.
David Berthelot, Nicholas Carlini, Ian Goodfellow, Nicolas Papernot, Avital Oliver, and Colin A Raffel. Mixmatch: A holistic approach to semi-supervised learning. Advances in neural informa- tion processing systems, 32, 2019.
Malik Boudiaf, Romain Mueller, Ismail Ben Ayed, and Luca Bertinetto. Parameter- free online test- time adaptation. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pp.
2022.
Dian Chen, Dequan Wang, Trevor Darrell, and Sayna Ebrahimi. Contrastive test-time adaptation. 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 295-20 995-305, 2022. URL lattps::/
semant westtciisshh.... wttp://ooooo100002036600000
Sanyuan Chen et al. Waylm: Large-scale self-supervised pre-training for full stack speech process- ing. IEEE Journal of Selected Topics in Signal Processing (J-STSP), 2021.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hi- erarchical image database. In 2009 IEEE conference computer vision and pattern recognition, pp. 248-255. leee, 2009.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010.1 1929, 2020.
10

---

Published as a conference paper at ICLR 2024
Timur Garipov, Pavel Izmailov, Dmitrii Podoprikhin, Dmitry P Vetrov, and Andrew G Wilson. Loss surfaces, mode connectivity, and fast ensembling of dnns. Advances in neural information pro- cessing systems, 31, 2018.
Taesik Gong, Jongheon Jeong, Taewon Kim, Yewon Kim, Jinwoo Shin, and Sung-Ju Lee. Note: Ro- bust continual dest-time adaptation against temporal correlation. In Neural Information Processing Systems, 2022. _ https: /tapss///pp.. /api. semant: cscholas www.aarrrrgg.oo.eeeoeeeeeeeee wwwp/oorgg//ooooooooooo :528::::2222400
lan Goodfellow, Yoshua Bengio, Aaron Courville, and Yoshua Bengio. Deep learning, volume 1. MIT Press, 2016.
Alex Graves, Abdel rahman Mohamed, and Geoffrey E. Hinton. Speech recognition with deep recurrent neural networks. In Proc. IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 6645-6649, 2013.
Hao Guo, Jiyong Jin, and Bin Liu. Stochastic weight averaging revisited. Applied Sciences, 13(5): 2935, 2023.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 770-778, 201
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In Proceedings of the IEEE conference computer vision and pailern recognition, pp- 770-778, 2016b.
Dan Hendrycks and Thomas Dietterich. Benchmarking neural network robustness to common ruptions and perturbations. Proceedings of the International Conference on Learning Represen- tations, 2019a.
Dan Hendryeks and Thomas Dietterich. Benchmarking neural network robustness to common cor- ruptions and perturbations. arXii preprint arXiv: /903. 12261, 2019b.
Dan Hendrycks, Steven Basart, Norman Mu, Saurav Kadavath, Frank Wang, Evan Dorundo, Rahul Desai, Tyler Zhu, Samyak Parajuli, Mike Guo, Dawn Song. Jacob Steinhardt, and Justin Gilmer. The many faces of robustness: A critical analysis of out-of-distribution generalization. ICCV, 2021.
François Hernandez, Vincent Nguyen, Sahar Ghannay, Natalia Tomashenko, and Yannick Esteve. Ted-lium 3: Twice much data and corpus repartition for experiments speaker adaptation. In Speech and Computer: 20th International Conference, SPECOM 2018, Leipzig, Germany, September 18-22, 2018, Proceedings 20, pp. 198-208 Springer, 2018.
Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the knowledge neural network. arXiv preprint 1503 02..0333..,,,,.. 2015.
Geoffrey E. Hinton et al. Deep neural networks for acoustic modeling in speech recognition: The shared views of four research groups. IEEE Signal Processing Magazine, 29:82-99 2012.
Junyuan Hong, Lingjuan Lyu, Jiayu Zhou, and Michael Spranger. Mecta: Memory-economic con- tinual test-time model adaptation. In The Eleventh International Conference on Learning Repre- sentations, 2022.
Wei-Ning Hsu, Benjamin Bolte, Yao-Hung Hubert Tsai, Kushal Lakhotia, Ruslan Salakhutdinov, and Abdelrahman Mohamed. Hubert: Self-supervised speech representation learning by masked prediction of hidden units. IEEE/ACM Transactions on Audio, Speech, and Language Processing (TASLP), 2021.
Sergey Ioffe and Christian Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. In International conference on machine learning, pp. 448-456 pmlr, 2015.
11

---

Published as a conference paper at ICLR 2024
Pavel Izmailov, Dmitrii Podoprikhin, Timur Garipov, Dmitry Vetrov, and Andrew Gordon Wil- son. Averaging weights leads to wider optima and better generalization. arXiv preprint arXiv:1803.05407,22000 2018.
Frederick Jelinek. Statistical methods for speech recognition. 1997.
Jacob Kahn, Morgane Rivière, Weiyi Zheng, Evgeny Kharitonov, Qiantong Xu, Barre-Emmanuell Mazaré, Julien Karadayi, Vitaliy Liptchinsky, Ronan Collobert, Christian Fuegen, et al. Libri- light: A benchmark for with limited or no supervision. In ICASSP 2020-2020 IEEE Interna- fional Conference on Acoustics, Speech and Signal Processing (ICASSP), pp- 7669-7673, IEEE, 2020.
James Kirkpatrick, Razvan Pascanu, Neil Rabinowitz, Joel Veness, Guillaume Desjardins, Andrei A Rusu, Kieran Milan, John Quan, Tiago Ramalho, Agnieszka Grabska- -arnskkakaaiikkaa, al. Overcom- ing catastrophic forgetting in neural networks. Proceedings of the national academy of sciences, 114(13):321133526 2017.
Pang Wei Koh, Shiori Sagawa, Henrik Marklund, Sang Michael Xie, Marvin Zhang, Akshay Bal- subramani, Weihua Hu, Michihiro Yasunaga, Richard Lanas Phillips, Irena Gao, et al. Wilds: A benchmark of in-the-win whil distribution shifts. In International Conference on Machine Learning, pp. 5637-5664. PMLR, 2021.
Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. Imagenet classification with deep convo- lutional neural networks. Advances in neural information processing systems, 25, 2012.
Yanghao Li, Naiyan Wang, Jianping Shi, Xiaodi Hou, and Jiaying Liu. Adaptive batch normalization for practical domain adaptation. Pattern Recognition, 80: 60:109-117, 117, 2018.
Guan-Ting Lin, Shang- Wen Li, and Hung yi Lee. Listen, adapt, better wer: Source-free single- utterance test-time adaptation for automatic speech recognition. In Proc. Interspeech, 2022.
Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer: Hierarchical vision transformer using shifted windows. In Proceedings of the IEEE/CVF international conference computer vision, pp. 10012-10022, 2021.
Massimiliano Mancini, Hakan Karaoguz, Elisa Ricci, Patric Jensfelt, and Barbara Caputo. Kitting in the wild through online domain adaptation. In 2018 IEEE/RSJ International Conference Intelligent Robots and Systems (IROS), pp. 1103-1109. IEEE, 2018
Robert A. Marsden and Mario Döbler. test-time-adaptation. http://githa..commmarrdceeeeeeee time-adaptation, 2022.
Robert A Marsden, Mario Döbler, and Bin Yang. Universal test-time adaptation through weight ensembling, diversity weighting, and prior correction. arXiv preprint arXiv:2306.0000 00650000000000 2023.
Kevin P Murphy. Probabilistic machine learning: Advanced topics. MIT press, 2023.
Shuaicheng Niu, Jiaxiang Wu, Yifan Zhang, Yaofo Chen, Shijian Zheng, Peilin Zhao, and Mingkui Tan. Efficient test-time model adaptation without forgetting. In International conference machine learning, pp. 16888-16905. PMLR, 2022.
Shuaicheng Niu, Jiaxiang Wu, Yifan Zhang, Zhiquan Wen, Yaofo Chen, Peilin Zhao, and Mingkui Tan. Towards stable test-time adaptation in dynamic wild world. ArXiv, abs/2302. 12400, 2023.
httlh:ts///// //api www.ntieschooioooee.e.eee 2ttpaaarrr.orrrcpps10:2222222000
Myle Ott, Sergey Edunov, Alexei Baevski, Angela Fan, Sam Gross, Nathan Ng, David Grangier, and Michael Auli. fairseq: A fast, extensible toolkit for sequence modeling. arXiv preprint arXiv:1904.01033 2019.
Vassil Panayotov, Guoguo Chen, Daniel Povey, and Sanjeev Khudanpur. Librispeech: an corpus based on public domain audio books. In 2015 IEEE international conference on acoustics, speech and signal processing (ICASSP), pp. 2206-5210. IEEE, 2015.
12

---

Published as a conference paper at ICLR 2024
Adam Paszke, Sam Gross, Francisco Massa, Adam Lever, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. Pytorch: An imperative style, high- performance deep learning library. Advances in neural information processing systems, 32, 2019.
Xingchao Peng, Qinxun Bai, Xide Xia, Zijun Huang, Kate Saenko, and Bo Wang. Moment matching for multi-source domain adaptation. In Proceedings of the IEEE International Conference on Computer Vision, pp. 1406-1415, 2019.
Vineel Pratap, Awni Hannun, Qiantong Xu, Jeff Cai, Jacob Kahn, Gabriel Synnaeve, Vitaliy Liptchinsky, and Ronan Collobert. Wav2letter++: A fast open-source speech recognition sys- tem. In ICASSP 2019-2019 IEEE International Conference on Acousties, Speech and Signal Processing (ICASSP), pp. 1460-6466 IEEE, 2019.
Joaquin Chinooero-Candelle Masashi Sugiyama, Anton Schwaighofer, and Neil D Lawrence. Dataset shift in machine learning. Mit Press, 2008.
Alexandre Rame, Matthieu Kirchmeyer, Thibaud Rahier, Alain Rakotomamonjy, Patrick Gallinari, and Matthicu Cord. Diverse weight averaging for out-of-distribution generalization. Advances in Neural Information Processing Systems, 1582111110086 2022.
Amelie Royer and Christoph H Lampert. Classifier adaptation at prediction time. In Proceedings of the IEEE Conference Computer Vision and Pattemn Recognition, pp. 1401-1409, 2015.
Sebastian Ruder. An overview of gradient descent optimization algorithms. arXiv preprint
1609.0477 2016.
Evgenia Rusak, Steffen Schneider, Peter Vincent Gehler, Oliver Bringmann, Wieland Brendel, and Matthias Bethge. Imagenet-d: A new challenging robustness dataset inspired by domain adapta- tion. In ICML 2022 Shift Happens Workshop, 2022.
Simo Särkkä and Lennart Svensson. Bayesian filtering and smoothing. 2023.
URL https:
//api. www.eteetttt..e www.mmmlnnccccca......m... www.choolaa..rrgCooous....... 22M.D:::621333333
Steffen Schneider, Alexei Baevski, Ronan Collobert, and Michael Auli. wav2vee: Unsupervised pre-training for speech recognition. In Proc. Interspeech, 2019.
Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv: 1409. 1556, 2014.
Kihyuk Sohn, David Berthelot, Nicholas Carlini, Zizhao Zhang, Han Zhang, Colin A Raffel, Ekin Dogus Cubuk, Alexey Kurakin, and Chun-Liang Li. Fixmatch: Simplifying semi-supervised learning with consistency and confidence. Advances in neural information processing systems, 33:596-608. 008, 2020.
Sining Sun, Binbin Zhang, Lei Xie, and Yanning Zhang. An unsupervised deep domain adaptation approach for robust speech recognition. Neurocomputing, 257:79-87, 2017. URL https::/ api wtt.::accccco.ooooooooooe..c..ee
Dequan Wang, Evan Shelhamer, Shaoteng Liu, Bruno Olshausen, and Trevor Darrell. Tent: Fully test-time adaptation by entropy minimization. arXiv preprint arXiv:2006.1077 2020.
Haohan Wang, songwei Ge, Zachary Lipton, and Eric P Xing. Learning robust global representa- tions by penalizing local predictive power. In Advances in Neural Information Processing Sys- tems, pp. 10506-10518 10518, 2019.
Qin Wang, Olga Fink, Lue Van Gool, and Dengxin Dai. Continual test-time domain adaptation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp- 7201-7211, 2022.
Mitchell Wortsman, Gabriel Ilharco, Samir Ya Gadre, Rebecca Roelofs, Raphael Gontijo-Lpees, Ari S Morcos, Hongseok Namkoong, Ali Farhadi, Yair Carmon, Simon Korblith, et al. Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference time. In International Conference on Machine Learning, pp. 23965-23999 PMLR, 2022.
13

---

Published as a conference paper at ICLR 2024
Qizhe Xie, Minh- Thang Luong, Eduard Hovy, and Quoc V Le. Self-training with noisy student improves imagenet classification. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 10687-10698, 2020.
Tao Yang, Shenglong Zhou, Yuwang Wang, Yan Lu,, and Nanning Zheng. Test-time batch normal- ization. arXiv preprint arXiv:2205. 10210, 2022.
Longhui Yuan, Binhui Xie, and Shuangliang Li. Robust test-time adaptation in dynamic scenarios. 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 15922- 15932, 2023, /ttts://pppp owt.aanw...csoooo.....ooooorpppa
Elad Ben Zaken, Shauli Ravfogel, and Yoav Goldberg. Bitfit: Simple parameter--cffcient fine-tuning for transformer-based masked language-models, arXiv preprint arXiv:2106. 2021.
Marvin Zhang, Sergey Levine, and Chelsea Finn. Memo: Test time robustness via adaptation and
augmentation. Advances Neural Information Processing Systems, 35:38629-38642, 2022.
Bowen Zhao, Chen Chen, and Shu-Tao Xia. Delta: degradation-free fully test-time adaptation. arXiv preprint arXiv:2301.130. 2023.
14

---

Published as a conference paper at ICLR 2024
A
DERIVATIONS
For completeness, included derivations for each component of the CMF inference process. These
derivations follow Särkkä & Svensson (2023) and conform to the notations in Murphy (2023). All
parameters (Si, S2, A, Q, H, Rt) in
A.1 SOURCE-CONTUGATED TRANSITION MODEL
Given S₁ and S2 are known constants, assume that the likelihood for ((t-1) is
p(o(t-))((())) (o Si),
(21)
and the conjugate prior the likelihood is
(0)))= - (V600))) S2).
(22)
The conditional distribution ()() obtained using Lemma A.3 from Särkkä Svensson (0023)
psft ,1)) Nos AA |(
(1
A) Q).
(23)
where
A
(24)
Q S,52 + 51)
We directly parameterize the transition model with (A, Q) rather than by using (2).
A.2 PREDICT STEP
The posterior of the previous step is given by
p(olt-1))o((t-))) )l)))(())))))))))--))))))))))))) = = (o p1- 1|t-1: 1(-1)e-1).
(25)
According to Lemma A.2 from Särkkä & Svensson (202)) Eqs. (5) and (25), the joint distribution of (tt)) and dit- given ((1:-1) (6l)) is
(( ((t-1) 11= ((0)))
- -(di
(1)))) (((d
)( )p(bit 10(011-1100H--))
= -N((() + (1 - 1) ()(((((())))) (0((-1) 11 Ec-1je-1)
= N ((0tt)))
((10)
|2',>')
(26)
where
11
1-111-1
Ap + - (I-A)00)
(27)
E AE
E
A )):
Σ'
and the marginal distribution of ((0) is obtained using Lemma A.3 from Särkkä Svensson
2023
(0 , = (N( Eej--),
Meje-11
(28)
where
Mit-1 Ap (1- A)
(29)
E = AE + Q.
15

---

Published as a conference paper at ICLR 2024
A.3 UPDATE STEP
According to Lemma A.2 from Särkkä & Svensson 2023 Eqs. (23) and (28), the joint distribution of ( and g( is
p( 00000011-11 ,(())) - p(o00 11o
,,()))
N(O001HO R)N(("
Majo-11 E(c-1)
(30)
N (NN((()))) (000) 1e", 12')
where
1¹"
14-1/t-1
H14p11
(31)
Eijt-1
E"
HE
H R
and the conditional distribution of s11) is obtained using Lemma A.3 from Särkkä & Svensson
2023
p(010))0(1)) ,6 =N(ss 149ps Eeje);
(32)
where
K; = Ezpe 11 (H2 + R)-
Mele Maje-1 K.(0(t) Hu(((11); Egje Esje-1 KHHEaH--
(33)
A.4 SIMPLIHICATION
We chose (A, Q, H, It) to scalar values (a, 9, 1, - -). Thus, Eq. (29) is
Mdj-1 = 0/4-111-11 + (1 ()
(34)
Eqp-1 = -
and Eq. (33)
8. (1- - ())((2//-1++ q),
Fej = 3-Mjj11 + (1 - -3))
(35)
Egp = 1.E
where Br - KG. Consequently, M4je-1, E+ Meles Eeje and Bg are scalars.
B DETAILS OF IMPLEMENTATION
B. EXPERIMENTS OF IMAGE CLASSIFICATION
We focused the numerous classes, potential corruption, and natural distribution shifts that cur in the wild world (Niu et al. 2023). ImageNet-(lis standard TTA benchmark used to evaluate robustness against corruption. ImageNet contains 1,281,167/50.000 training/testing data respec- tively. ImageNet-C is a dataset applied to ImageNet according to 15 types of damage (gaussian noise, shot noise, impulse noise, defocus blur, glass blur, motion blur, zoom blur, snow, frost, fog, brightness, contrast, elastic transform, pixelate, and jpeg compression) at five severity levels. Each corruption was considered domain, and severity level of has been selected. D109 is a dataset concerning natural distribution shifts provided by Marsden et al. (2023). This dataset consists of five domains (clipart, infograph, painting, real, and sketch), D109 classes were based on DomainNef.
http:///onodo.cr///oo////////2/2 Y2jRo__c_________
nttp://a..00.ee00
16

---

Published as a conference paper at ICLR 2024
including 109 classes that overlapp with ImageNet. For other natural shifts, used the Rendition and Sketch datasets. Each dataset consisted only of domains and was used in the TC-CS scenario. Rendition iineluded 30,000 images with various artistic renderings of 200 ImageNet classes, pri- marily collected from Flickr and filtered by Amazon MTTurk annotators. Sketch dataset consists of 50,000 images, with 50 images for each of the 1,000 ImageNet classes. This dataset is constructed from Google image queries with the standard class name "sketch of"" and is only searched within the "black and white" color scheme.
(a) CS
(c) TC-LS
Adaptation Order
Adaptation Order (6 0.0)
(b) TC-CS
Adaptation Order
Adaptation Order (6 5.0)
Domains:
Label Claases:
Domain
Domain
Domain
Class
Class
Class
Figure 3: Illustration of the OTTA scenarios.
Four scenarios were considered: covariate shifts (CS), temporally covariate shifts (TC- CS), remporally-correaatee label shifts (TC-LS) over CS, and TC-CS over TC-CS. CS involves pre- viously defined domains streamed into the model, which were all mixed up. Hence, the likelihood of temporally adjacent input data being drawn from the same domain is low NNu et al. (2022). Conversely, the TC-CS scenario, each domain was applied sequentially as previously described; therefore, temporally adjacent input data were likely to belong to the same domain. Meanwhile, TC-LS simulates the characteristics of real online data, where labels are temporally interrelated, and data of the same class appears multiple times (Gong al.) 2022). This methodology entails the creation of a non-i.i.d partition for given number of tokens across a set number of classes. The Dirichlet distribution, driven by the concentration parameter 5 0, determines how data from each class were allocated to each token.
We implemented the CMF method based the benchmark in Marsden & Döbler (2022 and used the associated default hyperparameters. For all the vision experiments, the learning rate was set to 0.00025 for Viijand and Swir] 0. 0.0002 for ResNet-5(] and 0.00001 for D2V] Unless specifically men- tioned, we used (a, q) (0.99, 0.005) for CMF. Furthermore, because each model had different number of learnable parameters, we adjusted by multiplying the ratio of each model's number of learnable parameters to that of ViT by q. The SGD optimizer was used for model training in all scenarios. The iteration for the adaptation was set to
vit
Swin
Fesnet50.ntm
17

---

Published as a conference paper at ICLR 2024
B.2 EXPERIMENTS OF SPEECH RECOGNITION
The LibriSpeecifdataseé which is widely used for pre-training in speech recognition, contains audio recordings of speakers reading excerpts from Project Gutenberg e-books. It included 1,000 hours of utteannces with subsets of 100 h, 360 h, and 500 h for training. D2V-Libri was trained both self- supervised and supervised on the 960h subset. Libri-Lighffis a collection of English voice audios suitable for training speech recognition systems with limited or no supervision. It is derived from the open-source audiobooks of the LibriVox project, which contain over 60,000 of audio. D2V-Vox was self-supervised trained on Ligri-Light and supervised trained on LibriSpeech 960h.
We used T and CV as the target data, each representing different scenario. TED provide an official test dataset of lectures covering various topics and areas. This test dataset was divided into 11 datasets per speaker, each which had a different domain. This simulated an actual situation where professionals speak continuously and at length. The average duration of utterances per speaker in the test dataset was 0.24 h. CV is a crowdsourcing project supported by volunteers who read Wikipedia sentences and record samples at 48kHz. The sampling rate is resampled to 16kHz to match the learning conditions of the source model. We used the Common Voice Corpus 5.1 (dated 7/14/2020). Unlike TED, the test dataset was not divided per speaker. This test dataset contained approximately 25 h of utterances, simulating an extremely long utterance environment.
A learning rate of 0.0006 was selected for D2V-Libr[annd D2V-Vox to maintain the CMF approach used in the vision experiments. The Adam optimizer was used for model training in all speech recognition experiments. The number of iteration I for the adaptation followed Lin et al. (2022 using 10. We also applied the masking augmentation (Hsu al. 2021, Chen al. [2021] [Schneider al. 2019) Baevski et al. 2019 2020 (002a)) four times, considering diversity. For fair comparison, both comparison targets, SUTA-cont. and SUTA-episodic, used the same learning rate and and the same augmentation was applied.
We set (ax, q) (0.8, 0.005)
C ADDITIONAL ABLATION STUDY
C.1 EFFICIENCY OF METHODS
Table 7: Comparison of the computation efficiency for ResNet-50 on Rendition in the scenario of TC-CS.
Method
Average Error Rate (%)
#Forwards
#Backwands
Train Param (%)
Source LAMR
63.8
30,000
99.4
30,000
TENT
57.4
30,000
30,000
0.21
CoTTA
57.4
90.000
30,000
100
RoTTA
60.8
90,000
30,000
0.21
SAR
57.2
46,279
30,111
0.12
EATA
54.2
30,000
5,440
0.21
ROID
51.4
48,610
37,220
0.21
CMF
50.7
48,610
37,220
0.21
Table 7 provides overview of the computational efficiency of the different methods, measured by the number of forward and backward passes required during the Table (lbbiide an overview the computational efficiency of the different methods, measured by the number of forward and backward
passes required during the test-time adaptation. CoTTA and RoTTA, owing to their reliance on the
paccher-ttuden modeling approach (Hinton et al. 2015: Xie et al. 2020)] Sohn et al., 20200] Berthelot et al. 2019), these methods require three forward passes: two for the teacher model and one for the
https://ww.coenslr...rrrz
base 1s_ 9.0.960h.pp..
http://aaaaaappplleeeeeeeerrrrrr 960h.pt
SUTA
18

---

Published as a conference paper at ICLR 2024
student model. Correspondingly, backward pass required for the teacher model. EATA selectively filtered the samples during learning. This filtering mechanism ensured the superiority of EATA in efficiency, demanding fewer backward passes than TENT. SAR requires more forward and backward passes than TENT because incorporates a general forward and a forward with filtering. ROID adopts filtering strategy (i.e., DW-SLR loss) akin EATA while incorporating data augmentation driven by SCE loss, resulting in an augmented count of forward and backward passes relative TENT. By leveraging both DW-SLR and SCE losses, the CMF mirrors ROID in terms of efficiency. We also investigate the memory cost for ROID/CMF: each method consumes M for D2V and (0339MiB//106770
for Swin. The CMF requires slightly more memory than ROID because it uses hidden variables. However, it does not require much additional computation because inference process of the CMF does not perform backward, which dominates the memory cost. To summarize, the superior performance of CMF is noteworthy, even with similar computational and memory requirements as ROID.
C.2 DIVERSITY DIAGNOSIS AND PRESCRIPTION
vital factor of the CMF framework the parameter diversity, which gauges how much target models deviate from the source model. A moderate level of diversity often indicates that our framework is likely to operate more effectively. To quantify this diversity, we employed cosine similarity metrics between the source and target models across different domains. The diversity was calculated the inverse of this similarity, specifically as Average Cosine Similarity where Average Cosine Similarity is calculated for domain in the dataset.
ImageNet-C
-D109
0.007
0.006
DIVERSITY
0.005
0.004
0.002
0.001
0.000
4
9
11
12
13
14
15
ADPATION ORDER (->)
(a) Comparison of the diversity on ImageNet-C and D109
CMF
CMF (+SCE)
CMF (LR ++)
0.35
0.006
0023
0.005
0.33
1
0032
0.001
0.00
DAUSSAR
IMPUISE
arrocus
GLAMS
zootal
rad
CONTRANT
PINELATE
ADAPTATION ORDER (-9)
ADAPTATION ORDER (+)]
(b) ImangeNet-C (+SCE)
(c) D109 (LR
Figure 4: Comparison of the diversity for VIT in the scenario of TC-CS.
Figure provided insights into the diversity variation across different data sets and scenarios, ex- plicitly tocusing on ViT model. We also explored how diversity changes when additional SCE loss functions (+SCE) are introduced or when the learning rate decreased (LR J). When the "LR 4"" method was applied, the learning rate was 0.0001 for the ViT/Swin model, down from
19

---

Published as a conference paper at ICLR 2024
initial value of 0.00025, and to 0.000005 for the D2V model, which started at 0.00001. Our analysis revealed that the D109 dataset exhibits significantly higher diversity than the ImageNet-C dataset, shown in Figure (a). When SCE loss applied to the ImageNet-C dataset, which showed lower diversity, an increase in diversity was observed (Figure 4 (b)). Conversely, in the case of the D109 dataset, which presented higher diversity level, reducing the learning rate decreased diversity (Figure
Table 8: Comparison average error rates (%) for each prescription in the scenario of TC-CS.
Average Error Rate (%)
Dataset
Method
Prescription
ViT
Swin
D2V
ImageNet-C
TENT
54.5
64.0
51.9
CMF
56.0
64.0
52.7
+ SCE
53.6
60.1
51.5
D109
TENT
83.3
66.4
62.9
CMF
60.9
52.5
50.6
LR 4
54.1
51.7
48.6
This manipulation of diversity through adjustments, the introduction of SCE loss or the reduction in learning rates, has been shown to positively impact the performance of different models, as evi- denced Table 8 Thus, managing diversity was crucial strategy for optimizing the functionality of the CMF approach. The pain of managing diversity was largely solved by using DW-SLR loss, as discussed in Section 4.2
C.3 LIMITATION OF FIXED SOURCE MODEL
Table 9: Comparison of average error rates (%) for WE and CMF in the scenario of TC-CS.
Method
Model
ImageNet-C
D109
Rendition
Sketch
DW-SLR+SCE
VIT 47.640.06
48.3+0.92
58.4+0.05 44.6+0.25
Swin
51.1#0.30
50.0+0.10
47.0+0.14
59.1
D2V
46.3+0.15
47.4+0.41
42.2+0.13
55.6+0.07
DW-SLR+SCE+WE
VIT
45.0+0.09
45.0+0.04
44.2+0.13
58.6+0.00 04
Swin
47.040.26
45.140.10
46.0+0.10
58. 58.9+0. 11
D2V
44.8+0.01
44.240.06
41.8+0.11
56.2+0.05
DW-SLR+SCE+CMF
ViT 44.8+0.12
43.420.07
42.7+0.20
57.0+0.08
Swin 46.6+0.12
43.640.12
44.1#0.24
56.7+0.13
D2V
43.5+0.04
42.3+0.11
40.0+0.06
53.9+0.03
Conventional weight ensemble (WE) methods utilize fixed source model. In contrast, the CMF approach updates the source model. evaluate the performance of both WE and CMF, we set the loss to DW-SLR SCE and measured their respective performances in TC-CS scenario.
When WE is added, significant performance improvements in datasets such as Sketch or Rendition are not readily observable. Conversely, substituting WE with CMF leads substantial performance enhancements across all datasets, including the two aforementioned, and notably outperforms WE in the ImageNet-C and D109 datasets. These results highlight the limitations of the fixed source model and clearly delineate the contribution of CMF, which continually updates the hidden model with both the target and source model information.
20

---

Published as a conference paper at ICLR 2024
C.4 SENSITIVITY OF HYPERPARAMETERS IN CMF
Table 10: Comparison of average error rates (%) for sensitivity of q in the scenario of TC-CS.
CMF (2)
Model
Source
TENT
0.01
0.0075
0.005
0.0025
0.0001
ViT
60.2
54,5=0.04
44.9+0.09
44.9+0.09
44.8+0.12
44.8+0.14
44,9=0.05
Swin
64.0
64.0=0.14
46.7+0.25
46.5+0.05
46.6+0.12
46.5+0.18
46.920.18
D2V
51.8
51.9=0.09
43.4=0.08
43.4=0.09
43.5+0.04
43.7=0.15
44.5=0.02
Table 11: Comparison of average rates (%) for sensitivity of in the scenario of TC-CS.
CMF (a)
Model
Source
TENT
0.999
0.95
0.9
0.8
VIT
60.2
54,5=0.04
45.2+0.03
44.8+0.12
44.7=0.06
44.9+0.09
44.920.06
Swin
64.0
64.0=0. 14
46.9+0.12
46.6:0.12
46.740.13
46.840.24
47.1=0.20
D2V
51.8
51.9=0.09
43.6+0.17
43.5+0.04
44.2+0.11
44.620.04
44.820.04
Table 12: Comparison of average error rates (%) for sensitivity of y in the scenario of TC-CS.
CMF (7)
Model
Source
TENT
0.999
0.99
0.95
0.9
0.8
ViT
60.2
54.5=0.04
45.2+0.11
44.8+0.12
45.5+0.09
46.7+0.07
49.2=0.00
Swin
64.0
64.0=0.14
47.140.21
46.620.12
47.2+0.
48. 48.5+0.20
50.7=0,28
D2V
51.8
51.9=0.09
43.2+0.22
43.5+0.04
45.3+0.04
46.8+0.06
48.620.11
An analysis was conducted understand the impact of adjusting hyperparameters (q, a, 7) in CMF, as outlined in Algorithm [ The default experimental settings used were (q 0.005, = 0.99, 7 0.99). By varying one parameter while keeping the others constant, the experiments, as shown in Tables TO 11 and 12 revealed that the impact of 7, which directly influences the target model, was most significant. contrast, effects of and alpha were comparatively lesser. Despite these hyperparameter variations, CMF still demonstrated superior performance compared to Source and TENT.
D LIMITATIONS
Our proposed CMF relies on the KF that uses a linear Gaussian model. In the context of CMF, this linear Gaussian model employs a linear system for the transition and emission of DNN's parameters.
While such linear systems have been frequently used in previous studies {Izmailov et al. 2018 Garipov et al., 20188 Guo et al., 2023) and have shown superior performance various applications as demonstrated in our paper, it is challenging to guarantee that all applications will depend on this linear system. According to the sensitivity analysis introduced in Appendix [C.4. we observed that model is relatively more influenced by the hyperparameter 7. This result implies that the linear system may not function in certain environments. To overcome this limitation, we plan future work to identify applications where the application of such linear systems is challenging and to apply simple non-linear functions (for example, confidence- based parameter selection). This approach aims enhance the adaptability and effectiveness of CMF in broader range of applications.
21

---

Published as a conference paper at ICLR 2024
E DETAILS OF RESULTS
This section provides detailed experimental results from 4.1
Table 13: Average error rate (%) and their corresponding standard deviations on ImageNet-C in the scenario of CS.
Mathed
Medel
shot
ingule:
deficess
mutiion
frost
fog
bright
elastic
pisclatc
jpss
ResNer
71.7
85.7a0.95
TENT
59.6
59.1
57.2
61.7
53.9
72.2
56.8
12.
15.1200
43.5
57.0
40.8
39.5
05.5
42.7
50,500,06
92.6
36.6
83.0
K2.0=000
COTTA
72.1
13.9.0.01
51.2+0.00
ResNer-50
90.3
86.9
76.9
90.6
80..
71.1
61.5
19.5.0.10
ROTTA
65.1
70.9
73.5
79-4
60.9
62.7
07.6
47.4
42.6
28.5
30.6
59.5
70.3
57.5
60.3
50.5
58.720:4
12.9w0:0
42.7
59.4
39.7
36-0
67.1
43.2
51.3.000
83.2
78.9
60.8
79.600.0
SAR
51.5
41.9
52.3.0012
72.2
51.3
67.1
53.9
65.5
50,7s0.07
69.9
54.8
72.5e1.44
FATA
57.9
52.7
62.9
56.2+0.20
71.5
79.9
76.2+20.33
ResNes-0
55.6
10
ROID
60.8
$7.3
$3.4
$2.6
50.9
50,7-0.08
55.0-0.36
42.0
54.3
59.7
47.400.0
ResNas-0
72.4
55.0
67.6:00.0
smf
55.8
58.9
48.9
50.9
52.9
47.4
40.6
49.0=3.10
52,1=0.12
41.6
51.6
41.6
45.7:0003
Table 14: Average error rate (%) and their corresponding standard deviations on ImageNet-C in the scenario of TC-CS. We choose q = 0.00025 for ResNet-50.
Metiod
Model
Adeetation Oreer (-1)
Avg.
gruesian
shot
impolee
defecus
glass
motion
zoom
snow
frost
fog
bright
contrnt
elastic
piselate
jptz
ResNes.50
$2.4
71.9
74.01
75.2
74.5
66.2
55.7
61.3
62.7
50.3
37.4
71.7
50.3
47.0
52.7
62.7:00.8
TENT
Swa
VIT
63.6
66.9
59.9
62.9
55.0
63.5
65.8
79.2
75.6
66.2
55.0
65.4
61.4
67.4
53.9
59.0
55.8
45.4
51.6
47.9
28.2
32.2
61.2
62.5
53.5
74.0
50.7
$2.8
42.4
59.3
54.5:0.04
54.0=0.14
D2V
43.9
43.1
43.3
69.4
76.4
57.9
67.1
40.5
44.7
51.3
27.3
47.0
60.5
18.5
43.2
51,9:0.09
Ro
$4.5
83.7
84.3
54.6
83.8
72.4
60.3
64.9
66.9
50.6
54.6
80.4
54.3
49.3
57.4
67.5-0.08
COTTA
Swis
VIT
65.5
71.3
67.1
69.9
65.2
75.4
72.8
68.6
81.6
74.0
63.9
63.9
68.2
66.2
57.9
56.6
44.6
51.8
46.0
43.1
26.9
28.5
61.7
92.9
71.6
56.4
40.3
$6.1
49.0
59,0
64.2=0.01
60.4:0.02
D2V
43.9
43.3
43.4
69.7
78.0
59.3
68.9
40.1
44.2
35.2
26.2
49.3
67.1
600.
43.0
51.7a0.22
Re
$8.4
83.2
83.3
92.2
85.7
74.7
60.6
66.9
66.0
53.9
35.5
77.2
56.0
50.5
55.4
68.620.16
ReTTA
Swa
VIT
65.5
71.0
69.3
67.1
73.8
64.9
73.2
68.9
80.4
73.3
62.7
62.8
67.2
65.2
56.9
55.6
49.7
44.1
45.7
42.9
27.9
29.1
80.3
59.0
69.2
$4.5
6110
59.7
59.0
49.8
63.420.01
591:0.05
D2V
43.9
43.3
43.3
69.7
77.8
59.4
68.7
39.8
42.5
35.8
26.2
49.7
66.6
60..1
43.3
51.320.01
ResNer. 50
$2.6
75.8
72.9
77.7
74.2
65.5
55.7
61.4
62.4
50.3
36.7
69.3
49.4
45.5
51.4
62.120.18
SAR
Swa
VIT
61.3
63.5
55.7
59.4
$4.4
55.0
62.0
77.1
73.8
61.4
53.8
65.0
$7.0
71.7
53.9
65.5
67.8
45.1
63.3
45.9
32.0
29.1
70.22
55. 1
71.8
51.5
54.5
49.2
63.0
40.3
51.720.14
5d 66.9c1.21
D2V
44.3
43.8
43.7
69.7
77.5
57.1
66.8
41.2
41.4
41.9
26.3
48.2
69.3
57.1
41.9
51.0=0.12
ResNet 50
77.1
67.2
65.6
73.6
69.1
62.1
53.6
55.8
59.5
45.2
35.8
63.5
47.6
44.0
43.7
58.200.15
FATA
Swa
VIT
63.2
61.6
55.6
56.3
$4.9
53.8
67.6
60.0
56.8
64.3
54.2
52.7
S4.5
54.9
51.9
51.5
43.5
47.2
43.5
42.9
26.3
29.1 1
49.2
46.6
48.7
54.6
61.5
46.4
47.5
39.8
52.9-0.55
69.90006
D2V
42.3
48.1
59.2
78.0
84.6
71.6
77.5
55.2
59.0
$7.9
46.9
73.9
76.7
71.6
59.2
64.4-1344
ReeNee- 50
73.0
62.8
62.7
69.6
66.8
57.2
49.1
52.2
50.7
43.5
33.3
59.3
45.3
41.3
45.9
$4.7.0.44
ROID
Swi
VIT
58.0
$7.6
$1.6
S1.S
$1.4
$2.2
62.9
55. 1
57.6
52-4
9
5
43.5
47.2
44.2
45.6
50.9
59.5
36.0
96.2
24.2
26.0
43.9
45.0
45.8
46.5
50.4
19/3
43.5
16.3
47.0-0.36
45.0-0.06
D2V
42.5
40.5
40.1
64.0
64.6
60.6
57.6
37.0
36.6
31.7
24.5
39.7
57.3
47.5
$7.1
44.8-0.01
ResNin-S0
73.0
60.6
61.8
69.9
66
57.8
50.0
53.5
57.6
44.6
144
58.2
45.3
42,0
45.6
$4.0-0.16
CMP
Swis
VIT
$7.6
58.4
50.8
50.5
40.9
$0.9
64.7
65.6
5K.1
51.0
507
7
47.1
46.9
44.3
46.1
40.5
30.4
36.7
37.9
24.6
16.7
44 9
42.4
48.1
41.1
47.9
19.6
40.3
36.1
44.
46.6=0.12
D2V
42.9
40.5
39.8
64.0
63.4
49.5
35.1
36.7
34.9
32.4
23.4
35.4
$2.0
44.9
35.6
43.5=0.84
22

---

Published as a conference paper at ICLR 2024
Table 15: Average error rate (%) and their corresponding standard deviations on ImageNet-C in the scenario of TC-LS (Ø- = 0.0) over TC-CS. We choose = 0.00025.
Method
Model
Ackepiation (nlr(-))
Avg.
gaassias
sbot
impulse
defocus
glass
motion
20001
SBOW
frost
Sog
beight
contrast
clastic
pisclate
jpcs
58.7
$3.9
54.3
58.4
58.7
99.9
99.9
99.9
50.6+0
TENT
76.4
75.5
74.9
89.00
99.8
43.6
00.6
65.4
70.6
38.8
25.1
58.6
47.8660.06
65.8
64.1
66.6
71.4
58.6
56.9
59.8e0
COTTA
70.3
19.3
74.4
72.8
81.0
71..2
39.5
51.2
3017
73.0
59.6
63.5a0.7
D2V
43.8
42.3
42.0
77.2
57.4
66.6
40.5
41.5
44.0
25.2
45.6
65.6
59.5
50.8+0.04
VIT
66.6
68.1
71.0
69.6
71.0
55.7
64.8
59.3
52.2
50.8
33.2
83.9
6030
$4.9
76.5
64.720.16
ROTTA
Swin
69.8
67.9
72.0
77.6
77.9
7002
72.9
39.5
55.8
66.9
42.1
83.3
75.6
97.4
$8.5
71.860.15
D2V
43.8
42.0
42.0
69.8
74.5
50.3 3
67.4
40.2
30.5
40.1
20.0
74.1
72.5
72.8
51.4
54.5e0.03
VIT
55.9
51.9
56.2
56.4
56.4
56.0
36.7
61.8
43.0
35,8
25.4
43.9
50.5
37.7
36.3
48.360.28
SAR
Swin
62.3
61.1
62.1
78.9
78.5
56.9
67.6
66.9
00:1
19.7
27.4
46.9
68.3
57.6
47.5
60.1100.4
D2V
44.
41.8
41.0
67.8
72.0
54.8
63.6
39.2
39.1
58.3
25.6
43.7
63.6
51.2
36.0
4H.3
VIT
59.5
64.0
70.7
77.1
73.7
75.2 2
12.7
21.8
67.6
73.5
56.6
98.9
77.4
71.9
66.5
71.8s1
EATA
65.4
72.8
76.6
81.8
79.3
75.9 9
13.5
72.3
68.2
69.3
55.9
$1.4
68.6
78.3
69.8
72.760077
49.2
72.1
N1.3
71.88
SM.242.21
39.0
39.6
44.1
LAME
42.1
$2.8
40.00
30.5
29.8
30.2
49.4
62.4
39.9
31.3
34.3
22.7
29.9
55.9
41.5
34.5
38.960.07
25.7
23.9
23.8
24.0
22.0
15.8
18.3
14.9
12.3
10.3
14.9
12.2
16:240.06
ROID
25.8
23.2
22.0
33.0
29.7
13.9
12.2
6.8
14.7
13.7
14.3
18.1+003
D2V
12.2
11.8
11.5
33.6
35.7
18.3
30.2
12.6
11.5
9.6
7.3
11.8
26.4
15.8
12.9
17.450.21
VIT
25.1
21.9
23.3
24.6
21.2
15.5
16.0
15.0
12.8
10.3
6.5
14.6
11.4
9.8
10.0
15.9:0004
CMF
Swin
24.9
2006
2008
35.2
27.7
17.1
17.3
12.1
11.4
10.0
64
13.6
11.8
12.4
12.0
16.750.10
D2V
12:4
12.0
11.9
28.8
23.9
15.6
22.4
11.2
10.2
8.9
6.3
11.3
17.9
13.0
9.7
14,4+6.24
Table 16: Average error rate (%) and their corresponding standard deviations on ImageNet-C in the scenario of TC-LS (6- = 0.01) over TC-CS. We choose g = 0.00025.
Mathad
Modd
Adiptation Order (-))
Axg.
gaussian
sbot
impulse
defecus
glass
motion
z00m
snow
frost
Eog
bright
contrast
clantic
piselate
ip9g
58.
53.7
60
TENT
60.7
57.3
55.8
77.2
21.3
53.6
99.6
99.9
85.
D2V
43.3
41.4
40.6
62.7
35.8
41.7
39.7
42.7
58.3
52.3
656
65.
63.6
66. 7
67.0
58.7
41.7
30.8
91.2
50.940
CNTTA
Swin
2033
74.5
72.8
65.6
70.0
$1.1
61.2
90.6
65 43
D2V
43.7
42.2
42.0
77.0
$7.4
66.7
41.3
46.9)
59.9
41.6
50.06006
VIE
65.3
64.7
65.6
67.7
68.5
36.0
51.5
55.2
48.5
46.7
30.6
71..5
54.7
76.4
66.3
50.210117
RoTTA
Swin
68.3
64.8
66.6
77.7
75.7
67.7
70.1
56.3
53.3
61.9
38.2
77.8
71.3
96.2
SA.3
D2V
43.5
41.3
41.0
68.7
71.6
56.7
65.0
39.3
38.4
38.7
28.3
66.7
67.9
66.0
49.2
52.3+1104
VIT
55.0
50.9
$3.6
$7.7
64.6
53.7
$4.4
60.7
40.9
35.5
26.8
64.3
$1.9
40.9
36.0
68, (3,740 29
SAR
Swin
$7.0
56.3
$7.0
78.2
75.8
64.3
63.1
69.5
59.4
51.7
26.3
44.6
72.1
54.5
46.0
Sk 46- 86
D2V
43.9
41.7
40.9
05.2
71.7
54.9
63.5
39.3
39.1
39.5
25.2
44.5
58.2
50,0
39.3
47.920.00
VIT
54.6
50.4
52.5
57.2
53.6
49.9
49.8
$1.3
45.8
43.4
30,3
51.1
44.5
41,0
4117
47.7 12
RATA
Swin
$4.1
50.5
54.6
65.3 2
61.3
53.2
49.6
48.0
46.1
42.2
27.6
$0.6
47.6
52.9
45.7
$0,040.5 35
D2V
52.2
$9.0
59.3
77.2
$4.5
20.3
76.7
55.3
59.3
$7.0
45.7
73.9
76.5
71.0
57.7
15.0
85.0
80.9
91.4
50.3
39.4
71.1
99.6
76.1
83.240
LAME
96.6
80.0
86.3
86.8
84.1
$4.3
92.7
$4.5
45.6
84.1
93.6
73.8
84.740.3 12
70.0
68.2
80.6
79.5all
49.2
46.0
37.2
35.9
28.2
36, 08
ROID
34.6
$2.2
50.1
20.4
29.6
36.1
35,940
47.4
43.3
44.6
45.6
41.9
34.5
32.7
19.8
34.
27.2
35,000.04
CMF
46.6
41.7
42.4
52.1
45.4
37.4
35.1
30.0
29,3
25.2
18.2
31.
29.2
31.5
35.140.16
D2V
34.5
32.6
32.6
48.5
45.1
16.6
41.5
38.2
27.1
23.2
16.9
28.7
30.2
32.6
25.5
31.740.44
23

---

Published as a conference paper at ICLR 2024
Table 17: Average error rate (%) and their corresponding standard deviations on ImageNet-C in the scenario of TCC-LS (6 = 0.1) over TC-CS.
Mothod
Model
Adaptation (-))r
Aug.
garescin
shot
impulse
defocess
glves
motion
snww
frost
fog
bright
contratt
choticc
pixelate
1191
58.2
53.7
54.6
59.1
66.6
99.8
90.9
80.1+0
TENT
60.5
57.0
99.3
98.5
93.9
14-1x01.2
433
41.4
62.8
41.3
19.2
42.8
52.0
47,740.03
65.6
63.6
66.6
66.9
41.5
30.8
91.4
62.1
50.9e0000
COTTA
70.3
69.3
74.6
72.8
69.9
62.7
59.7
05.7411
D2V
43.7
42.2
77.0
57.4
66.7
41.4
47.2
59.8
50.989
65.1
66.0
67.5
55.2
50.6
56.7
47.9
45
72.9
$4.0
74.5
59:260.07
ROTIA
Swin
68.0
64.0
65.2
77.9
75,0
66.9
69.5
56.1
51.7
00.9
37.4
76.4
70.8
96.0
$3.6
08.021022
D2V
43.5
41.2
30.9
68.4
71.1
36.4
54.5
39.1
38.3
38.5
28.3
65.5
67.5
67.2
49.1
52.049.01
VIT
54.0
50.5
53.4
57.6
60.8
$2.4
58.8
59.7
43.0
35.5 5
26-7
44.2
53.0
40.
36.0
48 48.4+0.00
SAR
Swin
56.7
55.8
56.5
77.4
76.4
66.7
64.7
67.2
59.0
47.5
26.3 3
45.1
75.1
$4.6
47.5
Sk. 45.4
D2V
43.9
41.7
40.9
68.7
71.7
$4.8
63.3
39.3
39.4
38.
25.3
44.7
58.1
49.8
39.2
48.000004
VIT
54,3
19.9
51.5
56.2
52.2
48.5
47.8
48.5
43.6
40.1
29.0
49.0
41.8
39.6
39.2
46, L20. 17
EATA
Swin
53.8
49.4
12.6
64.0
$8.9
51.9
48.0
47.3
43.6
30.7
27.3
48.7
45.2
50.2
43.5
-8,300
D2V
57.6
59.0
39.4
77.4
$4.6
70.6
76.9
55.3
59.3
$7.0
46.0
73.4
76.5
71.1
$7.9
65.5x19.11
VIT
93.4
79.4
97.2
72:4
90.9
96.8
93.8
96.5
49.2
$9.9
28.9
99,
64.6
66.0
50.4
79.940.06
LAME
86.5
76.6
87.8
84.8
97.6
84.4
$1.9
99.5
99.3
96.
29,4
67.8
99.8
95.3
60.8
64,540 09
45.7
45.7
72.4
60.2
$9.4
95.0
95.9
95,0
70.120004
53.1
50.3
51.5
50.7
43.0
41.11
33.9
43. 35
ROID
44.3
39.3
35.7
41.300003
51.4
48.7
37.5
31.9
39.5
32.4
CMP
50.5
16.3
36.9
39.3
38.9
34.6
37.8
36.5
39.620.02
D2V
40.3
38.5
38.4
52.7
49.9
42.5
46.4
34.0
32.8
28.6
22.2
34.4
38.6
31.6
36,2+0.06
Table 18: Average error rate (%) and their corresponding standard deviations on ImageNet-C in the scenario of TC-LS (6- = 1.0) over TC-CS.
Method
Model
Adpptation Order (-9)
Aeg.
garscian
shot
impulse
defocus
glass
motian
Sost
bright
costrast
closiic
piselite
ViT
58.2
53.7
67.4
64.22
T3.7
95.7
99.8
99.9
TENT
60.5
57.4
60.3
77.3
84.8
99.2
99.8
96.922.22 27
$4.6
43.6
52.2
47.740
65.6
66.9
41.6
30.8
91.3
62.1
59.929)37
COTTA
70.3
74.6
72.8
51.2
65.5
90.1
65.840
D2V
43.7
42.2
42.0
77.0
57.4
66.7
41.4
66.9
59.8
50.910008
65.0
63.9
67.2
67.9
60.7
47.8
45.4
30.0
73.9
74.7
50:240
RoTUN
Swin
68.1
66.2
65
77.8
75.1
67.1
59.6
55.9
51.7
60.5
37.4
76.1
70.6
96.0
85.8
57.940.00 10
D2V
43.5
41.2
40.8
68.3
71.1
56.4
$4.5
39.1
38.3
38.5
28.3
65.5
67.4
67.2
49.1
51,91004
VIT
54.8
50.6
$3.0
$8.9
61.9
53.7
54.3
61.0
30.8
35.6
26.7
44.2
51.4
42.8
35.9
48.3+0125
SAR
Swin
36.5
36.5
57.5
75.9
75.7
63.1
69.3
69.1
59.9
45.0
26.5
46.4
71.8
55.0
47.0
58.5
D2V
43.9
41.7
40.9
68.3
71.7
54.8
63.4
39.3
39.3
38.7
25.2
44.6
59.1
49.6
39.1
47 47.940.09
VIT
54.2
49.7
51.1
55.6
51.9
48.1
47,3
46.1
43.3
40.3
28.7
45.1
41.5
39.
38.7
45, 65.760.15
EATA
Swis
53.3
49.0
52.1
63.0
58.0
$1.0
47.1
46.7
42.3
39.5
26.7
48.11
43.7
48.8
43.5
47 47.42) 39
D2V
78.0
86.6
84.8
91.8
94.5
39.7
91.4
33.0
$3.6
$4..
90.7
91.3
91.7
$9.5
85.1
87.0218.44
VIT
03.8
1
97.1
72.6
91.1
96.8
94.0
96.
49.2
56.3 9
20.3 2
9
64.7
65.9
50.7
00.040
LAME
Swin
86.1
76.7
87.8
84.9
97.9
85.1
62.2
99.8
99.3
96.0
20.3 9
87.7
99.8
95.4
60.9
64.60006
D2V
45.9
45.8
45.3
72.2
60.2
$7.4
90.2
95.2
96.0
95.5
62.8
70.24007
VIT
55.2
50.3
51.4
$0.6
48.2
42.3
42.8
39.7
37.7
32.8
25.0
41.0
35.4
33.8
34.4
41.24003
ROID
50.0
44.2
30.3
40.00
41.988103
39.4
33.5
41.2x0.01
32.1
CMF
459
16.5
49.460.11
DZV
40.3
38.4
38.3
$2.9
50.2
42.3
46.2
33.9
32.7
28.5
22.2
34.5
39.2
31.4
35.120.03
24

---

Published as a conference paper at ICLR 2024
Table 19: Average error rate (%) and their corresponding standard deviations on ImageNet-C in the scenario of TCC-LS (6 = 5.0) over TC-CS.
Mothod
Model
Adaptation (-))r
Aug.
gareecan
shot
impulee
defoces
glves
motion
1nww
frost
fog
bright
conrratt
chosicc
piselate
1191
58.2
53.7
54.4
68.2
99.8
90.9
80.240
TENT
61.0
57.7
99.1
434
41.3
62.8
41.6
42.7
52.3
65.6
63.6
66.8
41.4
91.6
61.9
50.960137
COTTA
70.3
69.3
74.6
72.7
70.0
51.3
62.2
90.0
05
D2V
43.5
42.2
42.0
77.0
57.4
66.7
41.4
46.8
59.8
50.980.10
65.1
63.9
64.2
67.2
67.9
55.2
50.7
58.5
47.5
45.2
30.0
72.7
53.8
73.7
63.4
59.060.44
ROTIA
Swin
68.1
64.1
65.1
77.9
75.2
67.1
59.0
36.0
51.8
61.2
37.5
70.1
70.7
96.0
83.5
08.021.16
D2V
43.5
41.2
30.8
68.3
71.1
36.3
54.5
39.1
38.2
38.5
28.3
65.1
67.3
67.1
49.0
51.910005
VIT
55.0
50.7
54.9
66.6
60.1
48.
61.9
63.1
57.1
51.5
45.0
57.9
64.2
54.1
$2.0
65.5+12.62
SAR
Swin
56.6
55.0
56.9
80.7
73.7
65.3
69.0
70.9
60.6
54.2
25.8
44.7
73.6
55.3
45.2
50 28
D2V
44.0
41.7
40.9
68.4
71.7
$4.8
63.5
39.2
39.1
38.6
25.2
44.3
S5.1
50.1
39.3
47.9+0008
VIT
54.2
19.7
51.1
55.6
51.5
47.9
47.2
47.8
43.3
40.0
28.7
47.5
41.8
39.3
38.8
45. 25.620.1 17
EATA
Swin
53.5
49.5
52.0
63.7
58.2
51.5
46.9
45.5
42.5
38.0
26.6
48.11
44.4
49.3
43.5
47.6 25
D2V
58.9
59.3
39.7
77.4
$4.8
70.5
77.1
55.4
59.5
59.1
46.3
75.3
76.7
71.2
58.3
65.911892
VIT
93.4
79.6
97.2
73.6
96.5
93.9
96.6
49.3
$9.9
29.5
99
65.1
66.2
51.2
60,240.00
LAME
86.3
77.3
88.0
85.5
97.8
85.0
62.2
99.6
99.5
96
30,1
99.8
95.6
84.940
45.7
46.0
73.3
95.1
96.0
95.99
70.520.12
53.2
50.2
51.5
50.5
42.9
25,0
43.2
33.9
43.330005
ROID
44.2
30.5
39.3
35.6
41.2+003
48.4
37.5
31.8
39.2
32.2
39.540.83
CMP
50.5
16.0
36.7
48.8
41.5
39.2
38.9
34.3
29.
57.2
36.3
D2V
40.3
38.4
38.3
$2.9
50.0
42.5
46.1
33.8
32.5
28.4
34.3
41.7
38.1
38,080005
Table 20: Average error rate (%) and their corresponding standard deviations on ImageNet-C in the scenario of TC-LS (6- = 0.01) over CS. We choose 0.00025.
Method
Model
Adoptation Order (-9)
Aeg.
garscian
shot
impulse
defocus
glass
motiat
Sostt
bright
costrast
clisii
piselite
1889
ViT
86.3
85.4
X5.
55.5
301.1
53.3
79.3
82.7
79.5
TENT
90.3
89.5
90.4
84.7
35.9
K5.6
M6.622.6 22
43.1
42.0
$2.1
41.8
47.140009
60.9
59.6
72.5
95.3
61.5
60
COTTA
70.6
77.5
64.8
57.7
65.9220.0
D2V
42.6
41.6
41.8
73.4
55.3
65.6
40.3
33.9
64.3
55.1
45,720.05
85.6
86.0
75.0
66
71.5
64.7
93.8
RoTUN
Swin
$1.3
80.3
83.3
84.6
$3.1
78.9
$1.7
63.9
65.9
73.5
49.6
84.6
$2.6
96.4
$7.4
78.510
D2V
44.7
45.8
46.0
77.7
75.5
64.7
72.1
39.5
41.9
38.5
27.8
63.9
68.5
67.6
44.8
51.910077
VIT
64.2
62.4
62.9
50.0
64.0
52.7
60.8
$4.9
49.5
47.5
30.4
$3.9
53.3
48.7
42.5
14.1
SAR
Swin
21.5
70.0
72.2
74.0
76.6
61.4
66.1
66.7
61.6
57.8
32.6
62.3
73.0
79.8
55.6
65. 53
D2V
43.2
42.0
42.3
62.7
68.7
52.4
62.5
39.3
39.5
35.5
24.3
43.0
50.2
$2.4
40.5
47.24000
VIT
79.5
78.6
79.1
79.8
78,
73.9
75.4
68.1
62.1
65.7
40.7
90.5
63.7
68.5
54.3
60
EATA
Swis
84.5
85.1
X5.
86.7
$5.8
76.7
76.0
72.1
71.4
69.8
43.2
86.1
74.3
59.2
71.5
77.12
D2V
83.2
82.9
82.9
90.9
93.8
35.3
90.2
$1.4
$1.5
$1.7
778.5
83.9
903.7
$8.5
83.4
85311189
VIT
36.3
36.1
36.2
36.3
36.4
16.2
36.4
16.1
35.9
36.2
35.6
56.7
36.0
36.
35.8
36, 14.00000
LAME
Swin
37.5
37.3
37.5
37.5
$7.8
17.4
57.7
17.2
$7.1
37. 3
36.6
37.5
37.5
37.9
37.3
37. 37.440.0 12
D2V
36.1
35.9
35.9
36.5
36.6
36.4
36.6
36.1
36.3
36.3
35.9
36.5
36.4
36.3
36.340.11
VIT
27.0
26.2
26.1
32.0
23.4
29.2
23.4
19.4
18.6
10.8
26.6
25.5
21.5
17.3
23.61005
ROID
30.9
30.1
25.01
28.6811 16
14.4
14.4
27.0
16.4
21.2
18.80001
27.1
26.2
28.9
25.3
CMF
20.2
26.3
29.0
22.3
25.0
31.2
27.140
DZV
14.4
14.0
14.1
24.3
26.0
18.2
25.3
13.4
13.4
13.2
15.7
19.3
14.4
17.120.99
25

---

Published as a conference paper at ICLR 2024
Table 21: Average emror rate (%) and their corresponding standard deviations on D109 in the scenario of CS.
Method
Model
clipart
infograph
painting
real
sketch
Avg.
ResNet-50
56.0
80.8
50.2
25.4
64.4
55.640.08
VIT
81.8
90.9
74.2
48.7
88.4
76.8+0.36
TENT
Swin
66.1
83.9
55.8
25.4
76.5
61.5#0.41
D2V
60.6
83.0
52.3
25.5
68.0
57.9+0.42
ResNet-50
56.4
78.3
50.8
25.4
65.4
55.3+0.04
ViT
56.6
76.4
44.6
21.9
67.1
53.3+0.04
CoTTA
Swin
52.5
73.8
44.1
20.5
64.9
51.240.03
D2V
48.6
73.0
40.8
20.2
56.6
47.8+0.01
ResNet-50
57.2
78.1
48.5
24.1
65.9
54.8+0.04
VIT
53.2
73.4
42.9
21.6
63.2
50.9+0.05
RoTTA
Swin
49.1
71.4
41.4
20.1
60.9
48.640.05
D2V
46.8
71.9
40.3
20.1
55.1
46.8+0.03
ResNet-50
54.4
77.3
49.2
24.9
62.3
53.6+0.07
VIT
66.2
82.9
54.1
26.1
76.4
61.240.36
SAR
Swin
55.8
78.4
46.7
21.6
67.3
53.9+0.08
D2V
48.8
74.0
40.6
20.1
56.9
48.140.08
ResNet-50
53.0
78.0
49.4
24.3
60.8
53.140.09
VIT
50.1
71.5
41.7
20.7
58.4
48.5+0.11
EATA
Swin
49.3
71.8
42.0
19.9
61.1
48.8+0.12
D2V
46.3
71.6
39.4
19.4
54.5
46.2+0.05
ResNet-50
51.0
75.8
46.7
23.7
57.3
50.9+0.04
VIT
48.6
69.7
40.6
20.5
55.2
46.9+0.02
ROID
Swin
48.2
69.9
40.6
19.6
57.7
47.2+0.07
D2V
44.6
70.0
38.0
19.3
53.2
45.0+0.01
ResNet-50
47.6
75.8
46.3
24.0
53.5
49.4+0.21
ViT
44.0
67.8
39.6
20.6
50.2
44.5+0.08
CMF
Swin
43.9
67.8
39.3
19.7
53.3
44.8+0.04
D2V
41.7
67.4
36.4
18.7
49.8
42.8+0.05
26

---

Published as a conference paper at ICLR 2024
Table 22: Average (%) and their corresponding standard deviations D109 in the scenario of TC-CS. We choose 0.00025 for ResNet-50.
Adaptation Order (-)
Method
Model
Avg.
clipart
infograph
painting
real
sketch
ResNet-50
53.0
78.1
48.0
24.7
60.2
52.8+0.04
VIT
57.3
86.1
80.9
93.1
99.2
83.3+0.13
TENT
Swin
52.6
80.1
59.9
43.7
95.6
66.440.33
D2V
49.1
78.8
56.6
40.4
89.5
62.9+0.21
ResNet-50
54.7
79.7
49.5
24.6
62.5
54.2+0.07
ViT
56.8
76.0
45.0
21.8
66.8
53.3+0.03
CoTTA
Swin
52.6
73.7
44.2
20.5
64.8
51.2+0.01
D2V
48.7
72.9
41.0
20.1
56.5
47.8+0.02
ResNet-50
55.1
78.3
48.5
24.0
59.3
53.0+0.03
VIT
56.3
74.6
43.5
21.5
61.4
51.440.03
RoTTA
Swin
52.1
72.8
42.1
20.0
58.5
49.1
D2V
48.6
72.6
40.7
20.0
53.9
47.2+0.03
ResNet-50
53.3
77.9
47.5
24.5
58.7
52.6+0.01
VIT
55.7
82.6
53.0
21.5
73.5
57.3+0.41
SAR
Swin
51.2
78.4
48.6
20.8
68.6
53.5+1.05
D2V
48.3
74.4
42.9
20.3
56.5
48.5+0.10
ResNet-50
51.1
76.6
47.1
24.0
57.6
51.3+0.25
VIT
52.4
70.1
40.8
20.6
52.4
47.240.10
EATA
Swin
50.3
70.3
41.2
19.5
55.4
47.440.18
D2V
47.3
71.0
39.2
19.2
52.2
45.8+0.06
ResNet-50
45.6
74.3
44.8
23.1
52.7
48.140.07
VIT
46.2
68.2
39.9
20.5
50.2
45,0+0.04
ROID
Swin
46.1
67.7
39.8
19.7
52.2
45.1+0.10
D2V
44.0
69.0
37.7
19.3
51.2
44.2+0.06
ResNet-50
45.2
74.3
44.9
22.9
51.9
47.810.06
ViT
43.9
66.4
39.4
20.0
47.4
43.4+0.07
CMF
Swin
44.2
66.6
39.0
19.2
48.8
43.640.12
D2V
43.0
66.5
36.3
18.5
47.3
42.3+0.11
27

---

Published as a conference paper at ICLR 2024
Table 23: Average rate (%) and their corresponding standard deviations on D109 in the scenario of TC-LS 0.0) over TC-CS. choose = 0.00025.
Adaptation Order (-)
Method
Model
Avg.
clipart
infograph
painting
real
sketc
VIT
58.2
86.1
82.2
94.2
99.2
84.0+0.01
TENT
Swin
52.9
79.9
60.5
51.0
98.4
68.5+0.17
D2V
49.2
78.3
56.3
40.7
88.3
62.640.01
ViT
56.8
76.1
45.0
22.1
66.8
53.340.04
CoTTA
Swin
52.6
73.7
44.2
20.6
64.9
51.240.01
D2V
48.7
72.9
41.0
20.1
56.6
47.8+0.01
VIT
56.7
75.5
44.9
22.7
68.3
53.6+0.08
RoTTA
Swin
52.5
73.3
43.2
20.3
63.7
50.6+0.03
D2V
48.7
72.8
41.1
20.5
56.6
48.0+0.02
VIT
64.5
80.9
51.8
21.8
73.5
58.5+0.40
SAR
Swin
60.3
77.7
50.0
20.7
68.3
55.4+0.17
D2V
48.7
75.4
46.8
20.1
56.6
49.5+0.04
VIT
53.5
71.1
52.8
34.6
80.8
58.6+1.45
EATA
Swin
51.2
71.3
45.7
29.8
73.2
54.2+0.99
D2V
47.3
70.5
40.3
20.0
52.6
46.1+0.37
ViT
29.5
75,6
18.7
9.7
42.6
35.2+0.55
LAME
Swin
24.7
62.9
18.6
8.3
36.2
30.1 16
D2V
26.0
68.8
19.2
8.0
26.7
29.7+0.15
ViT
28.9
55.8
24.9
11.1
36.3
31.440.07
ROID
Swin
26.6
54.0
24.2
10.6
36.2
30.3+0.25
D2V
25.4
56.0
21.2
10.5
33.3
29.3+0.03
VIT 28.5
54.8
25.5
11.0
35.4
31.0+0.10
CMF
Swin
26.6
52.8
23.8
10.5
34.5
29.6+0.21
D2V
24.9
52.8
20.4
10.2
30.9
27.8+0.12
28

---

Published as a conference paper at ICLR 2024
Table 24: Average rate (%) and their corresponding standard deviations on D109 in the scenario of TC-LS 0.01) over TC-CS. We choose = 0.00025.
Adaptation Order (-)
Method
Model
Avg.
clipart
infograph
painting
real
sketc
VIT
57.5
86.1
80.8
93.5
99.2
83.4+0.30
TENT
Swin
52.7
79.9
59.5
48.6
96.8
67.5+1.86
D2V
49.1
78.7
56.5
40.9
89.3
62.9+0.35
ViT
56.8
76.1
45.0
22.0
66.9
53.4+0.01
CoTTA
Swin
52.6
73.7
44.2
20.6
64.9
51.2+0.02
D2V
48.7
72.9
41.0
20.1
56.5
47.9+0.00
VIT
56.7
75.4
44.6
22.5
67.3
53.3+0.08
RoTTA
Swin
52.5
73.3
43.1
20.3
62.8
50.4+0..3
D2V
48.7
72.8
41.1
20.4
56.2
47.8+0.00
VIT
62.8
82.2
51.1
21.8
74.8
58.640.80
SAR
Swin
52.5
78.9
46.8
20.7
69.5
53.7+0.53
D2V
48.3
75.2
45.1
20.4
56.4
49.1 14
VIT
53.3
70.3
42.1
26.7
61.3
50.7+1.20
EATA
Swin
50.5
70.6
43.4
22.1
61.6
49.6+0.41
D2V
47.3
70.9
41.5
22.7
52.9
47.1
ViT
43.2
82.9
29.4
14.7
54.0
44.8+0069
LAME
Swin
38.8
73.4
29.2
11.4
46.8
39.9+0.77
D2V
39.8
79.2
29.6
12.5
38.1
39.940.56
ViT
31.0
56.9
25.5
11.5
35.8
32.2+0. 10
ROID
Swin
28.6
55.0
24.5
11.2
36.4
31. 11
D2V
27.9
57.6
22.3
10.9
34.7
30.7+0.09
VIT
30.7
56.1
25.5
11.4
35.2
31.8+0.10
CMF
Swin
28.3
53.6
23.8
11.0
35.0
30.3+0.24
D2V
27.0
53.7
21.2
10.4
30.5
28.6+0.11
29

---

Published as a conference paper at ICLR 2024
Table 25; Average rate (%) and their corresponding standard deviations on D109 in the scenario of TC-LS 0.1) over TC-CS.
Adaptation Order (-))
Method
Model
Avg.
clipart
infograph
painting
real
sketc
VIT
57.3
86.3
81.0
93.9
99.2
83.5+0.08
TENT
Swin
52.7
80.0
59.4
44.6
95.3
66.440.38
D2V
49.1
78.9
56.5
40.5
89.3
62.9+0.29
ViT
56.8
76.0
45.0
21.8
66.8
53.3+0.04
CoTTA
Swin
52.6
73.7
44.2
20.5
64.8
51.24+0.3
D2V
48.7
72.9
41.0
20.1
56.5
47.9+0.00
VIT
56.5
75.1
44.1
22.1
65.1
52.6+0.06
RoTTA
Swin
52.3
73.0
42.6
20.2
61.1
49.8+0.05
D2V
48.7
72.7
40.9
20.2
55.3
47.6+0001
VIT
59.0
83.2
50.5
21.6
74.4
57.740.56
SAR
Swin
51.6
78.8
47.8
20.8
68.0
53.4+0.70
D2V
48.4
74.6
43.5
20.3
56.4
48.6+0.04
VIT
52.2
69.9
41.2
21.0
52.7
47.4+0. 16
EATA
Swin
50.2
70.0
41.2
19.8
55.9
47.4+0.21
D2V
47.3
70.7
39.2
19.4
51.9
45.7+0.08
ViT
74.0
95.3
57.6
37.2
80.5
68.940.24
LAME
Swin
70.3
92.2
57.7
27.9
74.9
64.6+0.25
D2V
69.8
94.5
57.6
32.8
68.2
64.6+0.25
ViT 37.9
63.2
30.9
13.6
41.3
37.3+0. 12
ROID
Swin
36.8
62.1
30.2
13.2
42.3
36.940. 11
D2V
35.5
63.7
28.0
12.8
41.7
36.3+0.06
VIT 36.6
61.2
30.5
13.7
38.4
36.1+0.11
CMF
Swin
35.7
59.8
28.8
12.5
38.4
35.0+0.05
D2V
34.4
60.3
26.6
12.1
36.8
34.1+0.13
30

---

Published as a conference paper at ICLR 2024
Table 26: Average rate (%) and their corresponding standard deviations on D109 in the scenario of TC-LS 1.0) over TC-CS.
Adaptation Order (-))
Method
Model
Avg.
clipart
infograph
painting
real
sketc
VIT
57.2
86.1
81.1
93.6
99.2
83.4+0.07
TENT
Swin
52.6
79.9
59.3
43.2
95.4
66.12
D2V
49.1
78.8
56.5
40.0
89.3
62.7+0.17
ViT
56.8
76.0
45.0
21.8
66.7
53.3+0.01
CoTTA
Swin
52.6
73.7
44.2
20.5
64.9
51.2+0.02
D2V
48.7
72.9
41.0
20.1
56.5
47.8+0.01
VIT
56.4
74.8
43.7
21.7
62.9
51.940.06
RoTTA
Swin
52.1
72.8
42.1
20.0
59.3
49.3+0.04
D2V
48.6
72.7
40.7
20.0
54.4
47.3+0.02
VIT
54.4
82.6
53.7
21.5
74.6
57.4+0.12 12
SAR
Swin
51.6
78.6
52.0
20.7
69.6
54.5+0.68
D2V
48.3
74.4
43.1
20.3
56.4
48.5+0.09
VIT
52.5
70.1
40.6
20.5
52.3
47.2+0.04
EATA
Swin
50.4
70.1
41.0
19.5
55.8
47.4+0. 10
D2V
47.3
71.1
39.1
19.2
52.0
45.7+0.04
ViT
96.6
99.6
85.1
71.1
97.8
90.0+0.09
LAME
Swin
94.1
99.5
85.0
59.1
96.8
86.9+0.24
D2V
94.8
99.6
84.7
68.5
94.1
88.3+0.13
ViT
44.1
67.5
37.6
17.0
48.2
42.940.03
ROID
Swin
44.1
67.0
37.3
16.5
50.3
43.040.06
D2V
42.4
68.2
35.0
15.8
49.3
42.2+0.04
VIT
42.4
65.7
36.9
16.3
45.4
41.3+0.06
CMF
Swin
42.8
65.4
35.9
15.5
46.7
41.3+00.4
D2V
41.3
65.5
33.5
15.0
45.1
40.1+0.10
31

---

Published as a conference paper at ICLR 2024
Table 27: Average emror rate (%) and their corresponding standard deviations on D109 in the scenario
TC-LS 5.0) over TC-CS.
Adaptation Order (-))
Method
Model
Avg.
clipart
infograph
painting
real
sketc
VIT
57.3
86.0
80.5
92.8
99.2
83.1+0.26
TENT
Swin
52.6
79.9
59.3
43.0
95.3
66.0+0.07
D2V
49.1
78.8
56.5
40.1
89.6
62.8+0.06
ViT
56.8
76.0
44.9
21.8
66.8
53.3+0.02
CoTTA
Swin
52.6
73.7
44.2
20.5
64.8
51.240.01
D2V
48.7
72.8
41.0
20.1
56.5
47.8+0.01
VIT
56.3
74.7
43.6
21.7
62.5
51.7+0.05
RoTTA
Swin
52.1
72.7
42.0
20.0
59.1
49.2+0.03
D2V
48.6
72.6
40.7
20.0
54.2
47.2+0.01
VIT
55.2
82.8
53.7
21.5
73.6
57.3+0.22
SAR
Swin
51.8
78.5
49.2
20.7
69.7
54.0+0.72
D2V
48.3
74.3
43.0
20.3
56.4
48.4+0.12 12
VIT
52.3
70.0
40.7
20.5
52.4
47.2+0.08
EATA
Swin
50.6
70.4
41.1
19.5
55.1
47.3+0.05
D2V
47.3
71.0
39.0
19.2
51.9
45.7+0.06
ViT
98.8
99.6
92.1
76.9
99.1
93.3+0.17 17
LAME
Swin
98.0
99.6
91.9
64.3
99.3
90.6+0.23
D2V
98.4
99.6
91.7
75.7
98.3
92.8+0.16
ViT
45.3
67.8
38.9
18.3
49.2
43.940.09
ROID
Swin
45.0
67.8
38.9
17.6
51.3
44..00
D2V
43.5
68.7
36.4
17.0
50.3
43.2+0.04
VIT
43.6
66.1
38.5
17.6
46.5
42.5+0.08
CMF
Swin
43.8
66.3
37.9
16.7
47.7
42.5+0.07
D2V
42.4
66.0
34.9
16.1
46.0
41.1+0.06
32

---

Published as a conference paper at ICLR 2024
Table 28: Average rate (%) and their corresponding standard deviations on D109 in the scenario
TC-LS 0.1) over CS. We choose q = 0.00025.
Adaptation Order (-))
Method
Model
Avg.
clipart
infograph
painting
real
sketc
ViT
82.1
91.1
73.5
47.4
88.6
76.5+0.52
TENT
Swin
66.5
84.0
55.6
25.1
76.5
61.5+0.31
D2V
60.4
83.0
51.8
25.2
67.7
57.640.46
ViT
56.8
76.4
44.7
21.9
67.3
53.420.02
CoTTA
Swin
52.4
73.9
44.1
20.5
65.0
51.2++0.3
D2V
48.6
72.9
40.8
20.2
56.5
47.8+0.03
VIT
56.8
75.2
44.9
22.5
67.5
53.4+0.05
RoTTA
Swin
51.5
72.8
42.9
20.4
64.1
50.4+0007
D2V
48.5
72.8
41.1
20.5
56.6
47.9+0.05
VIT
66.1
82.5
54.0
26.0
76.4
61.0+0.51
SAR
Swin
55.6
77.7
46.4
21.5
67.1
53.6+0.24
D2V
49.6
74.7
41.1
20.2
57.5
48.6+0.35
VIT
55.3
73.6
45.9
24.9
64.6
52.9+2.98
EATA
Swin
51.0
71.9
44.3
21.1
63.2
50.3+0.25
D2V
45.5
70.7
39.4
19.7
54.1
45.9+0.13
ViT
28.7
27.9
32.2
30.5
30.2
29.940.18 18
LAME
Swin
27.4
26.6
31.0
29.0
28.8
28.6+0.23
D2V
27.7
27.2
31.6
29.6
29.1
29.1
ViT
27.1
47.4
24.8
13.1
33.3
29.1
ROID
Swin
24.7
47.0
24.0
12.7
32.4
28.2+0.05
D2V
22.3
46.0
21.9
12.3
29.1
26.3+0.07
VIT
26.7
47.1
24.4
13.0
32.3
28.7+0.19
CMF
Swin
23.8
46.4
23.4
12.6
30.7
27.3+0.05
D2V
20.1
45.1
20.9
12.0
26.4
24.9+0.10
33


Reviews:
[{"review_id": "GdM5xmrhOp", "reviewer": "Reviewer_4Nt5", "summary": "The paper proposes parameter averaging/filtering for online test-time adaptation. The idea is to first find the task minimum, and then use the previous task to form a “prior” and average the parameters. The actual algorithm can be thought of as a simplified version of the Kalman filtering algorithm but they do not consider the full covariance because of high dimensionality. The paper shows the algorithm’s effectiveness on online test-time adaptation, where a full network has been pretrained and only normalization parameters are “adapted” to each task, where different tasks exhibit distribution shift on image styles and textures or speech environments.", "questions": "See above comments.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 4, "strengths": "- The proposed methodology is properly derived from a Kalman filter algorithm and has probabilistic interpretations.\n- The proposed methodology can be adapted and simplified to a deep network where only normalization parameters are changing.\n- Empirically, the proposed method can be applied on various backbone networks and achieve strong results on online test-time adaptation.", "weaknesses": "- The proposed method seems to work for online test-time adaptation which requires a well trained network. It would be good to investigate whether this could be a limitation. It would be good to understand whether it relies on a pretrained network or the continual style of training can also extend to a network from scratch. It would also be good to understand the limitation on the number of adaptation parameters and the number of tasks it can continually learn without forgetting. When does the method break down? Since the methodology is a general one, it would be good to understand its general characteristics. To this end, I would appreciate to see some toy experiments on parameter averaging that answers these questions.\n\n- It would be good to study on the sensitivity of each hyperparameter.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "hn96yqs321", "reviewer": "Reviewer_iWLp", "summary": "This paper addresses the  Online Test-time Adaptation (OTTA) by applying the Kalman filter to infer the test-time \"model parameters\". The proposed method is evaluated over various datasets.", "questions": "See weakness.", "limitations": "", "rating": 3, "confidence": 4, "soundness": 1, "presentation": 2, "contribution": 2, "strengths": "- Casting the OTTA challenge as a Bayesian filtering issue presents an interesting approach. The incorporation of the Kalman filter (KF) to facilitate this is noteworthy, as it offers a closed-form solution for posterior inference. However, the inherent linear assumption of the KF may not align well with many real-world scenarios.\n\n- The research commendably evaluates the proposed method across a spectrum of image classification and speech recognition tasks, and the results indicate reasonable enhancements.", "weaknesses": "- Utilizing the Kalman filter for sequential model parameter inference might not be optimal given its inherent assumptions. The Kalman filter operates under the presumption of linear system dynamics and posits a Gaussian distribution for the posterior. This is often misaligned with real-world scenarios where state posteriors frequently exhibit multimodal distributions.\n\n- While the exploration of the Kalman filter and other Bayesian filtering techniques for model adaptation/TTA is not entirely novel, it is crucial to delineate this work from previous contributions. It's recommended to rigorously compare, both theoretically and empirically, with established works like EKF[1] and PFDE[2]. Such a comparative analysis can better spotlight this paper's unique technical contributions.\n\n- In Section 3.4, the paper discusses simplifying Bayesian filtering computations. An analysis evaluating the accuracy of these approximations is warranted.\n\n[1] Abuduweili, A., & Liu, C. (2020, July). Robust online model adaptation by extended kalman filter with exponential moving average and dynamic multi-epoch strategy. In Learning for Dynamics and Control (pp. 65-74). PMLR.\n[2] Huang, H., Gu, X., Wang, H., Xiao, C., Liu, H., & Wang, Y. (2022). Extrapolative continuous-time bayesian neural network for fast training-free test-time adaptation. Advances in Neural Information Processing Systems, 35, 36000-36013.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "7SsKRx5rBg", "reviewer": "Reviewer_gT9R", "summary": "The paper introduces a novel framework called Continual Momentum Filtering (CMF) that leverages the Kalman Filter to strike a balance between model adaptability and information retention. The CMF framework alleviates the catastrophic forgetting issue and provides high adaptability to shifting data distributions. The paper provides examples of real-world situations where the CMF framework has been validated, including scenarios involving covariate and label shifts in speech recognition tasks and ImageNet-C. The results show that the CMF consistently outperforms state-of-the-art methods.", "questions": "Are there any sensitivity analyses regarding the parameter $I$ in algorithm 1?\n\nI am curious about the performance of replacing the “weight ensemble” in ROID with CMF？ (namely ROID with CMF).\n\nHow about the performance of “DW-SLR + SCE” without CMF?\n\nHow about the in-distribution performance of compared methods after the adaptation of out-of-distribution data? Please refer to the comparison manner proposed in EATA.\n\nCould the authors provide a computational complexity (wall-clock GPU time) comparison regarding the proposed CMF? \n\nCould CMF help MEMO work stably in the online setting? \n\nMore motivation/explanations from the high level about why CMF could achieve a better tradeoff between adaptability and information retention are preferred.\n\nCould the authors provide implementation details (hyperparameters) of baselines on different models and datasets?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The investigated problem, namely adaptability and anti-forgetting tradeoff, is practical for the real-world deployment of TTA methods. The resulting CMF framework is simple yet effective. \n\nExperimental evaluations on various models, datasets and scenarios are thorough and demonstrate the effectiveness of the proposed framework.", "weaknesses": "The performance gains compared with ROID are a bit marginal.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "SBymNhwyGf", "reviewer": "Reviewer_aPnP", "summary": "The paper focuses on online test time adaptation and introduces a Kalman Filter (KF) based approach on momentum filtering of DNNs.  The proposed method, referred to as CMF,  integrates a SGD-based optimization process with a KF-based inference (filtering) process. Some simplification techniques, such as momentum smoothing, have been employed to reduce time complexity. The experimental results demonstrate the competitiveness of the proposed method across multiple datasets.", "questions": "1) How does the proposed method compare to related Kalman filter-based techniques for adaptation, as discussed in [1,2]?\n\n2) What are the advantages and disadvantages of the proposed approach in contrast to replay-based methods [3,4] for mitigating catastrophic forgetting?\n\n3) In the alation study on “Effectiveness of the source-conjugated transition model”, do the hyperparameters alpha and gamma remain the same? It is better to explore the effects of hyperparameters alpha and gamma individually. \n\n4) What is the performance and computational cost comparison between the simplified scalar version of the KF process and the original matrix version of the KF process?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The paper studies a practical and important problem: online test-time adaptation, focusing on mitigating catastrophic forgetting during adaptation.\n\nThe proposed method, which employs a KF-based approach to filter the momentum parameter of DNNs, provides a sound solution to mitigate catastrophic forgetting in online test-time adaptation.\n\nOverall, the paper is well-structured and effectively communicates the details of the proposed method.\n\nFurthermore, in the experimental evaluation, the proposed method demonstrates superior performance compared to previous approaches, as reported in the paper.", "weaknesses": "1) This paper could benefit from a comparison with related works that utilize the Kalman filter for online adaptation tasks, as seen in [1,2].  Such a comparison or discussion regarding the difference between the proposed method and [1,2] would be beneficial.\n\n2) It is very common to use replay-based approaches to overcome catastrophic forgetting, such as ER [3] and A-GEM [4].  An analysis of the advantages and disadvantages of the proposed approach in contrast to replay-based methods would provide a more comprehensive understanding of how it addresses the issue of catastrophic forgetting.\n\n3) In the alation study on “Effectiveness of the source-conjugated transition model”, do the hyperparameters alpha and gamma remain the same? It is better to explore the effects of hyperparameters alpha and gamma individually. Since the roles of alpha and gamma are distinct and independent, isolating their impact would provide a more detailed understanding.\n\n4) The paper simplifies the inference process by using scalar parameters to replace matrix parameters in the KF process. It would be helpful to conduct an experiment on a smaller model to compare the performance of the simplified version with the original matrix version of the method. \n\n[1] A. Abuduweili, et al, “Robust online model adaptation by extended kalman filter with exponential moving average and dynamic multi-epoch strategy”, L4DC 2020.  \n[2] Y. Cheng, et al. \"Human motion prediction using semi-adaptable neural networks.\" ACC, 2019.  \n[3] A. Chaudhry, et al “On tiny episodic memories in continual learning”, arxiv 2019.  \n[4] A. Chaudhry, et al “Efficient lifelong learning with a-gem”, ICLR 2018.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)