E-BATS: Efficient Backpropagation-Free Test-Time Adaptation for Speech Foundation Models
Authors: Jiaheng Dong
Hong Jia
Soumyajit Chatterjee
Abhirup Ghosh
James Bailey
Ting Dang
Venue: NeurIPS 2025 poster ()
Keywords: Speech Recognition; Online Learning
Primary area: applications

Abstract: Speech Foundation Models encounter significant performance degradation when deployed in real-world scenarios involving acoustic domain shifts, such as background noise and speaker accents. Test-time adaptation (TTA) has recently emerged as a viable strategy to address such domain shifts at inference time without requiring access to source data or labels. However, existing TTA approaches, particularly those relying on backpropagation, are memory-intensive, limiting their applicability in speech tasks and resource-constrained settings. Although backpropagation-free methods offer improved efficiency, existing ones exhibit poor accuracy. This is because they are predominantly developed for vision tasks, which fundamentally differ from speech task formulations, noise characteristics, and model architecture, posing unique transferability challenges.
In this paper, we introduce E-BAT, first Efficient BAckpropagation-free TTA framework designed explicitly for speech foundation models. E-BAT achieves a balance between adaptation effectiveness and memory efficiency through three key components: (i) lightweight prompt adaptation for a forward-pass-based feature alignment, (ii) a multi-scale loss to capture both global (utterance-level) and local distribution shifts (token-level) and (iii) a test-time exponential moving average mechanism for stable adaptation across utterances. Experiments conducted on four noisy speech datasets spanning sixteen acoustic conditions demonstrate consistent improvements, with 4.1\%--13.5% accuracy gains over backpropogation-free baselines and 2.0$\times$–6.4$\times$ GPU memory savings compared to backpropogation-based methods. By enabling scalable and robust adaptation under acoustic variability, this work paves the way for developing more efficient adaptation approaches for practical speech processing systems in real-world environments.

E-BATS: Efficient Backpropagation-Fre Test-Time Adaptation for Speech Foundation Models
Jiaheng Dong
Hong Jia University of Auckland hong. jia@auckland. h
The University of Melbourne
jiaheng dong@student, unimel... edu.
Soumyajit Chatterjee
Abhirup Ghosh
Nokia Bell Labs, UK
University Birmingham g.ohhhhh. 1@bham. ac uk
James Bailey
Ting Dang The University Melbourne
The University of Melbourne baileyj @unimelb. edu.
Ting Dang@un imelb edu
Abstract
Speech Foundation Models encounter significant performance degradation when deployed in real-world scenarios involving acoustic domain shifts, such as back- ground noise and speaker accents. Test-time adaptation (TTA) has recently emerged as viable strategy to address such domain shifts at inference time without requir- ing access to source data or labels. However, existing TTA approaches, particularly those relying backpropagation, are memory-intensive, limiting their applicability in speech tasks and resource-constrained settings. Although packpropagation-fi methods offer improved efficiency, existing ones exhibit poor accuracy. This is because they are predominantly developed for vision tasks, which fundamen- tally differ from speech task formulations, noise characteristics, and model chitecture, posing unique transfcrability challenges. In this paper, we introduce E-BATS, the first Efficient BAckpropagations ffre TTA framework designed ex- plicitly for speech foundation models. E-BATS achieves a balance between adap- tation effectiveness and memory efficiency through three key components: (i) lightweight prompt adaptation for a forward-pass-based feature alignment, (ii) multi-scale loss capture both global (utterance-level) and local distribution shifts (token-level) and (iii) a test-time exponential moving average mechanism for stable adaptation across utterances. Experiments conducted four noisy speech datasets spanning sixteen acoustic conditions demonstrate consistent im- provements, with 4.1%-13.5% accuracy gains over backpropagation-Fiee baselines and 2.0x-6.4x GPU memory savings compared to backpropagation-bess -based meth- ods. By enabling scalable and robust adaptation under acoustic variability, this work paves the way for developing more efficient adaptation approaches for practi- cal speech processing systems in real-world environments. Code is available https: / thub. com/ wwt/Joaomeeaaaeeee JiahengDong/////// E-BATS
Introduction
Speech foundation models (SFM), large-scale pre-trained models that learn generalized represen- tations from vast amounts of unlabeled speech data, have shown strong performance for wide range of applications including voice assistants [1], transcription services [2], and accessibility tools [3]. These systems generally rely on the assumption that the training and test data follow the same distributions. In practice, this assumption is often violated, leading to significant performance drops under domain shifts caused by real-world acoustic variations such background noise, speaker accents, and microphone characteristics [4]. While domain adaptation [5, 6, 7, 8] and domain gen-
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

eralization [9, 10, 11, 12] have been extensively studied to address distributional shifts, they often require access labeled target domain data or continuous availability of raw source data. These requirements are seldom feasible in real-world scenarios following model deployment. Recently, Test-Time Adaptation (TTA) has emerged as an attractive solution, adapting pre-trained models to new domains during inference using only unlabeled test data.
Existing TTA methods can be broadly categorized into backpropagation-base (BP) and backpropagation-ffte (BP-free) approaches. While the former achieved state-of-the-art (SOTA) perfor- mance using entropy minimization [13, 14, 15, 16, 17, 18, 19] or leeuo--abeling techniques [20, 21], they have a large memory overhead, mainly due to gradient computation. Even when updates are limited to a small subset of model parameters, such as batch normalization layers [13, 14, 15] or early exits [22], these methods still require high GPU memory due to automatic differentiation frameworks. This significantly limits their practical use in continuous inference scenarios and on shourcce-consrainee devices. In contrast, BP-free TTA methods eliminate the need for gradient computation, making them more efficient and computationally lightweight. These methods either modify the model parameters during the forward pass [23, 24, 25, 26, 27] or learn new input prompt, a vector integrated with the partially processed input samples at an intermediate layer of the model [28].
Despite the promise of TTA, they are largely tailored to models that depend on Batch Normalization (BN), while SFMs use Layer Normalization (LN), limiting the applicability of BN-based TTA [16]- Addition- ally, SFMs include both CNN-based feature encoders extracting localized spectral features and transformer encoders processing global context. This is unlike models in other modalities like vision, which are either CNN-based (e.g.. ResNet (29]) or transformer- based (Vision Transformer [30]). Such architectural difference presents a fundamental challenge for BP- free feature adaptation. Furthermore, downstream tasks and noise characteristics differ significantly be- tween vision and speech tasks. As shown in Figure 1, vision models are commonly used for image classi- fication, one-to-one mapping task where noise typ- ically appears as spatial perturbations of pixels [31]. In contrast, speech recognition involves temporally varying noise across frames [32]. This requires more dynamic, multi-scale adaptation. Lastly, TTA methods often depend on large batch sizes for reliable adaptation, whereas TTA in speech tasks needs to process one utterance at time (batch size of 1) due to the high variability across speech utterances [16, 18]. Despite recent developments in TTA for SFMs [16, 19, 18, 17. 21]. they still heavily rely on the BP-based TTA methods with high computational overhead, and overlook the unique requirements of multi-scale adaptation.
(a)
(b)
VFMs
SFMs
SFMs
SFMs
Dog Car
A
Figure 1: The main difference between (a) Vision Foundation Models and (b) Speech Foundation Models (SFMs) is the sequential pipeline in SFMs that processes fixed-length frame of an utterance as an input and maps to distribution over [V] token classes.
mapping and must handle dynamic,
To address these challenges, we propose the first Efficient BAckpropagation-fre single-utterance TTA framework for SFMs, E-BATS, which achieves SOTA accuracy with memory efficiency. Here we focus on one of the most popular tasks on speech data - speech recognition. E-BATS consists of three novel modules: i) A lightweight, prompt-based tuning mechanism tailored for SFMs which directly adapts latent feature distribution using forward pass only; ii) a custom multi-scale loss function that captures both global (utterance-Ievel) and local (token-level) latent embeddings distribution shifts; and iii) test-time Exponential Moving Average (T-EMA) module that stabilizes prompt updates across dynamic utterances. The main contributions are threefold:
We introduce the first backpropagation-ffee TTA approach tailored explicitly for SFMs that achieves high accuracy and low memory consumption.
We propose a novel framework E-BATS, consisting of three novel modules to effectively address the multi-scale domain shifts and stable adaptation across the dynamic speech streams.
We validate E-BATS across four noisy datasets, sixteen acoustic environments, and two model architectures have demonstrated significant improvements in both accuracy and mem- ory, particularly 2x to 6.4x reduction in peak GPU memory usage over SOTA baselines.

---

2 Related Works
Inttory-nteesive BP TTA Methods. Traditional backpropagation-basse TTA methods are gen- erally memory-intensive, as they rely on gradient-based updates of model parameters, typically guided by entropy minimization parameters of the BN layers using entropy minimization. SAR [15] and EATA [14] are variants of TENT, which further filter out small portion of data samples that are unreliable and redundant. Although they only update a small portion of the overall parameters, the computation is still overhead due to backpropagation and the large batch size required for reliable adaptation [15]. More advanced methods, like COTTA [20], employ additional networks (e.g., teacher-student models) for adaptation, but the cost of significantly increased computational and memory overhead.
pseudo-labeling. TENT [13] was the first to adapt the affine
TTA for SFMs. A few recent TTA approaches have been tailored for SFMs, which typically extend memory-intensive BP TTA methods with speech-specific mechanisms [16. 18, 19J. SUTA [16] built upon TENT by updating CNN-based feature encoder layers alongside normalization parameters, incorporating techniques like temperature smoothing and Minimum Class Confusion loss. SGEM [18] and CEA [19] further improved upon this with advanced loss design for audio tasks, such as sequence-level entropy minimization incertainty-driverr frame prioritization. DSUTA [17] and AWMC [21] introduced additional subnetworks (c.g., fast-slow models and anchor-chaser-Lcader models) to enhance cross-utterance knowledge transfer. However, these methods still heavily rely on backpropagation, leading to significant memory overhead and scalability challenges as more layers updated beyond normalization.
BP-free TTA Methods. BP-free TTA methods offer an efficient alternative by updating the model solely via forward pass to achieve computational efficiency. These methods generally fall into three categories: (i) analytical adjustment of batch normalization statistics [23, 24, 25, 26), (ii) adaptation of the classifier using class prototypes [27] output probabilities [33], and (iii) optimization via evolutionary algorithms that circumvent gradicnt- based updates [28]. However, they typically offer lower adaptation accuracy compared to memory memory-intensive BP TTA methods [13, 20, 14, 15]. ppeechfoundatto Currently, BP-rree 77A for speech foundation models remains unexplored, primarily due to the unique challenges posed by sqquenceto-sequenee to-sequence learning, Angle-utterance adaptation rather than batches, and differences in model structures, particularly in normalization layers and feature encoders.
3 Methodology
3.1 Overview
We consider a covariate shift between the source and a target domain, such that the marginal distributions of speech differ, Pare (X) # Pegt(X), while the class prior (y)e Pigs (v) and the conditional distribution Psre X) Poge(y X) are preserved. We adapt an SFM Q pre-trained on source speech dataset Dare. improve transcription accuracy while maintaining the low peak memory usage on an unlabeled target speech dataset Digt- A target stream Digt consists of T utterances arriving sequentially, each is processed under an online, single-utterance setting with batch size of one. Each utterance X, is composed of variable number of frames N₂ and is represented as sequence of frame-level feature vectors, where each frame
represents short, fixed-duration segment of the audio signal: X, |+) ...,, N. 2/. E Rd
The model Ore is composed of two components, θ = h, where: (a) h is convolutional encoder that maps each input frame latent embedding: æj + 2 E Rd (b) g consists of a stack of transformer layers and a Connectionist Temporal Classification (CTC) classifier head, producing frame-level posterior distributions over CTC token classes v E V, where V consists of twenty-six alphabet token classes (a-z) and six special token classes (apostrophe, blank, etc.):
N, These posteriors are further decoded to produce each utterance's 's final g(Z)) { (Put 2 )
transcription Dt.
As shown in Figure 2, for test-time utterance X,, the model first employs Lightweight Prompt Adaptation (LPA) module, which directly modifies the CNN encoder fea- tures Z, by incorporating J learnable prompt vectors st.j. These prompts are sampled using the derivative-free Covariance Matrix Adaptation Evolution Strategy (CMA-ES), guided by multi-scale adaptation loss Teadgee The prompt that results in the lowest loss selected for adaptation. The loss comprises three components: entropy minimization (Len),, utterance-level (Lur), and token-level
System Overview.
3

---

①) LPA
Feature Encoders
(1) Mult-scale
(Isker-lered)
Loss Function
Tokenwise Feature
Lackes
Alignment
Adapt
interance
Encader
L Losdert
(m. C
A (roms dhe Come
candiatrs sampling
Promets
Utterance
Lyet
Alignment
Adapt
+ux (mx, 40
Sp.
Precolected Source
(Uttenance leve)
Stear Uteneeses
Single-utterance Backpropagation-fiee Adaptation
T-EMA utterances
Figure 2: Overall framework of E-BATS. For an utterance Xx: (i) Lightweight Prompt Adaptation (LPA):
latent features Z, are adapted using a set of J candidate prompts $t.j generated by CMA-ES in parallel, leading to J adapted representations. (ii) The adapted representations J are evaluated, and their corresponding prompts are ranked using multi-scale loss (entropy loss, utterance-level and token-level feature alignment). This ranking guides the iterative update of CMA- ES parameters over K iterations until the loss converges, at which point the best prompt is selected for adaptation. The CMA-ES parameters smoothed using T-EMA for next utterance adaption. (b) Test-time Exponential Moving Averuge (T-EMA): T-EMA stabilizes adaptation by smoothing the CMA-ES search trajectory across stream of utterances, facilitating robust prompt learning.
latent embeddings alignment (Lokke)). To promote stable adaptation across consccutive utterances, Test-time Exponential Moving Average (T-EMA) module (Figure 2(b)) incrementally updates the CMA-ES search distribution, enabling smoother evolution of the prompt vector $1,, over time.
3.2
Lightweight Prompt Adaptation (LPA)
To ensure memory efficiency, E-BATS adopts prompt tuning [34]. This technique introduces small set learnable parameters, called prompts, to guide the behavior of a pre-trained model on downstream task while keeping the original model weights fixed. While conventional prompt tuning approaches have primarily been developed for transformer-only architectures [30], typically by concatenating prompts with the model's inputs, propose novel LPA module (Figure 2a) designed specifically for SFMs that include convolutional components. The LPA integrates adaptive prompts directly into the convolutional layers, leveraging their effectiveness in extracting acoustic features [16]. Furthermore, rather than common strategy concatenating prompts with input features [28], our approach examines latent feature distribution shifts and leverages shifting patterns to guide adaptation.
Characterizing Distribution Shifts. To un-
derstand differences in speech distribution, first characterize the shift between the embed- ding vectors produced by source P(Zere) and target domain P(Znge ). We study the two most natural ways, covering both first and second or- der statistics: a) comparing shift in the centroid of the point clouds, and b) comparing the spread of the point clouds using their covariances. Inci- dentally, the popular distance measure between distributions, Fréchet Inception Distance, is linear combination of these two factors, justi- fying our choice. Figure 3 shows that, under various real-world noise and variability condi- tions, the shift in the mean accounts for up to 7.8x of the shift the covariance (we make sure they are in comparable scale, explained Ap- pendix A).
7.8x
7.2x
0.03
Coveriance shitt
3.8x
5.3x
0.01
2.0x
Syntheti nose
5 Simuateed Real background nose
Speaking variability
background rose
beckground nose
Figure 3: Comparing the source and target latent spaces across different acoustic conditions (same sample size for source and target domain within each condition). Blue and red bars indicate the mean and covariance shifts.

---

From this observation, we hypothesize that the shift between Dare and Dugt can be explained by simple geometric translation operation in the latent space; therefore. an adequate shift of target embeddings, Z5 would align the latent target vectors to the ones from the source, essentially mitigating the problem of domain shift. It important to note that, despite the potential for a translation operation, this remains non-trivial problem. In real-world scenarios, noise is not consistently introduced into clean samples, making it impossible to derive a simple solution based on an analyzable shifting vector. Therefore, adaptively learning the prompt becomes necessary.
Learning Prompt to Adapt. Following the above observation, we propose to add the embeddings Ze and the prompt vector 8, over all frames (Figure 2a) as:
2, Zi + 8, 1N:
where Z 2 .... 21 No
St Rd is the prompt vector for X, We exclusively optimize the prompt vector for each utterance while the rest of the model parameters remain frozen.
Prompt Optimization with CMA-ES. To identify the optimal prompt vector in a backpropagation- free manner, we adopt the CMA-ES [35]. For each test-time utterance X,, CMA-ES samples candidate prompt vectors $t,j from multivariate normal distribution (.),, which are parameterized by mean vector m, E R a covariance matrix C, E Rdxdd and a step size or E R>0- The performances of these candidates are then ranked based on loss function (explained in Section 3.3),
which informs the update of the distribution parameters m; (k) c((i)) , (a) for the next iteration k. The step size scales the covariance and thus controls the spread of candidate samples drawn from the distribution in the next iteration. The iteration continues until the loss convergences. The final prompt vector 8, is selected as the candidate that minimizes the loss for adaptation (Appendix D):
8t arg min Ladagt (X,, 8t.5). $L,
3.3 Multi-scale Loss Function
The loss function Ludapt that effectively guides the prompt optimization is proposed to integrate entropy minimization (Lens) with feature alignment to the source domain latent embedding distribu- tions at multiple scales, i.e., utterance-level (Lust) and tokenwise (Looken). We optimize a weighted average using Ladapt aeent + BL cLicken where a and /3 are hyperparameters, and is chosen algorithmically based on the confidence in the prediction.
Entropy Minimization Loss with Blank Token Exclusion (Lent). Entropy minimization aims to improve prediction confidence, which is further improved with blank token exclusion for SFMs. SFMs for speech recognition include a special "blank' token class 2, which is primarily designed in CTC to address the alignment mismatch between input frames and output labels [36] with different lengths. In recent studies, it often indicates frames where no alphabetic character can be assigned, helping to eliminate the need identify ambiguous sound boundaries [36] and highlighting silent periods [37]. In practice, predictions for a large proportion of the frames often belong to this blank class, which introduces class imbalance [16]. We address this by defining the entropy loss only considering the set of frames X, that do not predict blanks. Formally, we use Shanon's 's entropy as
Lent [X,] Exjex, (()()))).
Despite its effectiveness, minimizing only Lene has trivial solution of predicting the blank token class for each frame (as highlighted in Section 5.5). Thus, we introduce an utterance-level latent embeddings alignment loss guide optimization towards the source domain embeddings correctly.
Utterance-level Latent Embeddings Alignment (Lur)). Utterance-level latent embeddings align- ment aims to align the global latent embedding distributions between the source domain and the target domain to avoid trivial solutions. At each transformer encoder layer, I, we compute the squared Euclidean distance between the source-domain centroid and the target-domain centroid of utterance- level latent embeddings (Figure 2a(ii).. Each utterance-level embedding is obtained by averaging the embeddings across all frames within that utterance. Effectively, we compute and store the centroids, Mare using Dare and Oree and compare against the centroids, Higt, computed from the current target utterance. This gives us the following loss component: Lust 1 El llPigt p where L is the number of transformer layers in e. Note that Mere requires small storage size, in the order of L x d, and as this does not correspond to an individual source sample, it is privacy preserving.
5

---

Adaptive Confidence Tokenwise Latent Embeddings Alignment (Ltoken). The tokens within a single utterance may not comprehensively represent all possible tokens. Aligning the centroids of the source and target utterance-level embeddings could lead to bias towards the majority tokens. To address this, we minimize the distance hetween the source and target latent distributions corresponding to token classes, where target token classes are estimated using pseudo-labels. To prevent unreliable pseudo-labels from distorting adaptation, we introduce adaptive confidence coefficients for the token- level loss. Specifically, when overall distribution shifts substantial or entropy is high, this indicates less reliable posterior probabilities and a higher risk of inaccurate token pseudo-labels, In such cases, the confidence for token-level loss is reduced to minimize the impact of misleading token predictions. Conversely, if the shifts are minor or entropy is low, stronger alignment of token-level distributions can be applied with greater confidence.
Formally, we represent the mean nad and the standard deviation ur of the distribution for each token class E V in a d-dimensional space at transformer layer 1. Then we define the loss as the average distance between distributions as:
Ltoken
Σ 1=0 NEV L |V]
v.d. Migt
+
oigt
v.d ssc
12)
(3)
Note that the storage cost for part and and is small and in the asymptotic order of x L x 32 d. This is privacy-pree preneerirn as the distribution parameters are sample-agnostic
We further introduce an adaptive confidence E [0, 1] to adjust the trustworthiness of token-level predictions. We propose using the inverse of the combined loss = Teett + T.utt as the confidence- based coefficient, where high H means lower confidence. We define the normalized coefficient using
min-max normalization with predefined bounds H Hmmax and Cmar as: Graa H Hmax where e is a small constant to prevent division by zero. This adaptive confidence-aware scaling strengthens token-wise control via prediction reliability and domain shift.
H-H Humn
Optimizing through CMA-ES. The CMA-ES parameters are optimized using Ludapt- driving the prompt optimization. By iteratively minimizing Ludopt for candidate prompt vectors within each utterance, CMA-ES could be updated effectively generate prompts that robustly mitigate both global (utterance-level) and local (token-level) acoustic domain shifts.
3.4 T-EMA across Utterances
To stabilize adaptation across the utterance streams, we propose T-EMA that updates the CMA-ES parameters incrementally, ensuring a smoother and more consistent search space for the prompt (Figure 2b). It leverages the knowledge from past utterances to initialize each new search more robustly, thereby reducing overfitting and mitigating model drift. This serves as the first smoother adaptation strategy for BP-free TTA in SFMs.
CMA-ES statistics parameters are carried over between utterances using the following weighted average scheme. We introduce EMA statistics, c.g., Memas that are updated using a hyperparameter y E [0, 1) to weight the past and current statisties values. Such an update happens when all K iterations have finished for an utterance. For example, at the end of processing t-th utterance, the
mean of the distribution is updated as Mema = Mema + (1- 7) m; (K) where the mema is initialized with my when & = O. Then, m:41 is set to memo from the previous round for next utterance prompt learning. The other statistics, covariance and step size are updated in the same way.
4 Experiments
Datasets. We evaluate the proposed method on four datasets across sixteen acoustic conditions to assess its effectiveness under varying domain shifts. The test sets encompass three categories of acous- tic variability: synthetic noise, single-domain distributional shifts, and multi-domaii distributional shifts, reflecting the range of conditions encountered in real-world deployment. Following [16, 19], we introduce synthetic noise to the LibriSpeech fest-other split [38] with additive Gaussian noise with zero mean and varying standard deviations (a E (0.0, 0.005, 0.01, 0.015, 0.02)) to simulate covariate shifts. We use the CHiME-3 dataset [39] representing single domain shift in a sample, including four acoustic environments: bus, café, pedestrian area, and street junction. We further use CHiME-3-MM M-x that creates dynamic stream by concatenating CHiME-3 environments to emulate

---

Table Word Error Rate (WER) on various noisy conditions using Wav2Vec2ForCTC-BB Lower value means better adaptation performance. Bold represents the best performance for BP-free TTA, while underlined means the best for both BP-based and BP-free TTA.
Method BP-
Gaussian noise
CHiME3
CHiME3 (Mixed)
TED
Common Voice
free
0.0
0.005
0.01
0.015
0.02
Avg
(Single)
Source
8.6
13.9
24.4
39.5
54.5
28.2
34.2
34.2
13.2
36.8
TENT
8.5
14.0
24.1
39.2
54.3
28.0
34.1
34.1
13.1
36.8
EATA
X
14.1
18.1
27.0
37.9
51.3
29.7
33.1
39.9
14.1
61.3
SAR
8.4
13.6
22.9
36.0
49.9
26.2
33.6
34.7
13.0
38.2
COTTA
X
9.2
12.6
18.1
39.3
54.5
26.7
32.9
34.3
12.8
36.9
CEA
7.5
11.1
16.4
23.8
33.6
18.5
26.8
26.8
12.0
31.5
SGEM
X
7.3
10.9
16.4
23.8
33.9
18.5
27.2
27.1
11.9
31.2
AWMC
9.5
11.7
16.6
23.9
31.8
18.7
34.0
33.9
13.6
37.9
SUTA
X
7.3
10.9
16.5
24.1
34.1
18.6
26.8
26.8
11.9
31.5
CSUTA
13.1
17.5
24.5
31.4
37.0
24.7
26.5
32.6
15.6
135.0
DSUTA
9.0
11.7
16.1
21.1
24.1
16.4
24.0
24.1
12.7
36.1
T3A
10.0
15.9
26.8
42.7
58.6
30.8
35.9
35.8
14.6
38.8
LAME
9.1
15.0
26.0
42.4
58.2
30.1
36.0
36.0
14.0
38.8
FOA
8.7
13.9
22.7
33.3
45.3
24.8
31.7
31.1
13.3
38.2
Ours
7.7
10.5
14.8
19.9
25.3
15.6
24.0
24.3
12.5
30.6
non-stationary acoustic shifts [17]. Common Voice (CV) [40] introduces variability in speaker accents, recording devices, and environments and TEDLIUM-v2 (TED) [41] comprises oratory speech from TED talks with diverse accents, speaking styles, and syntactic structures.
Baseline Methods. We compare E-BATS against 13 SOTA TTA baselines. The BP methods
include general approaches of Episodic TENT [13], SAR [15], EATA [14], and CoTTA [20], well as speech-specifte methods: SUTA [16], CEA [19], SGEM [18], DSUTA [17], CSUTA [17], and AWMC [21]. The BP-free methods include LAME [33], T3A [27], and FOA [28]. Dataset and baseline details are provided in Appendix B.
Implementation Details. All TTA baselines are configured for per-utterance adaptation with batch size of 1. For E-BATS, we set the CMA-ES population size J 50. The loss function coefficients are a = 1.0 and 3 = 2.0. We use Haui 0.0, Hmax = 5.0 in calculating the confidence-weighted coefficient with Cmax 2.0 optimized over (1.0, 1.5, 2.0, 2.5, 3.0, 3.5, 4.0}. Evaluation is performed using two commonly used SFMs, Waw2Vec2ForCTCC-BB
[42] and HuBERTFOrCTCC Large [43]; both models fine-tuned on LibriSpeech and then adapted in experiments. The pre-collected statistics are sourced from clean LibriSpeech data samples. For T-EMA, we select y 0.9 for Wav2Vec2 and 0.8 for HuBERT after tuning over (0.7, 0.8, 0.9, 0.95, 0.99). We use Word Error Rate (WER) [44] as the evaluation metric, which measures the fraction of incorrectly predicted words the dataset. lower WER indicates better performance. We further conducted sensitivity analyses on key hyperparameters in Appendix C.3, including the CMA-ES population size (J), the number of iteration steps (N), the loss component weights (a, B). and the T-EMA decay factor (7), confirming that the chosen settings yield stable and robust performance across diverse configurations. All experiments are conducted on a single NVIDIA A100 GPU. Implementation details are provided in Appendix B.
Results and Discussion
5.1 Comparing accuracy to SOTA
Experiments using Wav2Vec2ForCTCC-B (Table 1) show that E-BATS consistently outperforms all BP-free TTA baselines across datasets, with WER reductions ranging from 0.8% to 20.0% over the strongest alternative. Its performance gains increase with noise severity, achieving at least 3.4% improvement at Æ 0.005 and 20.0% at = 0.02, highlighting robust adaptation under challenging conditions. T3A LAME degrade the source model, indicating that updating only the final classifier is insufficient. FOA, which also uses prompt tuning, performs better but remains less effective than E-BATS, likely due to difficulties adapting transformer layers for acoustic shifts (Section 5.5).
7

---

Table 2: WER on various noisy conditions using HuBERTForCTC-Large Lower better. Bold: best among BP-free; underlined: best overall.
Method BP-
Gaussian noise
CHiME3
CHiME3 (Mixed)
TED
Common Voice
free
0.0
0.005
0.01
0.015
0.02
Avg
(Single)
Source
4.2
5.0
6.4
9.0
12.8
7.5
16.5
16.5
9.1
21.4
TENT
x
4.2
4.9
6.3
8.8
12.5
7.3
16.4
16.4
9.0
27.5
EATA
7.4
8.4
9.5
11.3
13.7
10.1
16.2
18.9
9.7
34.4
SAR
X
4.0
4.7
6.3
8.6
12.2
7.2
16.4
17.3
9.0
21.7
COTTA
4.4
6.3
8.3
11.0
7.0
16.2
15.3
9.0
25.8
CEA
X
3.8
4.2
5.1
6.7
9.1
5.8
14.2
14.1
8.1
18.3
SGEM
3.7
5.3
5.3
6.9
9.3
6.1
14.2
14.2
8.3
18.4
AWMC
X
5.5
6.4
8.2
10.7
14.3
9.0
15.9
17.2
9.9
21.9
SUTA
3.8
4.2
5.1
6.8
9.2
5.8
14.2
14.2
8.2
18.4
CSUTA
X
6.0
6.8
7.9
9.5
11.9
8.4
14.7
16.2
10.2
90.0
DSUTA
4.6
5.0
6.0
7.1
8.8
6.3
13.3
13.5
8.7
27.4
T3A
14.4
15,8
18.9
24.2
30.2
20.7
27.9
32.3
22.5
46.2
LAME
4.5
5.3
6.9
9.8
13.9
8.1
17.5
17.5
9.7
22.6
FOA
4.5
5.3
6.8
9.2
12.9
7.7
16.4
16.7
9.3
22.8
Ours
4.3
4.9
5.9
7.5
9.5
6.4
14.0
14.0
9.3
20.1
Compared to BP-based methods, E-BATS achieves the lowest WER in 3 out of 5 datasets, with up to 30.7% relative improvement, and remains competitive. Methods such as EATA and CoTTA, which depend larger batch sizes rision-specifie strategies, perform poorly across the board. While TENT and SAR are more resilient with small batches, they still underperform relative to E-BATS, showing that adapting only normalization layers is inadequate. On the other hand, the performance limitations of SGEM, SUTA, and CEA stem from their utterance-level reset strategy. This prevents them from transferring the already learned knowledge for adapting further utterances as model weights are reinitialized every time. Notably, DSUTA, despite continuous adaptation, performs 5.5% worse than E-BATS Common Voice, the most diverse test condition. This suggests that frequent parameter updates may lead to catastrophic forgetting. In contrast, E-BATS updates only prompt vectors, preserving the pre-trained model and enabling effective, stable adaptation across varied domains. Additional results across different fine-grained noise conditions are in Appendix C.
5.2 Memory Efficiency
Beyond accuracy, E-BATS demonstrates substantial memory efficiency across all evaluated methods, as shown Figure Compared BP-based TTA methods, it reduces peak GPU memory usage by 1.5x to 5.9x. Specifically, relative to DSUTA, CEA, and SGEM, E-BATS achieves memory savings of 3.3x, 3.2x, and 2.8x, respectively, while outperforming them in WER. This efficiency attributed to lightweight prompt tuning and the T-EMA mechanism, which avoids wradient-based updates. Compared to BP-free baselines, E-BATS maintains comparable or lower memory usage while achieving lower WER, indicating a favorable balance between efficiency and performance.
5.3 Performance with Different Backbone Models
Avg GPU Usogo (MB)
When using HuBERTFFoCTCCLarre backbone, larger SFM than Way2Vec2ForCTC-BBaa E-BATS continues to outperform all BP-free and most BP-based TTA methods
6000
5000
4000
across datasets, as summarized in Table 2. On average, achieves 1.8% 17.1% lower WER than BP-free base- lines. While performance is comparable to ba-based meth-
33.0%
33.29
ods under certain conditions, E-BATS surpasses them in challenging scenarios such as CHiME-3 single-domain and mixed-domain settings. More notably, E-BATS offers sub- stantially better memory efficiency at larger model scale, with 2.4x to 6.8x lower GPU memory usage compared to BP-based approaches (detailed in Appendix C and Figure 6). As model size increases, memory usage grows for all TTA methods; however, E-BATS scales more gracefully, exhibit-
1000
LAME
Fox
SUTA
DSUTA
CSUTA
AWMMC
Ea7a
sal
CEA
COTTA
Mothoes
Figure 4: Balance between Average Peak GPU memory usage (bar) and av- erage WER (percentages %) for differ- ent TTA methods across all datasets.

---

Table Ablation study on three key components.
Prompt Adaptation
Loss Function (W/T-AA)
Loss Function (w/o T-EMA)
T-EMA Mechanism
Feat
Trans
WER
Lent
Lutt
Leoken
WER
Lent
Lute
Lioken
WER
T-EMA
Reset
WER
24.0
24.0
V
v
25.4
v
24.3
34.2
24.3
/
25.5
26.5
24.5
/
49.6
25.4
ing only moderate increase in memory demand. This makes it particularly suitable for on-device or resource-constrained environments where backpropagation is infeasible.
5.4 Memory Efficiency Across Utterance Lengths
12000
Figure 5 shows the peak GPU memory usage as utter-
Prsk GPU Merrory Usage Uhage (MB)
10000
SUTA
Ours
DSUTA
ance duration increases on the TED dataset using the
model. TED is selected due to its
8000
wide range of utterance lengths. We compare E-BATS against four lop-performing BP-based TTA methods. These base- lines exhibit rapidly growing memory consumption with in- creasing utterance duration, reaching 6-12 GB for 30-second clips. In contrast, E-BATS displays near-lineat memory profile, increasing from 1.1 GB at 1 second to just over 1.9 GB at 35 seconds, particularly suitable for deployment scenarios with strict varying memory constraints.
6000
4000
2000
0
20
30
Utforancc Duratin (scconds)
Figure 5: Peak GPU memory of TTA on TED as audio duration increases.
5.5 Ablation Study
We investigate the effectiveness of three key components in E-BATS.
Prompt Adaptation. We compare our proposed method, which injects prompts into Z, to adapt the CNN latent feature representations directly, with a variant that concatenates prompts with Ze at the input to the transformer encoder, following conventional prompt-tuning approaches. As shown in Table 3, adapting within the CNN-based feature encoder yields better performance (24.0 vs. 34.2 WER) on CHiME3 (single). This advantage comes from CNNs' ability to capture localized spectral features (e.g., pitch, formants), which are crucial for handling acoustic domain shifts. In contrast, transformer encoders focus global contextual dependencies (e.g., sentence-level semantics), making them less effective at modeling fine-grained acoustic variations under domain mismatch.
Loss Function Components. We evaluated each component of the loss function under two settings: (1) CHiME-3-Single with (w/) T-EMA, representing stable and consistent distribution shift, and (2) CHiME-3-Mix without (w/o) T-EMA, reflecting more diverse shifts. For the single-domain shift with T-EMA, we observed that each loss component contributed to overall performance (24.0 WER), with the token-level loss providing additional improvements (from 24.3). In contrast, for the mixed distributional shifts, adaptation relied heavily utterance-level alignment (from 49.6 to 25.5), as expected due to the increased shifts. Compared with only using Teets combining T.ues effectively prevents trivial solutions caused by entropy minimization of predicting all frames the blank token class or collapsing into a single character. This is further explained and analyzed in Appendix C.4. Moreover, the adaptive weighting mechanism of the token-level loss ensured its reduced confidence. facillitating more reliable adaptation under this setting (25.4). These findings not only underscore the importance of all loss components across different scenarios but also highlight the critical role of confidence-bast based adaptive weighting, allowing the loss to emphasize the most reliable signals under varying conditions.
T-EMA and Reset Strategy. We evaluate the effectiveness of the T-EMA under dynamic domain shifts (CHiME3-Mix) by comparing it against two alternatives: (i) a reset variant that reinitializes CMA-ES parameters at the start of each utterance, and (ii) variant that performs continuous adaptation without any resetting mechanism. As shown in Table 3, T-EMA consistently achieves lower WER than both variants. The rexel variant yields the worst performance (26.5), indicating that discarding adaptation history prevents the accumulation of knowledge. Conversely, omitting reset entirely leads to sub-optimal results (25.4), suggesting that preserving historical information is important but needs to be regulated to avoid overfitting. T-EMA provides principled balance

---

between stability and adaptability across utterances. The effectiveness of T-EMA is further analyzed with an increasing number of target domain samples in Appendix C.5.
6 Conclusions and Discussion
Conclusions. In this paper, we propose E- E-BATS, the first backpropagation- -feee free rest-time adaptation method for Speech Foundation Models that effectively balances adaptation accuracy and memory efficiency. E-BATS introduces a lightweight prompt adaptation module that direcrly adapts CNN- based feature encoders to mitigate acoustic domain shifts. A novel multi-scale loss function combining entropy minimization with utterance-level and token- wise feature alignment ensures fine-grained control over speech feature adaptation. Additionally, the test-time Exponential Moving Average mechanism stabilizes continuous adaptation in dynamic speech streams. Experimental results across four noisy datasets and diverse acoustic conditions demonstrate its superior performance, particularly in memory efficiency as the model size increases significantly.
Limitations. Although E-BATS is more theoretically efficient than other baseline methods through computation complexity comparision, which reported in Appendix C.9, the iterative CMA-ES optimization introduces additional adaptation latency in practical environments. Specifically, the current implementation of CMA-ES does not fully exploit GPU parallelization, leading to sequential computation steps per utterance. While this latency is acceptable for scenarios without strict real-time requirements, it might pose challenges for latency- applications.
Acknowledgments and Disclosure of Funding
This research was supported by The University of Melbourne's Research Computing Services and the Petascale Campus Initiative.
References
[1] Veton Kepuska and Gamal Bohouta. Next-generation of virtual personal assistants (microsoft cortana, apple siri, amazon alexa and google home). In 2018 IEEE 8th annual computing and communication workshop and conference (CCWC), pages 99-103, IEEE, 2018.
[2] Yashesh Gaur, Walter S Lasecki, Florian Metze, and Jeffrey P Bigham. The effects of auto- matic speech recognition quality on human transcription latency. In Proceedings of the 13th International Web for All Conference, pages 1-8, 2016.
[3] Oscar Saz Torralba, William Ricardo Rodríguez Dueñas, and Eduardo Lleida Solano. Devel- opment of based tools for accessibility to computer services. Computación y Sistteas, 15(1):7-15,200
[4] Jinyu Li, Li Deng, Yifan Gong, and Reinhold Haeb-Umbach. An overview of noise-robust automatic speech recognition. JEEE/ACM Transactions on Audio, Speech, and Language Processing, 22(4):745-777, 2014.
[5] Ashly Ajith and Gopakumar. Domain adaptation: A survey. In Computer Vision and Machine Intelligence: Proceedings of CVMI 2022, pages 591-602. Springer, 2023.
[6] Zijian Li, Ruichu Cai, Guangyi Chen, Boyang Sun, Zhifeng Hao, and Kun Zhang, Subspace identification for multi-source domain adaptation. Advances in Neural Information Processing Systems, 36:34504-34518 2023.
[7] Aibek Alanov, Vadim Titov, and Dmitry P Vetrov. Hyperdomainnet: Universal domain adapta- tion for generative adversarial networks. Advances in Neural Information Processing Systems, 35:29414 29426, 2022.
[8] Mingsheng Long, Zhangjie Cao, Jianmin Wang, and Michael I Jordan. Conditional ad adversarial domain adaptation. Advances in neural information processing systems, 31, 2018.
[9] Kaiyang Zhou, Ziwei Liu, Yu Qiao, Tao Xiang, and Chen Change Loy. Domain generalization: survey. IEEE transactions on pattern analysis and machine intelligence, 45(4):4396 4415, 2022.
10

---

[10] Yu Ding, Lei Wang. Bin Liang, Shuming Liang, Yang Wang, and Fang Chen. Domain general- ization by learning and removing domain- features. Advances in Neural Information Processing Systems, 35:24226- 242239, 2022.
[11] Junbum Cha, Sanghyuk Chun, Kyungjae Lee, Han-Cheol Cho, Seunghyun Park, Yunsung Lee, and Sungrae Park. Swad: Domain generalization by seeking flat minima. Advances in Neural Information Processing Systems, 34:22405-22416 2021.
[12] Shanshan Zhao, Mingming Gong, Tongliang Liu, Huan Fu, and Dacheng Tao. Domain gen- eralization via entropy regularization. Advances in neural information processing systems, 33:16096-16107, 2020.
[13] Dequan Wang, Evan Shelhamer, Shaoteng Liu, Bruno Olshausen, and Trevor Darrell. Tent: Fully test-time adaptation by entropy minimization. arXiv preprint arXiv:2006.10726, 2020.
[14] Shuaicheng Niu, Jiaxiang Wu, Yifan Zhang, Yaofo Chen, Shijian Zheng, Peilin Zhao, and Mingkui Tan. Efficient test-time model adaptation without forgetting. In International confer- ence on machine learning, pages 16888-16905. PMLR, 2022.
[15] Shuaicheng Niu, Jiaxiang Wu, Yifan Zhang, Zhiquan Wen, Yaofo Chen, Peilin Zhao, and Mingkui Tan. Towards stable test-time adaptation in dynamic wild world, arXiv preprint arXiv:2302. 12400, 2023.
[16] Guan-Ting Lin, Shang-Wer Wen Li, and Hung-yi Lee. Listen, adapt, better wer: Source- free single-utterance test-time adaptation for automatic speech recognition. arXiv preprint 2.ti::2203.142222. 2022.
[17] Guan-Ting Lin, Wei-Ping Huang, and Hung-yi Lee. Continual test-time adaptation for end-to- end speech recognition noisy speech. arXiv preprint arXiv:2406. 1064, 2024.
[18] Changhun Kim, Joonhyung Park, Hajin Shim, and Eunho Yang. Sgem: Test-time adaptation for automatic speech recognition via sequential-level generalized entropy minimization. arXiv preprint arXiv:2306.0199 019811 2023.
[19] Hongfu Liu, Hengguan Huang, and Ye Wang. Advancing test-time adaptation in wild acoustic test settings, arXiv preprint arXiv:2310. 0.505, 2023.
[20] Q. Wang, O. Fink, L. Van Gool, and et al. Continual test-time domain adaptation. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 7201-7211, 2022.
[21] Jac-Hong Lec, Do-Hee Kim, and Joon-Hyuk Chang. Awmc: Online test-time adaptation without mode collapse for continual adaptation. 2023 IEEE Automatic Speech Recognition and Understanding Workshop (ASRU), pages 1 1-8. IEEE, 2023.
[22] Hong Jia, Young Kwon, Alessio Orsino, Ting Dang, Domenico Talia, and Cecilia Mascolo. Tinytta: Efficient test-time adaptation via early-exit ensembles on edge devices. Advances in Neural Information Processing Systems, 37:43274-43299, 2024.
[23] M Jehanzeb Mirza, Jakub Micorek, Horst Possegger, and Horst Bischof. The norm must go on: Dynamic unsupervised domain adaptation by normalization. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 14765-14775, 2022.
[24] Bowen Zhao, Chen Chen, and Shu-Tar Xia. Delta: degradation-free fully test-time adaptation. arXiv preprint arXiv:2301.13018 2023.
[25] Steffen Schneider, Evgenia Rusak, Luisa Eck, Oliver Bringmann, Wieland Brendel, and Matthias Bethge. Improving robustness against common corruptions by covariate shift adaptation. Advances in neural information processing systems, 33:1 1539-11551111 2020.
[26] Zixian Su, Jingwei Guo, Kai Yao, Xi Yang, Qiufeng Wang, and Kaizhu Huang. Unraveling batch normalization for realistic test-time adaptation. Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pages 15144411444, 2024.
11

---

[27] Yusuke Iwasawa and Yutaka Matsuo. Test-time classifier adjustment module for model domain generalization. Advances in Neural Information Processing Systems, 34:2427-2440, 2021.
[28] Shuaicheng Niu, Chunyan Miao, Guohao Chen, Pengcheng Wu, and Peilin Zhao. Test-time model adaptation with only forward passes. arXiv preprint arXiv:2404. 001,:200.000650, 2024.
[29] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770-778, 2016.
[30] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010.11929 2020.
[31] Dan Hendrycks and Thomas Dietterich. Benchmarking neural network robustness to common corruptions and perturbations. arXiv preprint arXiv: 12261, 2019.
[32] Jinyu Li, Li Deng, Yifan Gong, and Reinhold Haeb-Umbach, An overview of noise-robust automatic speech recognition. IEEE/ACM Transactions on Audio, Speech, and Language Processing, 22(4):745-777, 2014.
[33] Malik Boudiaf, Romain Mueller, Ismail Ben Ayed, and Luca Bertinetto. Parameter- free online test-time adaptation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pages 8344-8353, 2022.
[34] Brian Lester, Rami AI-Rfou, and Noah Constant. The power of scale for parameter-efficient prompt tuning. arXiv preprint (XXv::214400661 2021.
[35] Nikolaus Hansen. The cma evolution strategy: A tutorial. arXiv preprint arXiv: arXiv: 1604.0072,
2016.
[36] Alex Graves, Santiago Fernández, Faustino Gomez, and Jürgen Schmidhuber. Connectionist temporal classification: labelling unsegmented sequence data with recurrent neural networks. In Proceedings of the 23rd international conference on Machine learning. pages 369-376, 2006.
[37] Mostafa Shahin, Julien Epps, and Beena Ahmed. Phonological level wav2vec2- mispro- nunciation detection and diagnosis method. Speech Communication, page 103249, 2025.
[38] Vassil Panayotov, Guoguo Chen, Daniel Povey, and Sanjeev Khudanpur. Librispeech: an asr corpus based on public domain audio books. In 2015 IEEE international conference acoustics, speech and signal processing (ICASSP), pages 5206-5210. IEEE, 2015.
[39] Jon Barker, Ricard Marxer, Emmanuel Vincent, and Shinji Watanabe. The third 'chime's sseee's separation and recognition challenge: Analysis and outcomes. Computer Speech & Language, 46:605-626, 2017.
[40] Rosana Ardila, Megan Branson, Kelly Davis, Michael Henretty, Michael Kohler, Josh Meyer, Reuben Morais, Lindsay Saunders, Francis M Tyers, and Gregor Weber. Common voice: A 2019.
massively-multilingee multilingual speech corpus. arXiv preprint
[41] François Hernandez, Vincent Nguyen, Sahar Ghannay, Natalia Tomashenko, and Yannick Esteve. Ted-lium 3: Twice as much data and corpus repartition for experiments on speaker adaptation. In Speech and Computer: 20th International Conference, SPECOM 2018, Leipzig, Germany, September 18-22, 2018, Proceedings 20, pages 198-208. Springer, 2018
[42] Alexci Baevski, Yuhao Zhou, AbdeIrahman Mohamed, and Michael Auli. wav2vec 2.0:
A framework for self-supervised learning of speech representations. Advances in neural information processing systems, 33; 13:4914--2,460 2020.
[43] Wei-Ning Hsu, Benjamin Bolte, Yao-Hung Hubert Tsai, Kushal Lakhotia, Ruslan Salakhutdinov, and Abdelrahman Mohamed. Hubert: Self-supervised speech representation learning by masked prediction of hidden units. IEEE/ACM transactions on audio, speech, and language processing, 29:3451-3460, 2021.
12

---

[44] David S Pallett. Performance assessment of automatic speech recognizers. Journal of Research of the National Bureau of Standards, 90(5):371, 1985.
13

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: See Abstract and Section (Introduction, third and fourth paragraph). Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results
be expected to generalize other settings. It is fine to include aspirational goals as motivation as long as is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations the work performed by the authors? Answer: [Yes]
Justification:
See Section 6 (Conclusion and Discussion, The limitations paragraph.)
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well- sell-specffication, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-texx system might not be used reliably provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?
Answer: [NA]
Justification: We include no formal theorems or lemmas in this work. All claims are validated through experimental results.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated referenced in the statement of any theorems.
14

---

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Justification: See Section 3 (Methodology) and Appendix D (Detailed algorithms). Our code is publicly available at: btttps://gin http:::://gttuu..oom/ com/ wwtp:///o/oomaiioaiiiaaa wiwahennnngg/-BBAta
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how replicate the results, access a hosted model (e.g.. in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm. (b) the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibilityy the case of closed-source models, may be that access to the model limited in some way (c.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification: Our code publicly available at: Code is available at: https://github. com/ Ji awwnaDongnE---Baa EB-ons
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips. cc/ Aublic/guides htt////oeemmmmiiiiooooooooooo
for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" is an acceptable answer. Papers cannot be rejected simply for not
15

---

reproducible, they including code, unless this is central to the contribution (e.g., for new open-souree benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https:
/ //nips. cc/ cc/public//peecccccccccc public/guidess /eggddd///o/duuabiiiii ieeiiisssnnoolol onPolicy) for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments should state which ones are omitted from the script and why. At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing much information possible in supplemental material (appended the paper) recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: See Section 4 (Experiment), Appendix (Details of datasets and baseline settings) and Appendix D (Detailed algorithms).
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core the paper to a level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment statistical significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [Yes]
Justification: See section (Methology: FID figure), Appendix A (Detailed FID scores), and Appendix C.2 (Ablation study with different seeds). Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi-
dence intervals, statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall
with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula,
call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
should be clear whether the error bar is the standard deviation the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
Question: For each experiment, does the paper provide sufficient information on com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Answer: [Yes]
16

---

Justification: See Section 4 (Experiment: Implementation Details) and Figure Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).
9. Code ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https: : /nwuriiurppp.ee cc htce//ullcc/Ethicseeeee ines?
Answer: [Yes]
Justification: See Appendix E. Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (c.g., there a special consid-
eration due to laws regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed? Answer: [Yes]
Justification: See Appendix E (The second paragraph and the last sentence in the first paragraph)
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA No, they should explain why their work has no societal
impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation. generating fake profiles, surveillance), fairness considerations (e.g., deploymeett of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, scraped datasets)?
Answer: [NA]
Justification: Our work does not release produce any new data generative model artifacts-rather
Guidelines:
The answer NA means that the paper poses such risks.
17

---

Released models that have high risk for misuse or dual-use use should be released with necessary safeguards to allow for controlled use of the model. for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: All the data and models are correctly cited and follow the license and terms of use explicitly. See Appendix E and Appendix B.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include a URL.
The name of the license (e.g.. CC-BY 4.0) should be included for each asset. For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
If assets released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi wwwodee.hoe.... com datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
this information is available online, the authors are encouraged to reach out to the asset's s creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: See Section (Methodology), Section (Experiment details) and Section (Limitation). Our code will be publicly available upon acceptance (See Abstract, last sentence).
Guidelines:
The answer NA means that the paper does release new assets. Researchers should communicate the details of the dataset/code/moodl as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification: No crowdsourcing new human- data collection conducted in this work.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
18

---

Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, curation,
or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/teview based on the requirements of your country institution) were obtained?
Answer: [NA]
Justification: No IRB approval was required for this study because we did not collect any new human-subject data.
Guidelines:
The answer NA means that the paper does involve crowdsourcing nor research with
human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent)
may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere the NeurIPS Code of Ethics and the guidelines for their institution. For initial submissions,
not include any information that would break anonymity (if applicable), such the institution conducting the review.
16. Declaration of LLM usage
Question: Does the paper describe the usage of LLMs if it is important, original, or non-standard component of the core methods this research? Note that the LLM is used only for writing, editing, formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration is not required.
Answer: [NA]
Justification: All text, figures, and analyses related with core methods were created solely by the authors.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, or non-standard components.
Please refer to LLM policy (https: pettta://neurriiin http://iis..ce//oeeeeeeeeeeeeeee wwtp://oooeeeenee/225//////////e for what should or should not be described.
19

---

A FID score of detailed domain shifts
To accurately quantify distributional shifts between the source and target domains, employ the Fréchet Inception Distance (FID), a metric effective in capturing differences in both mean and covariance statistics of feature embeddings. Specifically, given the CNN-extracted latent embeddings 2 |2] 22 ..... 2, [ for each utterance X,. first average these embeddings across all frames to obtain the utterance-level representation N; ΣN of Rd due the variable length of each utterance.
Considering the utterance-level embeddings from the source dataset sre and the target dataset Digr- we calculate their empirical mean vectors and covariance matrices as:
N
N₂
Σ
Uurc
2
Pigt
Derel
Daer
N₂
XiEDen
i=1
XxEDgy
N₂
1
Due
28
Mare
2/
Mure
Derel
N
N
i=1
N2
Eng
Σ
Mug
Σ
2
Mug
+ tel,
Digl
N
Ne
XiEDai
where is added to ensure numerical stability.
The final FID between the source and target domains computed as:
FID(Overall) | Magall2 112 + Tr Exre + Engs 2( Engs)
where the first part the Mean shift, and the second part is the Covariance shift. Mean Covariance Meon shift
shift Ratio will be calculated as Covariance shiff"
Table 4: FID Scores and Mean Shift Ratios for Various Noisy Conditions
Noisy Condition
Mean shift
Covariance shift
Mean/Covariance shift Ratio(x)
Gaussian noise
0.031
0.004
7.8
Single-domain environment noise (simulated)
0.016
0,003
5.3
Single-domain environment noise (real)
0.029
0,004
7.3
Mixed-domain environment noise
0.019
0,005
3.8
Speaking Style variability
0.004
0.002
2.0
Table 5: FID Scores and Mean Shift Ratios for detailed Gaussian noise and single- domain environment noise Noisy Condition
Mean shift
Covaraince shift
Mean/Covariance shift Ratio (x)
Gaussian noise 0.005
0.017
0.003
5.7
Gaussian noise 0.01
0.028
0.003
9.3
Gaussian noise 0.015
0.036
0.004
9,0
Gaussian noise 0.02
0.042
0.004
10.5
Cafe-real
0.031
0.003
10.3
Bus-real
0.031
0.003
10.3
Pedestain-real
0.031
0.004
7,8
Street-real
0.022
0.004
5.5
Cafe-simu
0.021
0.003
7.0
Bus-simu
0.011
0.003
3.7
Pedestain-sime
0.017
0.004
4.3
Strect-simu
0.015
0,003
5.0
20

---

B Experiments
B.1 Datasets
Gaussian Noise Data. Following [16, 19]. we corrupt the LibriSpeech [38] test-other split with zero-mean additive Gaussian noise to provide covariate shifts different ampli- tudes (a E {0.0, 0.005, 0.01, 0.015, 0.02)). This setting provides a controlled evaluation of robustness incremental noise severity. Single-Doman Background Noise Data.
CHiME-3-single: It is noisy version of WSJ corpus with artificial and real-world environmental noises at 16 kHz. We utilize the official simulated and real enhanced eval- uation sets from CHiME3 [39], which cover four challenging acoustic environments: bus, cafe, pedestrian area, and street junction. This setting simulates domain-specifie, scene-consistent background conditions.
Multi-Domain and Wild Real-World Data.
CHiME-3-Mix: All CHiME-3 scenarios are combined into dynamic stream to simulate continuously shifting acoustic environments, similar to setups in continual test-time adaptation [17].
Common Voice (CV) [40]: A crowdsourced project where volunteers contribute by reading Wikipedia sentences to produce 48 kHz audio samples. To align with the source ASR models' training conditions, we resampled these recordings to 16 kHz. The test set from the en-June-22nd-2000 release was used to evaluate robustness against different speaking styles, accent variability, and crowd-sourced audio quality issues, TEDLIUM-Y2 (TED): Consists of oratory speech from TED conference videos with high quality stored at 16 kHz. We use the official test for experiments, which introduces mismatches recording quality and presentation style speech, diverging from the read speech in LibriSpeech or Common Voice, and thus providing natural domain shift. Following [16]. transcripts across all datasets are converted to uppercase and stripped of punctuation, retaining only apostrophes.
B.2 Baseline methods
The baseline methods include both BP TTA and BP-free TTAs.
BP TTAS:
TENT [13]. A fully test-time adaptation method that minimizes entropy by updating BatchNorm affine parameters online. We use it in episodic version since the batch size is small.
SAR [15]. A sharpness-aware and reliable entropy minimization method that selectively filters samples with large gradients and encourages the model weights to converge to a flat minimum, improving stability under wild domain shifts.
EATA [14]. An efficient TTA framework that selectively adapts on samples with lower uncertainty to reduce gradient noise and also mitigates catastrophic forgetting through Fisher regularizer.
CoTTA [20]. A continuous TTA method that maintains teacher-student adaptation strategy with stochastically restoring certain model parameters.
SUTA [16]. A single-utterance test-time adaptation method based entropy minimization and minimum class confusion, adapted for CTC-based ASR.
CSUTA. A continous version of SUTA with iteration step of 1, which is examined one baseline in the work of [17].
DSUTA [17]. A dynamic variant of SUTA that adaptively resets or retains model updates based on domain shift detection with fast-slow adaptation strategy.
CEA [19]. Confidence-enhancee frame-level adaptation with short-term consistency regu- larization, proposed for wild acoustic test conditions.
SGEM [18]. A method leveraging beam-search logits and generalized entropy minimization for autoregressive ASR adaptation at sequence-leved granularity.
AWMC [21]. pseudo-labeling-ben basedd continual TTA algorithm for ASR that employs an anchor model, leader model and chaser model to achive stabel continous adaptation wihout forgetting.
21

---

BP-free TTAs:
LAME [33]. training-free approach that corrects model outputs probabilities by estimating distribution drift in feature space.
T3A [27]. A TTA technique that adjusts classifier via prottttpss without requiring backward passes.
FOA [28]. forward-only approach that optimizing learnable prompts with activation shifts to avoid forgetting issue and trivial solutions.
B.3 Baseline methods hyperparameter setting
The detailed baseline settings and the hyperparameter tuning are presented for both BP TTA and Br-free TTA.
BP TTA baslines The hyperparameter settings for TTA methods organized follows: For Speech Foundation Models (SFMs), we follow the configurations specified in the original papers for SUTA [16], DSUTA [17], CEA [19], and SGEM [18]. For CSUTA [17] and AWMC [21], which do not have released code, we adhere to the hyperparameters outlined in their official papers and the implementations from [17]. Additionally, consistency.
set the model to evaluation mode maintain
For Visual Foundation Models (VFMs), to adapt the methods for SFMs with batch size of 1 (BS=1), we follow the guidelines presented in [16, 19, 28]. All optimizers are configured to use the Adam optimizer with the same learning rate the TTA baseline methods for SFMs. Episodic methods are set with 10 iteration steps. while continuous methods use single step. Minor adjustments are made for specific methods: for EATA [14], use e_margin 0.4 x In(32) and d marrrgnnn 1.0; = 0.3; and for CoTTA [20], the
for SAR [15), e_margin = 0.4 ln(3) and augmentation threshold is set to 0.2, with augmentation limited to adding Gaussian noise.
BP-free TTA baselines We follow the original hyperparameter settings for LAME [33] and T3A [27] as specified in their official papers. For FOA, the parameters are set as follows: a 0.1 (CMA-ES), a 0.05, and 7 0.1.
In our approach, adaptation for each utterance terminated early if the best fitness across iterations does not improve by at least 0.001 for three consecutive steps.
B.4 Backbone models
Wav2Vec2ForCTC-BBet [42] model employs a 12-layer Transformer encoder with CNN- based feature extractor, representing lightweight ASR architectures optimized for fast inference. HuBERTForCTC-Larg [43] model features a deeper 24-layer Transformer stack with a similar CNN front-end, offering more powerful and robust ASR framework.
C Detailed Experiment results
C.1 CHiME3
The detailed performance on CHiME3 dataset using Wav2Vec2ForCTC--B and HuBERTForCTC. large are shown in Tables 6 and 7 respectively. It presents the performance comparison across four different acoustic environments, including cafe, bus, pedestrian, and street. For each acoustic condition, we also include the simulated and real-world noise conditions. The performance also demonstrated the superior performance of our approach over BP-free TTA and most of BP-based TTA.
22

---

Table Comparison TTA methods across CHiME3-single (cafe, bus, pedestrian, street) by using Way2eee2FoCCTC--ee model. WER in bold is the best performance within BP-free TTA methods, and the underlined WER the best within both and BP-free TTA methods.
Cafe
Bus
Pedestrian
Street
Average
Method
Simu
Real
Simu
Real
Simu
Real
Simu
Real
Simu
Real
Overall
Source
20.1
58.7
14.6
56.2
17.9
55.5
18.7
32.2
17.8
50.7
34.2
BP adaptation
TENT (episodic)
20.0
58.4
14.5
55.9
17.8
55.2
18.7
32.0
17.8
50.4
34.1
EATA
20.0
55.3
14.9
54.1
18.3
51.6
18.9
31.4
18.0
48.1
33.1
SAR
19.6
56.2
14.4
56.5
17.8
52.8
18.7
33.0
17.6
49.6
33.6
COTTA
19.2
58.6
14.2
49.2
16.8
55.4
17.8
32.1
17.0
48.8
16.2
CEA
17.3
45.8
13.1
41.8
16.1
38.0
17.1
25.2
15.9
37.7
26.8
SGEM
17.4
45.5
13.0
43.1
15.8
38.8
17.1
25.6
15.8
38.5
27.2
AWMC
19.5
62.9
14.5
54.6
17.4
39.0
18.5
31.9
17.5
50.6
34.0
SUTA
17.1
45.1
13.0
42.5
16.2
38.1
17.5
25.3
16.0
37.8
26.8
CSUTA (1 step)
17.7
40.3
14.8
41.1
16.8
36.4
18.7
26.0
17
36.0
26.5
DSUTA
16.4
36.3
13.0
39.6
15.2
32.8
15.6
22.4
15.1
32.8
24.0
BP-free adaptation
T3A
20.9
61.3
15.3
59.0
18.5
58.4
19.6
33.6
18.6
53.1
35,9
LAME
20.9
61.6
15.2
58.9
18.6
58.9
19.8
33.9
18.6
53.3
36.0
FOA
19.9
52.5
14.6
50.5
18.0
48.1
18.6
31.0
17.8
45.5
31.7
Ours
16.1
37.9
13.1
37.4
15.1
33.1
15.4
24.0
14.9
33.1
24.0
Table 7: Comparison TTA methods across CHiME3-single (cafe, bus, pedestrian, street) by using HuBERTForCTC-lurge WER in bold is the best performance within BP-free TTA methods, and the underlined WER is the best within both BP and BP-free TTA methods.
Cafe
Bus
Pedestrian
Street
Average
Method
Simu
Real
Simu
Real
Simu
Real
Simu
Real
Simu
Real
Overall
Source
9.2
27.9
8.5
26.2
9.2
24.3
10.1
16.6
9.3
23.8
16.6
BP adaptation
TENT (episodic)
9.2
27.5
8.5
25.8
9.2
24.0
10.0
16.5
9.2
23.5
16.4
EATA
9.2
27.0
8.7
25.8
9.2
23.2
9.9
16.4
9.3
23.1
16.2
SAR
28.0
8.4
26.1
9.1
23.2
9.9
17.3
9.1
23.7
16.4
COTTA
9.3
26.8
8.5
25.5
9.1
23.9
9.9
16.4
9.2
23.2
16.2
CEA
8.6
22.5
8.1
22.0
8.3
19.6
9.2
14.7
8.6
19.7
14.2
SGEM
8.7
22.5
8.1
21.8
8.5
19.9
9.5
14.7
8.6
19.7
14.2
AWMC
9.5
25.5
8.9
23.9
9.4
22.5
10.4
16.5
9.6
22.1
15.9
SUTA
8.6
22.9
8.1
22.2
8.4
19.6
9.4
14.6
8.6
14.2
CSUTA (1 step)
9.3
22.8
8.9
22.2
9.5
19.7
9.9
14.7
9.4
19.9
14.7
DSUTA
8.7
20.3
8.3
20.0
8.6
17.5
9.3
13.9
8.7
17.9
13.3
BP-free adaptation
T3A
18.2
42.1
18.2
42.1
17.3
37.5
18.2
29.0
18.0
37.7
27.9
LAME
9.7
30.2
8.9
27.9
9.6
25.4
10.6
17.8
9.7
25.3
17.5
FOA
9.6
26.9
8.6
26.2
9.3
23.4
10.4
16.4
9.5
23.2
16.4
Ours
9.2
20.7
8.4
21.3
9.2
18.9
10.0
14.2
9.2
18.8
14.0
23

---

C.2
Ablation study with multiple seeds
We repeat each ablation experiment using three different random seeds in Figure ensure that our results are robust and not the artifact of any single initialization.
Table 8: Ablation study with meansstd WER (%) over 3 seeds
Prompt Adaptation
Loss Function (w/ T-EMA)
Loss Function (w/o T-EMA)
T-EMA Mechanism
Feat
Trans
WER
Lend Lutts
Leaken
WER
Lunt Lust
Linkon
WER
T-EMA
Reset
WER
24.0=0.0
24.0#0.0
25.4=0.0
24.3+0.0
34. 140.1
V
V
24.2+0.1
V
25.5+0.1
26.5+0.0
24.620.1
64.3+11.5
25.4+0.0
C,3 Sensitive Analysis
C.3.1 Sensitivity of TT-EMA Decay Parameter
Table below presents our analysis on the EMA parameter 7s where we evaluated performance across E {0.50, 0.60, 0.70, 0.80, 0.90, 0.95, 0.99) under the caf-real condition. We observed clear U-shaped WER curve: performance improves up to 7 = 0.90 (best WER 37.9%), and then degrades increases further. This behavior aligns with our expectation and is also observed in other datasets that smaller y values lead unstable, overly reactive updates, while larger values result in over-smoothing and hinder adaptation.
Table WER (%) of Wev2Vee2ForCTC on caf-real CHiME-3 for different T-EMA decay ?
0.50
0.60
0.70
0.80
0.90
0.95
0.99
WER (%)
39.1
39.1
38.4
38.1
37.9
38.3
39.9
C.3.2 Sensitivity of Loss Component Weights
To further understand how the relative weighting of loss components affects the CMA-ES optimization process, we conducted a sensitivity analysis of the loss weights and 3 for the entropy loss and utterance-level loss, which in turn leads to changes in the token-level weight (Minmax normalization Equation in Section 3.3). The WER results using the caf-real condition of the CHiME3 dataset are shown in Table 10 below. The results reveal that WER is remarkably stable across a wide range of weight settings. WER stays within a narrow range across different values of a and B3 showing broad area of near-optimal performance instead of a single best point. The results suggest that both components the loss must be balanced, as extreme values for either can hurt performance, The chosen setting (a 1.0, B 2.0) sits comfortably in this stable region and consistently yields the best performance.
Table 10: WER (%) of Wav2Vec2ForCTC on caf-real CHiME-3 under varying a (rows) and B (columns), representing varying weights for the token-level loss.
0.5
1.0
1.5
2.0
2.5
3.0
3.5
4.0
4.5
5.0
0.5
37.7
38.2
38.5
38.6
39.1
39.4
39,2
39.8
39,4
39.7
1.0
38.1
38.1
38.0
37.9
38.4
38.3
38.4
38.8
38.7
39.6
1.5
38.5
38.1
38.4
38.1
38.5
38.0
38.2
37.8
38.3
38.4
2.0
38.4
38.3
38.0
37.9
37.9
38.1
38.0
38.3
38.0
38.4
C.3.3 Sensitivity CMA-ES to Target Domain Complexity
We selected four target domains exhibiting increasing variability, quantified by the covariance shift offered by Fréchet Inception Distance (FID) of the embedding distribution between the source and target. These range from low to high variability: Gaussian noise 0.01), CHiME-3 cafe- simulated, CHiME-3 cafe-real, and CHiMB-3 mixed. As shown in Table 11 below, with CMA-ES

---

population size increasing from 10 to 100, the largest gains oceur moving from 10 to 40 candidates across all domains. Beyond 50 candidates, performance plateaus. These results show that while more complex domains (cafe-real, mixed) begin at higher absolute WER, they exhibit the same early-saturation behavior as simpler conditions. Therefore, sensitivity to domain complexity may not be directly related to the number of prompt candidates considered.
Table 11: WER (%) across different population sizes on four target domains with varying complexity.
Candidate Size
10
20
30
40
50
60
70
80
90
100
Gaussian 0.01)
16.2
15.6
15.2
14.9
14.8
14.8
14.7
14.7
14.7
14.7
Cafe-simulated
17.0
16.5
16.5
16.2
16.1
16.3
16.1
16.0
16.0
16.2
Cafe-real
41.8
39.3
38.3
38.3
37.9
38.0
37.9
37.9
38.0
37.8
CHiME3 Mixed
25.7
25.1
24.7
24.6
24.3
24.3
24.4
24.2
24.4
24.3
C.3.4 Sensitivity of CMA-ES to Candidate Size and Iteration Steps
We find the expected trade-off between the population size and optimization iterations with word error rate (WER) (Table 12 and Table below). As shown Table 12 (caf-real condition of CHiME-3 using Vay2VecFFoCTTC), increasing the CMA-ES candidate size from 10 to 40 yields the most significant gains. Table 13 demonstrates that moving from 5 to iterations captures most of the benefit. We observe the same early- sarly-saturation behavior across other conditions (e.g.. bus-real, street-real) and datasets.
Table 12: WER (%) of WTT2 on the caf-real condition of CHiME-3 under different CMA-ES population sizes (iteration steps 25).
Candidate Size
10
20
30
40
50
60
70
80
90
100
WER (%)
41.8
39.3
38.3
38.3
37.9
38.0
37.9
37.9
38.0
37.8
Table 13: WER (%) of Wav2Vec2ForCTC on the caf-real condition of CHiME-3 under different CMA-ES iteration steps (population size 50).
Iteration Steps
10
15
20
25
30
35
40
45
50
WER (%)
40.8
39.9
38.6
38.0
37.9
38.0
37.9
37.6
38.4
38.0
C.4 Impact of Utterance-level Loss on Trivial Predictions
We analyzed and compared the occurrence of trivial predictions under two conditions: (i) entropy-only and (ii) entropy with utterance-level loss, using the CHiME-3-Mix dataset. As shown in Table 14, observed two types of trivial solutions: blank predictions and single-character predictions (entropy O with only character predicted). It is evident that the utterance-level loss significantly reduces the incidence of trivial solutions from 37.5% to 0.61 %, thereby improving the WER.
C.5 Effectiveness of Continual Adaptation with T-EMA
We reported the WER against the number process utterances under two variations of the continual adaptation process: (i) T-EMA with resetting, reported in Table in the ablation study; and (ii) T-EMA with exponential averaging (proposed), presented in Table and Section 3.4. We evaluated their performance across varying numbers of target samples from 100 to 800 in Table 15 below, using the LibriSpeech dataset with Gaussian noise (a 0.015). With the resetting approach, performance increases up to 300 samples and then declines with additional samples, suggesting that resetting CMA-ES prevents leveraging prior adaptation. contrast, the proposed exponential averaging method demonstrates relatively stable performance, indicating that even with small number of samples, reliable adaptation can be achieved. This is likely due the T-EMA mechanism accumulating knowledge from carlier utterances and updating CMA-ES in more favorable region
25

---

Table 14: Trivial solutions on CHiME-3-Mix number and percentage of trivial predictions among all predictions.
Configuration
Blank
Single-char
Total problematic
WER (%)
Entropy-only
42 (1.59%)
948 (35.91%)
990 (37.50%)
49.6
Utterance-level loss
12 (0.45%)
4 (0.15%)
16 (0.61%)
25.5
Table 15: WER (%) against the number of processed utterances with T-EMA (resetting) and T-EMA (proposed) under the LibriSpeech dataset with Gaussian noise (a = 0.015).
Method
100
200
300
400
500
600
700
800
T-EMA (Resetting)
20.4
19.7
19.8
20.4
20.8
20.8
21.1
21.3
T-EMA (Proposed)
17.9
17.0
17.2
17.4
17.6
17.5
17.8
17.7
Performance Gain
2.5
2.7
2.6
3.0
3.2
3.3
3.3
3.6
of the search space. The improved performance over the resetting mechanism further demonstrates that our method is robust to the number of samples used for adaptation, and that the proposed T-EMA effectively stabilizes the adaptation process.
C.6 Memory Usage
Figure 6 shows the memory usage comparison between two SFMs: Wav2Nec2ForCTC- and HuBERTFooCTC--arge The increasing model size leads to significant increase in memory usage for adaptation, especially for the BP-based TTA, whereas our method demonstrates only slight increase in memory usage.
Avg Peak GPU Memory Usage (MB)
Wav2Vec2ForCTC-Base
12500
HuBERTForCTC-Large
10000
7500
5000
2500
Ours A3A LAME
FOA
SUTA
DSUTA
CSUTA
AWMC
TENT
EATA
SAR
SGEM CEA COTTA
Methods
Figure 6: Comparison of Average Peak GPU memory usage of different TTA methods across all datasets with two different scale backbones.
C.7 Controlling Prompt Vector Dimensionality for CMA-ES Efficiency
The computational cost of CMA-ES with increasing dimensionality d may cause concerns. Indeed, the full-rank CMA-ES algorithm has a per-iteration complexity of O + dd), where J number of sampled solutions (i.e., prompts in our case). To mitigate this cost, we design our prompt vector to have fixed dimensionality of 512, aligned with the embedding size commonly used in speech foundation models, ensuring that d remains tractable in practical scenarios. Furthermore, as
26

---

Table 16: WER (%) of Waw2Vec2ForCTC on the caf-real condition of CHiME-3 under different CMA-ES candidate sizes / (iteration steps 25).
J
10
20
30
40
50
60
70
80
90
100
WER (%)
41.8
39.3
38.3
38.3
37.9
38.0
37.9
37.9
38.0
37.8
Table 17: Per-utterance adaptation time complexity (big-O) for E-BATS and top TTA baselines.
Method
Time Complexity per Utterance
DSUTA
O(N (F E 1++++1 PP)
FOA
O(N J(F + ([F+E+1A Ld) d
E-BATS
O(N x [[F+ +++++d+VVVV)) +d1)
shown in Table 16, with our grid search of J, we observe the optimal always much smaller than d: in our configuration, we fix J = 50, which remains significantly smaller than d. This ensures that the asymptotic complexity does not exceed (dd). Thus, while CMA-ES theoretically scales cubically with d, our design choices effectively cap the computational overhead in the context of high-dimensional but fixed-size prompt vectors.
C.8 Adaptation performance on Speech Emotion Recognition Task
To further test the generality of method beyond ASR with CTC, applied E-BATS speech emo- tion recognition tasks (cross-entropy loss) using the IEMOCAP dataset and the Spainmmmrintemotiooon wwwenn-wwwwwwwweaaaaaaaa
model under additive Gaussian noise (a 0.02). Adaptation increased emotion prediction accuracy from 38.8% to 43.4%, demonstrating that our E-BATS frame- work can enhance the performance of other downstream tasks using SFMs.
C.9 Adaptation Speed and Computation Complexity
We compute the per-utterance time complexity to compare E-BATS with the top-performing backpropagation-basee (DSUTA) and backpropagation-free (FOA) methods (see Table 17). DSUTA scales with the large number of model parameters via backpropagation (P), often in hundreds of millions, whereas E-BATS only involves (dd)) and (d2) operations (with d = 512). Since P 2 d3 in most SFM models, E-BATS achieves significant efficiency gains over DSUTA. While both FOA and E-BATS share the same asymptotic complexity of Odd)) for CMA-ES updates, E-BATS prac- tically more efficient since FOA requires 3 multiplicative factor of (dd)) it needs three prompts for adaptation. FOA also incurs an additional (LL)) cost (over d) per prompt for prompt attention with L transformer layer encoders. Nonetheless, main focus, noted in the introduction, is the trade-off between accuracy and memory efficiency, which more critical for resource-constrained devices.
where:
F: cost of one forward pass
E: cost of entropy loss calculation
N: number of adaptation iteration steps per utterance
number of candidate prompts per iteration
V|: size of the token class
D Algorithms
The algorithm for LPA per utterance and for T-EMA shown Algorithm and Algorithm respectively.
27

---

Algorithm Lightweight Prompt Adaption (Per Utterance)
Require: CMA-ES params 6% steps K, Utterance X Ensure: Adapted predictions y
1: best_ 4-
2: for k = to do
3: Sample prompts S = [ 22 .... st.j] from CMA-ES 0r.k-1
4: Inject st with Z,and feedforward pass
5: Compute loss L Lddaps,all
D ad I adopt,
I J
6: if min.jet Ladaptss best_loss loss then
7:
best_lost jost_loss= L adapt.j
8:
Y, decode adapted output with st.5
9:
end if
10: & Update( - alls S
11: end for
Algorithm 2 T-EMA Updating Strategy
Require: Utterance data T, CMA-ES params oh Iteration Steps K
1: Initialize Φema = {Cema: Memas Tema < d0
2: for each utterance in T do
3: Φb <- oit Run Algorithm
4: EMA Update (œema: xK
5: Φb+1 +- Φema
6: end for
E Ethical Consideration
Our research fully complies with the NeurIPS Code Ethics. We We exclusiell utilize publicly available datasets, pretrained models, baseline methods, and their accompanying code, strictly adhering to their respective licenses and usage protocols. We did not collect any new data, nor do our adaptation methods pose privacy risks or enable misuse. Thus, our work does introduce broader negative societal impacts, eliminating the need for additional safeguards beyond standard ethical research practices.
Moreover, our method has potential positive socictal impacts, including improving the accessibility and reliability of speech recognition technology in noisy real-world environments, thereby benefiting communication technologies, assistive systems, and applications serving diverse and inclusive user populations.
28


Reviews:
[{"review_id": "mYhSVCF3FB", "reviewer": "Reviewer_uVzm", "summary": "This paper proposes a novel backpropagation-free adaptation approach specifically designed for CTC (Connectionist Temporal Classification) models that incorporate pre-trained speech encoders. The method addresses a critical challenge in speech recognition: efficiently adapting models to new acoustic conditions and domains without the computational overhead of traditional gradient-based fine-tuning. The paper further introduces a sampling strategy over a set of learnable parameters, guided by a combination of losses at different levels. Experimental results are promising in terms of both accuracy and GPU memory efficiency.", "questions": "[1] Misuse of “blank” tokens: \n- The paper repeatedly discusses “blank” tokens in the context of speech foundation models. However, speech foundation models typically refer to encoders pre-trained via self-supervised learning (e.g., wav2vec, HuBERT), which do not use blank tokens. Blank tokens are specific to CTC or Transducer objectives. While the authors seem to apply a CTC objective during fine-tuning or adaptation, this is never clearly stated in the method section. CTC is only vaguely mentioned in the experimental setup, making the discussion around blanks confusing and misleading.\n\n[2] Misleading scope and title:\n-  The title and scope suggest a method tailored to “speech foundation models,” but the proposed method is in fact specific to models trained with the CTC objective. The foundation model merely serves as the encoder. CTC models and speech foundation models are conceptually distinct, and the proposed technique is not unique to foundation models—it could be applied to any encoder trained with CTC. Therefore, the title should reflect the method’s actual scope, i.e., adaptation for CTC models, not speech foundation models.\n\n[3] Incorrect explanation of blank tokens: \n- The statement that blank tokens represent “frames where no alphabetic character can be assigned, typically due to ambiguous sounds or silent periods” is inaccurate. In CTC, blank tokens address the alignment mismatch between input and output lengths and are essential to allowing flexible alignments—not limited to silences or ambiguities.\n\n[4] Prompt tuning claims:\n-  The authors write, “While conventional prompt tuning approaches have primarily been developed for transformer-only architectures…,” which is misleading. In speech processing, parameter-efficient adaptation using small learnable modules predates the rise of transformers. Concepts like adapter layers or bottleneck modules were already in use with LSTM or CNN-based models. Therefore, the claim about prompt tuning being “primarily developed for transformers” is historically inaccurate.", "limitations": "[1] On mean shift:\n-  The paper attributes distribution shifts mainly to mean shifts. However, most of the noise scenarios appear to be simple additive noise, where mean shift is expected. To strengthen this claim, the authors should test their method on other types of distortions (e.g., reverberation, channel effects) to confirm whether mean shift still dominates.\n\n[2] On token-level loss and pseudo-labels: \n- The token-level loss relies on pseudo-labels, which may be incorrect. If a wrong label has a source-domain-like hidden representation, the adaptation may reinforce the wrong prediction. This risk is acknowledged but not addressed convincingly. Moreover, the ablation suggests the token-level loss is not particularly effective. This should be discussed more critically.\n\n[3] Entropy loss risk:\n-  The same risk applies to the entropy loss. Encouraging low entropy for incorrect predictions may further entrench errors. More analysis is needed to understand the failure modes of this loss.\n\n[4] Missing ablation on utterance-level loss:\n-  The utterance-level loss appears to be the most reasonable component, aligning overall acoustic features like speaker or noise condition. However, the paper does not present an ablation study on using only this loss, which would help clarify its actual contribution.\n\n[5] Choice of LibriSpeech subsets: \n- The authors use LibriSpeech test-other, which is already relatively noisy. Why not use test-clean and apply synthetic noise to demonstrate the effectiveness more clearly?\n\n[6] CTC model details missing: \nIt is unclear how the CTC models are obtained. Were they fine-tuned from foundation models by the authors? Or are they taken from public checkpoints (e.g., Hugging Face, Fairseq)? This detail is essential for reproducibility and understanding the setup.\n\n[7] CommonVoice version: \n- The version of CommonVoice used in the experiments is not reported. This should be clearly specified.\n\n[8] Adaptation speed:\n-  There is no measurement of adaptation time. Since the paper positions itself as a test-time method, it would be helpful to include comparisons of time efficiency against baseline approaches", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 3, "strengths_and_weaknesses": "Strengths: \n- The use of sequence-level loss is well-motivated. \n- Low-level acoustic factors such as background noise, speaker characteristics, and environmental conditions typically affect entire speech sequences uniformly rather than individual tokens is sound. It makes sequence-level alignment and adaptation a reasonable and effective methodological choice that aligns with the underlying nature of acoustic domain shifts.\n\nWeaknesses:\n- There are several inaccuracies and conceptual issues in the method section regarding CTC, speech foundation models, and adaptation.", "was_revised": true, "final_justification": "I thank the authors for their great attempt in addressing the concerns, however, reading all reviews and comments I will keep my initial score.", "rebuttal": "## Response to Reviewer\nWe thank the reviewer for the very detailed feedback. We greatly appreciate the reviewer acknowledges the novelty and motivation of our backpropagation-free TTA method, highlighting its effectiveness in handling domain shifts through sequence-level loss and its efficiency in both accuracy and memory usage. Below, we address the reviewer's concerns, and we will include these results and analysis in the appendix of the revised version.\n\n---\n- **Q1:** Misuse of blank tokens\n\n  **Response:** Blank tokens are standard in CTC-based ASR models, and CTC objective is widely used in ASR systems. We will move the CTC setup earlier in Section 3.1, before detailing the method to clarify the scope. Importantly, the novelty of our proposed TTA method remains intact: our approach is specifically designed for speech foundation model encoder architectures, and to our knowledge, has not been addressed in previous work. We will revise Section 3.1 to clarify the scope with respect to CTC objectives.\n\n---\n- **Q2:** Misleading scope and title\n\n  **Response:** Thanks for bringing the discussion of the scope. Our method is fundamentally driven by the encoder architecture of speech foundation models (SFMs), but not the CTC objectives, therefore, we would keep the title and scope unchanged. Every component of E‑BATS, from injecting learnable prompts into the final CNN feature maps to applying our multi‑scale discrepancy loss across each transformer block, directly leverages the SFM encoder design. To further test the generality of our method beyond ASR with CTC, we applied E‑BATS to speech emotion recognition tasks (cross-entropy loss) using the IEMOCAP dataset and the `SpeechBrain/emotion-recognition-wav2vec2-IEMOCAP` model under additive Gaussian noise (σ=0.02). Adaptation increased emotion prediction accuracy from 38.8% to 43.4%, demonstrating that our E-BATS framework can enhance the performance of other downstream tasks using SFMs. We will include the results in the appendix.\n---\n- **Q3** Incorrect explanation of blank tokens\n\n  **Response:** We appreciate the chance to clarify. We acknowledge that blank tokens address the alignment mismatch in CTC tasks. However, we have observed the predictions in both silent and ambiguous sounds empirically in our experiments and thus mentioned them for simplicity. Moreover, our observations align with recent studies that have claimed that predicting blank tokens in CTC helps eliminate the need to identify inherently ambiguous label boundaries [1], and also highlights silent regions [2]. We will revise the text to include the claim regarding alignment to address this concern.\n\n---\n- **Q4:** Prompt tuning claims\n\n  **Response:** We'd like to highlight \"prompt tuning\" and \"adapter layers or bottleneck modules\" are two distinct parameter-efficient fine-tuning approaches [3]. Prompt tuning, introduced by Lester et al. [4], adds a small set of learnable “soft” tokens to the input sequence, which are learned via the Transformer’s attention mechanism. This allows improved performance by updating those prompt embeddings directly without modifying the model’s original weights. In contrast, adapter layers insert bottleneck modules within the network and train their internal weights and biases. In our work, we refer specifically to the modern concept of prompt tuning, which is primarily designed for Transformers. To address the reviewer’s concern, we will include references [3, 4] in Section 2 for clarity.\n\n---\n- **W1:** On mean shift.\n\n  **Response:** CHIME-3, TEDLIUM, and Common Voice datasets contains real-world environmental noise. Specifically, the CHIME-3 dataset includes recordings from four distinct acoustic environments: bus, café, street junction, and pedestrian area. Each of these presents unique natural reverberation characteristics, from the reflective interiors of buses and cafés to the more open acoustics found in street junctions and pedestrian areas. While including a broader range of conditions would be ideal, we believe that the 16 conditions investigated are sufficient to draw a conclusive result. We plan to address this in future work as more real-world datasets with reverberation become available, and appreciate suggestions from the reviewer regarding the dataset.\n\n---\n- **W2:** On token-level loss and pseudo-labels.\n\n  **Response:** We address this with the proposed adaptive weighting mechanism for the token-level loss, as described in Section 3.3. Specifically, the weight assigned to the token-level loss depends on the reliability of the pseudo-labels, which is measured by the entropy and the utterance-level loss. If the sum of the entropy and utterance-level loss is low, indicating more reliable pseudo-labels, we assign a higher weight to the token-level loss, and vice versa. This approach is intended to mitigate the risk of relying on incorrect pseudo-labels. \n\nRegarding effectiveness, in our design, token-level loss is incorporated to provide valuable support when distribution shifts are relatively minor or when pseudo labels are more reliable. We recognize that in severe scenarios requiring substantial adaptation, utterance-level loss and entropy serve as the primary and most effective drivers of adaptation. The proposed multi-level loss aims to address both subtle and significant distribution shifts.\n\n---\n- **W3:** Entropy loss risk.\n\n  **Response:** We analyzed the failure mode of entropy loss and further demonstrated that the proposed utterance-level loss can compensate for it. The results are presented in Table 1, using CHiME-3-Mix dataset. Entropy loss leads to blank token predictions and single-character predictions (entropy = 0 with only one character predicted), accounting for 37.5% of cases. By incorporating utterance-level loss, these problematic predictions are significantly reduced, resulting in a substantial improvement in WER. We will include a detailed analysis in the appendix. \n\n  **Table 1.** Trivial solutions on CHiME-3-Mix: number and percentage of trivial predictions among all predictions.\n  |Configuration|Blank|Single-char|Total|WER (%)|\n  |-|-|-|-|-|\n  |Entropy-only|42 (1.59%)&emsp;|948 (35.91%)&emsp;|990 (37.50%)&emsp;|49.6|\n  |+Utterance-level loss|12 (0.45%)&emsp;|4 (0.15%)&emsp;|16 (0.61%)&emsp;|25.5|\n\n---\n- **W4:** Missing ablation on utterance-level loss.\n\n  **Response:** We evaluated performance using only the utterance-level loss under the CHiME3-mix condition with Wav2Vec2ForCTC and achieved a WER of 26.8%. This result surpasses the performance with entropy loss alone and we appreciate the reviewer’s suggestion. We will include this additional ablation study in Table 3. Nonetheless, combining both losses yields even greater improvements of 25.5%. \n\n---\n- **W5:** Choice of LibriSpeech subsets.\n\n  **Response:** To enable a fair and direct comparison with baselines like SUTA and CEA, we used the same LibriSpeech test-other set for evaluation.\n---\n- **W6:** CTC model details missing.\n \n  **Response:** All CTC models used in our experiments are based on publicly available checkpoints from Facebook via Hugging Face, specifically `facebook/wav2vec2-base-960h` and `facebook/hubert-large-ls960-ft`. Moreover, as noted at the end of the abstract, we will release our code and full reproduction instructions upon acceptance to ensure complete transparency and reproducibility.\n\n---\n- **W7:** CommonVoice version.\n\n  **Response:** As presented in Appendix B.1, we adopted the test set from the en-June-22nd-2020 version which will be included as footnote in the revised version.\n\n---\n- **W8:** Adaptation speed\n\n  **Response:** We compute the per-utterance time complexity to compare E‑BATS with the top-performing backpropagation-based (DSUTA) and backpropagation-free (FOA) methods (see Table 2). DSUTA scales with the large number of model parameters via backpropagation (P), often in hundreds of millions, whereas E‑BATS only involves 𝒪(d²) and 𝒪(d³) operations (with d=512). Since P ≥ d³ in most SFM models, E‑BATS achieves significant efficiency gains over DSUTA. While both FOA and E-BATS share the same asymptotic complexity of 𝒪(d³) for CMA-ES updates, E-BATS are practically more efficient since FOA needs a 3³ multiplicative factor of 𝒪(d³) as it requires three prompts for adaptation. And FOA incurs an additional 𝒪(Ld²) cost (over d) per prompt for prompt attention with L transformer layer encoders. We will include the analysis in the appendix. Nonetheless, our main focus, as noted in the introduction, is the trade-off between accuracy and memory efficiency, which is more critical for resource-constrained devices.\n \n  **Table 2**: Per-utterance adaptation time complexity (big-O) for E-BATS and top TTA baselines.\n  |Method|Time Complexity per Utterance|\n  |-|-|\n  |DSUTA|𝒪(N × (F + E + P))|\n  |FOA|𝒪(N × [J (F + E + L d²) + d³])|\n  |E-BATS|𝒪(N × [J (F + E + L d + L\\|V\\|d) + d³])|\n\n  where:\n  - F: cost of one forward pass\n  - E: cost of entropy loss calculation  \n  - N: number of adaptation iteration steps per utterance  \n  - J: number of candidate prompts per iteration \n  - |V|: size of the token class\n\n---\nReferences:\n\n[1] Graves, A. ,et al . Connectionist Temporal Classification: Labelling Unsegmented Sequence Data with Recurrent Neural Networks. ICML 2006.\n\n[2] Mostafa, A., et al. Phonological Level Wav2Vec2‑based Mispronunciation Detection and Diagnosis Method. *Journal of Speech Technology*, 10(2), 123–135.\n\n[3] Wang, L., et al. Parameter-efficient fine-tuning in large language models: a survey of methodologies. Artif Intell Rev 58, 227 (2025). \n\n[4] Lester, B.,et al. The Power of Scale for Parameter‐Efficient Prompt Tuning. In EMNLP 2021."}, {"review_id": "3tAepiWLGK", "reviewer": "Reviewer_8uQQ", "summary": "In this paper, the authors propose E-BATS, a novel backpropagation-free test time adaptation framework for speech fundamental models. The method utilizes a lightweight cue adaptation module that combines multi-scale distributional alignment loss (entropy, part-of-speech level, and token level) and test time-exponential moving average to achieve stable, memory-efficient adaptation under dynamic acoustic conditions. Extensive experiments on multiple benchmarks show strong performance across a wide range of domain shifts, significantly outperforming existing BP-free methods and comparable to BP-based methods, while maintaining low memory usage.", "questions": "1. How sensitive is the adaptation performance to the number of CMA-ES candidates and iterations? Could performance degrade with fewer samples or iterations under tighter latency budgets?\n2. Is there any target domain condition where the pre-collected source stats perform poorly or even misguide the adaptation (*e.g.*, heavily accented speech)?\n3. Would a learnable or data-driven initialization of prompt vectors improve convergence or stability over the mean-zero initialization used in CMA-ES?", "limitations": "As mentioned by the authors, the CMA-ES optimization introduces latency, and GPU parallelism is not fully leveraged. This may hinder adoption in strict real-time settings.", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 4, "significance": 3, "originality": 3, "strengths_and_weaknesses": "**Strengths**\n- Clear and well-written paper with intuitive motivation and clear organization. Methodology is described in sufficient detail.\n- The proposed approach addresses a practical and under-explored setting: test-time adaptation for SFMs without backpropagation.\n- Significant memory savings (2x–6.4x) over BP-based methods while achieving competitive or superior performance.\n- Ablation study is thorough and validates the necessity of each component (prompt adaptation, multi-scale loss, T-EMA).\n\n**Weaknesses**\n- The effectiveness of the prompt candidate search via CMA-ES may be sensitive to the complexity of the target domain. There is no discussion on candidate sufficiency, especially for domains with high variability.\n- While the authors argue for the reliability of pre-collected source statistics, the domain generality or robustness of these statistics (*e.g.*, under accent or unseen noise types) is not thoroughly analyzed.\n- Entropy-only optimization is known to lead to trivial solutions, and although this is addressed via auxiliary loss terms, it would help to see more quantitative evidence on how frequently trivial predictions (*e.g.*, blanks) occur and how the loss balances affect that.", "was_revised": true, "final_justification": "My concerns have been addressed. I will keep my positive rating.", "rebuttal": "## Response to Reviewer\nWe thank the reviewer for the very detailed feedback. We greatly appreciate the reviewer recognition of our work addressing \"a practical and under-explored setting\" of the backpropagation-free TTA framework for speech foundation model with \"competitive or superior performance\" and \"significant memory saving.\" Below, we address the reviewer's concerns, and will include these results and analysis in the appendix of the revised version.\n\n---\n\n- **W1:** The effectiveness of the prompt candidate search via CMA-ES may be sensitive to the complexity of the target domain. There is no discussion on candidate sufficiency, especially for domains with high variability.\n  \n  **Response:** We selected four target domains exhibiting increasing variability, quantified by the covariance shift offered by Fréchet Inception Distance (FID) of the embedding distribution between source and target. These range from low to high variability: Gaussian noise (σ = 0.01), CHiME-3 café-simulated, CHiME-3 café-real, and CHiME-3 dynamic. As shown in Table 1 below, with CMA-ES population size increasing from 10 to 100, the largest gains occur moving from 10 to 40 candidates across all domains. Beyond 50 candidates, performance plateaus. These results show that while more complex domains (café-real, dynamic) begin at higher absolute WER, they exhibit the same early-saturation behavior as simpler conditions. Therefore, sensitivity to domain complexity may not be directly related to the number of prompt candidates considered. We will include this ablation in the appendix.\n\n  **Table 1.** WER (%) across different candidate sizes on four target domains with varying complexity.\n\n  | Candidate Size | 10   | 20   | 30   | 40   | 50   | 60   | 70   | 80   | 90   | 100  |\n  |---------------|------|------|------|------|------|------|------|------|------|------|\n  | **Gaussian (σ=0.01)**                | 16.2 | 15.6 | 15.2 | 14.9 | 14.8 | 14.8 | 14.7 | 14.7 | 14.7 | 14.7    |\n  | **CHiME-3 Single (café-simulated)**  | 17.0 | 16.5 | 16.5 | 16.2 | 16.1 | 16.3 | 16.1 | 16.0 | 16.0 | 16.2 |\n  | **CHiME-3 Single (café-real)**       | 41.8 | 39.3 | 38.3 | 38.3 | 37.9 | 38.0 | 37.9 | 37.9 | 38.0 | 37.8 |\n  | **CHiME-3 Dynamic**                  | 25.7 | 25.1 | 24.7 | 24.6 | 24.3 | 24.3 | 24.4 | 24.2 | 24.4 | 24.3 |\n\n--- \n\n- **W2:** While the authors argue for the reliability of pre-collected source statistics, the domain generality or robustness of these statistics (e.g., under accent or unseen noise types) is not thoroughly analyzed.\n \n  **Response:** Our pre-collected source statistics are derived from the clean LibriSpeech dataset, which serves as the training data for SFMs. In our experiments, they consistently improved adaptation without overfitting, even under diverse accents and noise types. We would appreciate any additional clarification on this weakness.\n\n---\n\n- **W3:** Entropy-only optimization is known to lead to trivial solutions, and although this is addressed via auxiliary loss terms, it would help to see more quantitative evidence on how frequently trivial predictions (e.g., blanks) occur and how the loss balances affect that.\n\n  **Response:** We analyzed and compared the occurrence of trivial predictions under two conditions: (i) entropy-only and (ii) entropy with utterance-level loss, using the CHiME-3-Mix dataset. As shown in Table 2, we observed two types of trivial solutions: blank predictions and single-character predictions (entropy = 0 with only one character predicted). It is evident that the utterance-level loss significantly reduced the incidence of trivial solutions from 37.5% to 0.61%, thereby improving the WER. We will include this analysis in the appendix.\n\n  **Table 2.** Trivial solutions on CHiME-3-Mix: number and percentage of trivial predictions among all predictions.\n\n  | **Configuration**           | **Blank**           | **Single-char**         | **Total problematic**     | **WER (%)** |\n  |-----------------------------|---------------------|--------------------------|----------------------------|-------------|\n  | **Entropy-only**            | 42 (1.59%) &emsp;   | 948 (35.91%) &emsp;     | 990 (37.50%) &emsp;       | 49.6        |\n  | **+ Utterance-level loss**  | 12 (0.45%) &emsp;   | 4 (0.15%) &emsp;        | 16 (0.61%) &emsp;         | 25.5        |\n\n\n---\n\n- **Q1:** How sensitive is the adaptation performance to the number of CMA-ES candidates and iterations? Could performance degrade with fewer samples or iterations under tighter latency budgets? \n\n  **Response:** We find the expected trade-off between the candidate numbers and optimization iterations with word error rate (WER) (Table 3 and Table 4 below). We will add a sentence in the main text and include these results in the appendix. As shown in Table 3 (caf-real condition of CHiME-3 using Wav2Vec2ForCTC), increasing the CMA-ES candidate size from 10 to 40 yields the largest gains. Table 4 demonstrates that moving from 5 to 25 iterations captures most of the benefit. We observe the same early-saturation behavior across other conditions (e.g., bus-real, street-real) and datasets.\n\n  **Table 3.** WER (%) of Wav2Vec2ForCTC on the _caf-real_ condition of CHiME-3 under different CMA-ES candidate sizes (iteration steps = 25).\n\n  | Candidate Size | 10   | 20   | 30   | 40   | 50   | 60   | 70   | 80   | 90   | 100  |\n  |--------------------|------|------|------|------|------|------|------|------|------|------|\n  | **WER (%)**        | 41.8 | 39.3 | 38.3 | 38.3 | 37.9 | 38.0 | 37.9 | 37.9 | 38.0 | 37.8 |\n\n  **Table 4.** WER (%) of Wav2Vec2ForCTC on the _caf-real_ condition of CHiME-3 under different CMA-ES iteration steps (candidate size = 50).\n\n  | Iteration Steps | 5    | 10   | 15   | 20   | 25   | 30   | 35   | 40   | 45   | 50   |\n  |-----------------|------|------|------|------|------|------|------|------|------|------|\n  | **WER (%)**     | 40.8 | 39.9 | 38.6 | 38.0 | 37.9 | 38.0 | 37.9 | 37.6 | 38.4 | 38.0 |\n\n- **Q2:** Is there any target domain condition where the pre-collected source stats perform poorly or even misguide the adaptation (e.g., heavily accented speech)?\n \n  **Response:** We clarify that the pre-collected statistics are derived from clean speech rather than noisy speech, and therefore are not misleading in any adaptation scenario. Specifically, SFMs are trained on clean speech using publicly available datasets such as LibriSpeech, and the pre-collected statistics are obtained from the clean LibriSpeech dataset. As a result, these statistics do not introduce noise or negatively impact the adaptation process. In the revised version, we will include a sentence explicitly stating that the pre-collected statistics are sourced from clean LibriSpeech data.\n\n- **Q3:** Would a learnable or data-driven initialization of prompt vectors improve convergence or stability over the mean-zero initialization used in CMA-ES?\n \n  **Response:** Thank you for the suggestion. While we currently use a mean-zero initialization in CMA-ES for simplicity and generality, we agree that learnable or data-driven initialization (e.g., from previous prompts, pre-trained embeddings, or lightweight estimators) could potentially improve convergence speed and stability, especially in early adaptation steps. This is an interesting direction that complements our T-EMA design and will be explored in future work.\n\n---\n\nThanks again for your comments! We remain open to addressing any additional concerns or clarifications you may have."}, {"review_id": "CPHgZdeZ9k", "reviewer": "Reviewer_Wcum", "summary": "This paper addresses a critical limitation of speech foundation models—their susceptibility to performance degradation under real-world acoustic domain shifts such as background noise and speaker accents. The authors explore Test-Time Adaptation (TTA) as a strategy to overcome such challenges without requiring access to source data or labels. The primary innovation is E-BATS, a backpropagation-free TTA framework tailored specifically for speech tasks.", "questions": "N/A", "limitations": "N/A", "rating": 4, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\n\n1. The paper identifies a clear gap in existing TTA methods—most are either memory-intensive (due to backpropagation) or ineffective (being designed for vision tasks). E-BATS provides a compelling, efficient, and accurate alternative for speech scenarios.\n\n2. The method integrates:\no Prompt-based adaptation, enabling forward-only updates that significantly reduce computational overhead.\no Multi-scale loss, which captures both global (utterance-level) and local (token-level) distributional shifts, often overlooked in vision-\nbased approaches.\no Exponential Moving Average (EMA), which ensures smoother and more stable adaptation across test utterances.\n\n3. The framework demonstrates strong generalization across four noisy speech datasets and sixteen acoustic domains, showing consistent and meaningful improvements.\n\nWeaknesses:\n\n1. The proposed multi-scale loss combines utterance-level and token-level objectives, but it remains unclear how the relative weighting of these loss components affects the CMA-ES optimization process. As CMA-ES is a black-box optimizer sensitive to the structure of the objective function, an ablation or sensitivity study would strengthen the work.\n\n2. The stability and effect of the EMA mechanism deserve more in-depth analysis, particularly regarding its sensitivity to hyperparameters and its role in generalization.\n\n3. I am not an expert in this specific subfield, so I cannot confidently assess whether similar models or ideas have previously been proposed. Additionally, while the reported results are promising, it is difficult to judge the practical significance of the improvements without deeper domain knowledge or context.", "was_revised": false, "final_justification": "", "rebuttal": "# Response to Reviewer\n\nWe thank the reviewer for the detailed feedback. We greatly appreciate your recognition of our work as offering a \"compelling, efficient, and accurate\" Test‑Time Adaptation framework tailored to real‑world acoustic shifts. Below, we address your questions. We will include these results and analysis in the appendix of the revised version.\n\n---\n\n- **W1:** The proposed multi-scale loss combines utterance-level and token-level objectives, but it remains unclear how the relative weighting of these loss components affects the CMA-ES optimization process. As CMA-ES is a black-box optimizer sensitive to the structure of the objective function, an ablation or sensitivity study would strengthen the work.\n\n  **Response:** To address this, we conducted a sensitivity analysis of the loss weights α and β for the entropy loss and utterance‑level loss, which in turn leads to changes in the token‑level weight c (Minmax normalization Equation in Section 3.3). The WER results using the caf‑real condition of CHiME3 dataset are shown in Table 1 below. The results reveal that WER is remarkably stable across a wide range of weight settings. WER stays within a narrow range across different values of α and β, showing a broad area of near-optimal performance instead of a single best point. The results suggest that both components of the loss must be balanced, as extreme values for either can hurt performance. The chosen setting (α=1.0, β=2.0) sits comfortably in this stable region and consistently yields the best performance. We will include this in the appendix.\n\n\n  **Table 1.** WER (%) of Wav2Vec2ForCTC on caf‑real CHiME‑3 under varying α (rows) and β (columns), representing varying weights for the token-level loss.\n\n  | α \\ β | 0.5  | 1.0  | 1.5  | 2.0  | 2.5  | 3.0  | 3.5  | 4.0  | 4.5  | 5.0  |\n  |-------|------|------|------|------|------|------|------|------|------|------|\n  | **0.5** | 37.7 | 38.2 | 38.5 | 38.6 | 39.1 | 39.4 | 39.2 | 39.8 | 39.4 | 39.7 |\n  | **1.0** | 38.1 | 38.1 | 38.0 | 37.9 | 38.4 | 38.3 | 38.4 | 38.8 | 38.7 | 39.6 |\n  | **1.5** | 38.5 | 38.1 | 38.4 | 38.1 | 38.5 | 38.0 | 38.2 | 37.8 | 38.3 | 38.4 |\n  | **2.0** | 38.4 | 38.3 | 38.0 | 37.9 | 37.9 | 38.1 | 38.0 | 38.3 | 38.0 | 38.4 |\n\n---\n\n- **W2:** The stability and effect of the EMA mechanism deserve more in-depth analysis, particularly regarding its sensitivity to hyperparameters and its role in generalization.\n\n  **Response:** Table 2 below presents our ablation study on the EMA parameter γ, where we evaluated performance across γ ∈ {0.50, 0.60, 0.70, 0.80, 0.90, 0.95, 0.99} under the caf-real condition. We observed a clear U-shaped WER curve: performance improves up to γ = 0.90 (best WER 37.9%), and then degrades as γ increases further. This behavior aligns with our expectation and is also observed in other datasets that smaller γ values lead to unstable, overly reactive updates, while larger values result in over-smoothing and hinder adaptation. We will include these sensitivity analyses and tables in the appendix of the revised submission.\n\n  **Table 2.** WER (%) of Wav2Vec2ForCTC on caf‑real CHiME‑3 for different T‑EMA decay γ.\n\n  | γ    | 0.50 | 0.60 | 0.70 | 0.80 | 0.90 | 0.95 | 0.99 |\n  |------|------|------|------|------|------|------|------|\n  | **WER (%)** | 39.1 | 39.1 | 38.4 | 38.1 | 37.9 | 38.3 | 39.9 |\n\n---\n\nThank you for your comments! Let us know if we can further clarify anything."}, {"review_id": "NyKvAMpnCC", "reviewer": "Reviewer_4Yrj", "summary": "This paper introduces E-BATS, a novel backpropagation-free framework for efficiently adapting speech foundation models to new, noisy environments at test-time. By optimizing a lightweight \"prompt\" vector using a forward-pass-only process, the method avoids the high memory costs of traditional backpropagation-based adaptation. Experiments demonstrate that E-BATS achieves strong accuracy, comparable to or exceeding state-of-the-art methods, while using significantly less memory.", "questions": "Regarding the continual adaptation process, it would be useful to see how performance evolves as more test samples from a target domain are processed. For example, a plot of WER against the number of processed utterances could help illustrate the adaptation dynamics over time.", "limitations": "yes", "rating": 5, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "## Strengths\n1. The proposed method was evaluated on two speech foundation models (Wav2Vec2 and Hubert) across several ASR domain shift scenarios. The experiments show the method's effectiveness under the tested conditions.\n\n2. The paper is clearly written and the methodology is easy to follow. The inclusion of an ablation study helps clarify the contribution of individual components of the proposed framework.\n\n## Weaknesses\n\n1. The evaluation is currently limited to the ASR task. Exploring the method's applicability to other speech tasks (e.g. emotion recognition, speaker verification) is necessary since Wav2Vec2 and Hubert are general-purpose speech foundation models.\n\n2. Based on the ablation study, the inclusion of the $L_{utt}$ and $L_{token}$ loss components provides a marginal improvement when the T-EMA module is active.", "was_revised": true, "final_justification": "My concerns have been well addressed. I have increased my score to 5.", "rebuttal": "## Response to Reviewer\n\nWe thank the reviewer for the detailed feedback. We greatly appreciate the reviewer's recognition of our work as offering a \"novel\" backpropagation-free Test‑Time Adaptation framework tailored to real‑world acoustic shifts with \"strong accuracy\" and \"using significantly less memory\". Below, we address the reviewer's concerns regarding applicability beyond ASR, the role of our loss components, and adaptation dynamics over time. We will include these results and analysis in the appendix of the revised version.\n\n---\n- **W1:** The evaluation is currently limited to the ASR task. Exploring the method's applicability to other speech tasks (e.g. emotion recognition, speaker verification) is necessary since Wav2Vec2 and Hubert are general-purpose speech foundation models.\n\n  **Response:** We further applied E-BATS to speech emotion recognition using the IEMOCAP benchmarking dataset and the `SpeechBrain/emotion-recognition-wav2vec2-IEMOCAP` model. Emotion prediction accuracy improved from 38.8% (source model without adaptation) to 43.4% (with E-BATS) under additive Gaussian noise (σ = 0.02). This shows that our E-BATS framework can generalize to other speech tasks. We will include these experiments in the appendix of the revised version.\n\n---\n- **W2:** Based on the ablation study, the inclusion of the $L_{utt}$ and $L_{token}$ loss components provides a marginal improvement when the T-EMA module is active.\n\n  **Response:** T-EMA is designed to maintain stability during the adaptation process by preventing drastic changes caused by individual utterances, resulting in more gradual and reliable model updates. With T-EMA applied, both utterance-level and token-level adaptations benefit from this enhanced stability, making them unlikely to be destabilized by any single utterance. As a result, utterance-level or token-level adaptation on top of T-EMA yields slight improvements. In contrast, without T-EMA, adaptation based on entropy loss alone (49.6%) can be significantly affected by inaccuracies from a single utterance. In such cases, utterance-level and token-level losses play a crucial stabilizing role (25.4%). Overall, these results further validates the reliability of utterance-level and token-level losses. We believe this provides valuable insights into the impact of different loss designs under various settings for SFMs, and we will clarify this further in the ablation study subsection.\n\n---\n\n- **Q1:** Regarding the continual adaptation process, it would be useful to see how performance evolves as more test samples from a target domain are processed. For example, a plot of WER against the number of processed utterances could help illustrate the adaptation dynamics over time. \n\n  **Response:** We reported the WER against the number of process utterances under two variations of the continual adaptation process: (i) T-EMA with resetting, reported in Table 3 in the ablation study; and (ii) T-EMA with exponential averaging (proposed), presented in Table 3 and Section 3.4. We evaluated their performance across varying numbers of samples from 100 to 800 in Table 1 below, using the LibriSpeech dataset with Gaussian noise (σ = 0.015). With the resetting approach, performance increases up to 300 samples and then declines with additional samples, suggesting that resetting CMA‑ES prevents leveraging prior adaptation. In contrast, the proposed exponential averaging method demonstrates relatively stable performance, indicating that even with a small number of samples, reliable adaptation can be achieved. This is likely due to the T-EMA mechanism accumulating knowledge from earlier utterances and updating CMA‑ES in a more favorable region of the search space. The improved performance over the resetting mechanism further demonstrates that our method is robust to the number of samples used for adaptation, and that the proposed T-EMA effectively stabilizes the adaptation process. We will include the results in the appendix.\n\n  **Table 1.** WER against the number of processed utterances with T-EMA (resetting) and T-EMA (proposed).\n\n  | Method                | 100  | 200  | 300  | 400  | 500  | 600  | 700  | 800  |\n  |-----------------------|------|------|------|------|------|------|------|------|\n  | T‑EMA (Resetting)     | 20.4 | 19.7 | 19.8 | 20.4 | 20.8 | 20.8 | 21.1 | 21.3 |\n  | T‑EMA (Proposed)      | 17.9 | 17.0 | 17.2 | 17.4 | 17.6 | 17.5 | 17.8 | 17.7 |\n  | **Performance Gain**  | 2.5  | 2.7  | 2.6  | 3.0  | 3.2  | 3.3  | 3.3  | 3.6  |\n\n---\n\nWe appreciate your time and welcome any further questions or suggestions."}, {"review_id": "ioIsn80Ctr", "reviewer": "Reviewer_LXCm", "summary": "E-BATS is a backpropagation-free Test-Time Adaptation (TTA) method developed for Speech Foundation Models (SFMs). It is designed to adapt effectively to domain shifts that commonly occur in real-world scenarios, such as background noise, speaker accent, and device variation, while also minimizing memory usage. The method inserts prompt vectors into a CNN-based feature encoder, allowing the adjustment of the feature space without changing the model weights. These prompts are optimized without backpropagation using Covariance Matrix Adaptation Evolution Strategy (CMA-ES). To further enhance performance, the authors employ entropy minimization, a multi-scale loss based on both utterance-level and token-level alignment, and a technique called T-EMA. The approach shows strong performance compared to various benchmarks across four datasets and two different models.", "questions": "I wrote my questions in weaknesses.", "limitations": "yes", "rating": 4, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "1. Strengths\n\n- The authors are the first to propose a backpropagation-free test-time adaptation (TTA) method specifically for speech foundation models.\n\n- They support their claims through extensive comparisons across various models, datasets, and methods, as well as comprehensive ablation studies.\n\n- In addition, the figures presented in the paper are clear and intuitive, making the overall methodology easier to understand.\n\n2. Weaknesses\n\n- The CMA-ES method appears to rely on the covariance matrix, which raises the question of whether the computational cost becomes significant as the dimensionality $d$ increases.\n\n- While the overall results in the paper focus on speech recognition, speech foundation models are typically used across a wide range of downstream tasks beyond just ASR. Including results on additional tasks would make the method appear more broadly applicable and universal.\n\n- As someone less familiar with backpropagation-free TTA methods, I am also curious about the temporal dynamics of speech. Since speech signals often exhibit varying characteristics across frames, would it be possible to design the prompts to vary across frames as well, rather than keeping them fixed?", "was_revised": true, "final_justification": "Given that my score was already on the positive side, I will retain it as is.", "rebuttal": "## Response to Reviewer\nWe thank the reviewer for the detailed feedback. We greatly appreciate the reviewer's recognition of our work as offering the \"first\" backpropagation-free Test‑Time Adaptation framework tailored for speech foundation models with \"strong performance\" and \"extensive comparisons\". Below, we address the reviewer's concerns regarding the computational cost of CMA-ES with dimensionality *d*, applicability beyond ASR, and the applicability of designing prompts to vary across frames. We will include these results and analysis in the appendix of the revised version.\n\n---\n\n- **W1:** The CMA-ES method appears to rely on the covariance matrix, which raises the question of whether the computational cost becomes significant as the dimensionality *d* increases.\n\n  **Response:** We appreciate the reviewer’s concern regarding the computational cost of CMA-ES with increasing dimensionality *d*. Indeed, the full-rank CMA-ES algorithm has a per-iteration complexity of *O*(*J* *d*² + *d*³), where *J* is the number of sampled solutions (i.e., prompts in our case). To mitigate this cost, we design our prompt vector to have a fixed dimensionality of 512, aligned with the embedding size commonly used in speech foundation models, ensuring that *d* remains tractable in practical scenarios. Furthermore, As shown in table 1, with our grid search of J, we observe the optimal J always much smaller than d, in our configuration, we fix J=50, which remains significantly smaller than *d*. This ensures that the asymptotic complexity does not exceed *O*(*d*³). Thus, while CMA-ES theoretically scales cubically with *d*, our design choices effectively cap the computational overhead in the context of high-dimensional but fixed-size prompt vectors.\n\n  **Table 1.** WER (%) of Wav2Vec2ForCTC on the _caf-real_ condition of CHiME-3 under different CMA-ES candidate sizes *J*  \n  _Iteration steps = 25_\n\n  | *J*   | 10   | 20   | 30   | 40   | 50   | 60   | 70   | 80   | 90   | 100  |\n  |-----|------|------|------|------|------|------|------|------|------|------|\n  | WER (%) | 41.8 | 39.3 | 38.3 | 38.3 | 37.9 | 38.0 | 37.9 | 37.9 | 38.0 | 37.8 |\n\n---\n\n- **W2:** While the overall results in the paper focus on speech recognition, speech foundation models are typically used across a wide range of downstream tasks beyond just ASR. Including results on additional tasks would make the method appear more broadly applicable and universal.\n\n  **Response:** We further applied E-BATS to speech emotion recognition using the IEMOCAP benchmarking dataset and the `SpeechBrain/emotion-recognition-wav2vec2-IEMOCAP` model. Emotion prediction accuracy improved from 38.8% (source model without adaptation) to 43.4% (with E-BATS) under additive Gaussian noise (σ = 0.02). This shows that our E-BATS framework can generalize to other speech tasks. We will include these experiments in the appendix of the revised version.\n\n---\n\n- **W3:** As someone less familiar with backpropagation-free TTA methods, I am also curious about the temporal dynamics of speech. Since speech signals often exhibit varying characteristics across frames, would it be possible to design the prompts to vary across frames as well, rather than keeping them fixed?\n  \n  **Response:** This is an interesting point raised. While we recognize that speech signals exhibit varying characteristics over time, especially across extended durations, our current approach is based on the practical assumption, common in domain adaptation for speech, that distribution shifts (such as speaker accent or background noise) tend to affect an utterence relatively uniformly. For example, a speaker’s accent or background noise typically remains stable over a typical 10-second utterance, resulting in a coherent distribution shift within that utterence. This justifies our use of a global prompt.\n\n  That said, we also acknowledge that learning frame-wise prompts could potentially capture finer-grained variations, but this is technically challenging: speech utterances naturally vary in length, so adopting frame-wise prompts would require padding or truncating to a uniform length and flattening a much larger prompt tensor for optimization (e.g., with CMA-ES), which substantially increases complexity and decreases optimization stability. Nevertheless, varying the frame-wise prompt remains an interesting direction and will be considered in the future.\n\n---\n\nWe appreciate your feedback and are happy to provide further clarification if needed."}]

Author rebuttal:

Meta-review: This paper addresses the problem of backpropagation-free test-time adaptation, which the AC considers a highly practical and necessary direction for broad real-world applications of TTA. The proposed approach builds on the solid foundation of FOA and extends it to speech foundation models.

The overall novelty and technical contributions are acknowledged by four of the five reviewers, as well as the AC. Although Reviewer uVzm raised some concerns, he/she did not provide detailed follow-up feedback after the rebuttal. The AC carefully reviewed the paper and examined the author's response to the raised issues by Reviewer uVzm. The AC finds that most concerns have been addressed and therefore recommends acceptance of this paper.

The authors are kindly reminded to incorporate the new rebuttal results and discussions into the final version.
Decision: Accept (poster)