Self-Taught Recognizer: Toward Unsupervised Adaptation for Speech Foundation Models
Authors: Yuchen Hu
Chen Chen
Chao-Han Huck Yang
Chengwei Qin
Pin-Yu Chen
EngSiong Chng
Chao Zhang
Venue: NeurIPS 2024 poster (2024)
Keywords: Automatic speech recognition
speech foundation models
unsupervised domain adaptation
Primary area: speech_and_audio

Abstract: We propose an unsupervised adaptation framework, Self-TAught Recognizer (STAR), which leverages unlabeled data to enhance the robustness of automatic speech recognition (ASR) systems in diverse target domains, such as noise and accents. STAR is developed for prevalent speech foundation models based on Transformer-related architecture with auto-regressive decoding (e.g., Whisper, Canary). Specifically, we propose a novel indicator that empirically integrates step-wise information during decoding to assess the token-level quality of pseudo labels without ground truth, thereby guiding model updates for effective unsupervised adaptation. Experimental results show that STAR achieves an average of 13.5% relative reduction in word error rate across 14 target domains, and it sometimes even approaches the upper-bound performance of supervised adaptation. Surprisingly, we also observe that STAR prevents the adapted model from the common catastrophic forgetting problem without recalling source-domain data. Furthermore, STAR exhibits high data efficiency that only requires less than one-hour unlabeled data, and seamless generality to alternative large speech models and speech translation tasks. Our code aims to open source to the research communities.

Self-Taught Recognizer: Toward Unsupervised Adaptation for Speech Foundation Models
Yuchen lst
Chen Chen
Chao-Han Huck Yang"
Chengwei Qin
Pin-Yu Chen
Eng Siong Chng' Chao Zhang'
Nanyang Technological University 2NVIDIA Research
IBM Research "Tsinghua University
{yuchen005, chen1436) .nt.. edu.sg, huckyånvidia.com com
Abstract
We propose unsupervised adaptation framework, Self-TAught Recognizer (STAR). which leverages unlabeled data to enhance the robustness of automatic speech recognition (ASR) systems in diverse target domains, such as noise and accents. STAR developed for prevalent speech foundation models based on Transformer-related architecture with auto-regressive decoding (e.g., Whisper, Canary; SeamlessM4T), Specifically, we propose a novel indicator that empiri- cally integrates step-wise information during decoding assess the token-level quality of pseudo labels without ground truth, thereby guiding model updates for effective unsupervised adaptation. Experimental results show that STAR achieves an average of 13.5% relative reduction in word error rate across 14 target do- mains, and it sometimes even approaches the upper- bound performance of su- pervised adaptation. Meanwhile, we observe that STAR prevents the adapted model from the catastrophic forgetting problem without recalling source-domain data. Furthermore, STAR exhibits high data efficiency that only requires less than one-hour unlabeled data, and seamless generality alternative large speech models in recognition and translation tasks. Our code is publicly available at: betpp://giithbb com/YUCHEEOO5//TTAA Adapt
Introduction
Human speech. characterized by its inherent acoustic nuances [70] and variability across speakers 27]. is further complicated by the diverse and unpredictable environments. These factors contribute to significant domain distinctions the speech signal, with differences accent, speaking style, and background noise (visualized in Appendix[B] Consequently. this diversity poses significant challenges in the field automatic speech recognition (ASR), especially under diverse conditions (51).
In recent years, advancements in ASR technology (30 84 12 73 have been boosted, primarily by the of deep neural models and supervised learning with high-quality datasets. In particular, end-to-end ASR models pre-trained on industry-scale datasets have been made publicly available the research community, such as OpenAI Whisper (73), Meta SeamlessM4T 4 and NVIDIA Canary [7I]. Considering the high diversity of speech domains, even a well-trained ASR foundation model usually performs less satisfactorily when encountering domain shift problem [49 85 This performance degradation stems from a critical dilemma: collecting and labelling sufficient training data the target domain immensely time-consuming and labour-intense the domain adaptation process of ASR models. Some existing efforts [32] 44) focus on leveraging labelled source domain and unlabeled target domain data enhance the ASR performance, as shown
thus hindering
Equal Contribution.
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

Unlabeled speech
Unlabeled speech only (source-free)
UDA
M
Pseudo labels
UDA with * STAR
Labeled speech
STAR
(source)
M
Speech Model
Large Speech Model
Selected samples
Figure 1: Illustration of unsupervised domain adaptation (UDA) and source-free UDA frameworks. (i) UDA problem. (ii) Source-free UDA by self-training. STAR works by selecting high-quality pseudo labels and guiding the ASR foundation model's adaptation at the token level.
in Fig. ()). This solution is generally known as unsupervised domain adaptation (UDA)
24
32
44
and has been widely explored in both machine learning and speech processing communities.
when encoun- tering an unfamiliar speech domain is first illustrated. Despite the unawareness of the ground truth labels of our heard speech, individuals can learn speech-to-text mapping from their self-directed transcriptions, particularly when they have high confidence (see Fig. (). This learning mechanism has parallel in machine learning, known as "self- sself-training" [75] 90 41), which typically involves two stages. First, pre-trained model generates the pseudo labels on target-domain data. Then, these data with pseudo labels, along with the associated confidence levels, are used to adapt the model.
In the context of the UDA problem in ASR, the human "self-directed" ability [3 17
Meanwhile, unlike approaches in existing ASR literature, which often require the source data (data used to pre-train the ASR model in source domains) to achieve UDA 663 [1) humans, as the gold standard of speech communication, can address UDA issues in ASR without requiring any source data. Considering the exhibited generality of the speech foundation models with Transformer-related architectures based on the attention mechanism, it is the opportune moment to centre the attention mechanism on addressing the source-free UDA problem within the realm of ASR. Specifically, we study to adapt the pre-trained Whisper model using small amount of unlabeled data from the target domain to become a domain-specifit speech recognizer in different scenarios without using any source data, based on the process analogous to the human speech recognition as shown in Fig. I (ii). We hereby highlight the significant potential value that research on cource-free UDA contributes to general ASR applications 33 (i) It circumvents the extensive computational resources by adapting the ASR models without using any source data. (ii) It can considerably improve ASR performance in the target domain using only a small amount of speech samples without ground-truth labels.
In this work, we propose a source-free UDA approach called Self-TAught Recognizer (STAR), which aims to enhance the performance of speech foundation models in specific target domains with unlabeled data. Based on the typical self-training scheme [92). STAR delves deeply into a general issue: Given the absence of ground-truth labels, how do we assess the quality of pseudo labels for guiding self-training? Unlike humans who can intuitively gauge their confidence in listening, the decoding "confidence scores" from attention-based ASR models are typically approximated by the pseudo posterior probabilities from softmax function (53] which may be unreliable due to the well-known over-confident issue of softmax (37]. Traditionally with HMM-based ASR, the confidence scores can be estimated based on lattice and confusion network data structures [18 61 91]
which, however, are difficult to obtain effectively the end-to-end ASR framework.
In pursuit of a better quality indicator, we explore the self-attention matrix obtained during auto- regressive decoding, as it is not only grounded on speech input but also focuses on linguistic acceptability (36). Specifically, we find the aggregated attention weights can be a more reliable indicator for measuring the quality of ASR-decoded tokens than the confidence scores. However, such an attentive score suffers from numerical instability, as recent findings 82 62 from linguistic perspectives, is normal for equally correct words (e.g.. prepositions and nouns) to receive different semantic roles in text sentence. This leads to the sub-optimality of using attentive scores alone to guide the fine-tuning process. We first substantiate these observations experimentally and then, in our STAR method, propose novel integration approach based on their distinct characteristics, resulting a both stable and reliable STAR indicator. Finally, it employed to guide the subsequent finetuning process in re- re-weighting manner, making a specific form of instructive adaptation.
2

---

Our experiments evaluate the proposed STAR in various practical scenarios, including background noise, speaker accents, and specific scenarios (e.g., interviews and talks). Comprehensive results show the significant gains from STAR that enhances Whisper by an average of 13.5 % relative word error rate (WER) reduction across 14 target domains. On some corpora, unsupervised STAR even approaches the upper bound of supervised adaptation using real labels. We also surprisingly observe that with informed finetuning, STAR prevents the adapted models from the common catastrophic forgetting problem without recalling source-domain data. Furthermore, we demonstrate that STAR enjoys: (i) remarkable data efficiency: it requires less than one hour of unlabeled data to adapt Whisper to its best performance target domains; (ii) seamless generality: it is applicable to many prevalent speech foundation models and can be easily extended to the speech translation task.
In general, our contributions are summarized as follows:
We direct our focus on source-free UDA in ASR with as one setting closed to real- world applications, where only a pre-trained speech foundation model and unlabeled speech samples are required to adapt to specific target domains.
We present score-based self-training approach called STAR that includes a novel indicator evaluate the pseudo-label quality and achieve informed finetuning, which significantly enhances the domain-specifie capabilities of speech foundation models across a wide range of target domains, including noise, accent, and specific scenarios.
Intensive experiments demonstrate that STAR effectively avoids the common catastrophic forgetting problem in adaptation. Our further analysis of data efficiency and generality shows its potential for real-world applications, such as incremental updates for voice assistant.
2 Related Work
Unsupervised Domain Adaptation in ASR. Since acquiring the ground truth speech transcriptions is often prohibitively expensive in the target domain, many existing efforts bootstrap from available
out-of-domain data to build an improved target domain model 779 59 93 Besides directly simu-
lating the target domain speech ( adversarial learning is frequently utilized to learn invariant representations mitigate domain shifts [33 23), which is also applied for front-end speech en- hancement [64]. Mcanwhile, teacher-student learning provides an alternative solution for efficient
adaptation (50 65]. These methods are also semi-supervised [86], since labels from the source domain are available. More recently, self-supervised pre-trained models (e.g. wav2vec2 (2 have been used for pseudo-labelling to achieve unsupervised adaptation 44 38].
Source-free Unsupervised Domain Adaptation. Given the potential presence of sensitive infor- mation in the source data 778), there is a high demand for source-free UDA methods that transfer
pre-trained source model to the unlabeled target domain without any source data [47 (66] 16] As a
long- long-discussed machine learning issue, the mainstream solutions include self-supervised knowledge distillation (57). contrastive learning (35), hidden structure mining [89], and eeeertainty-guiiee adap- tation (20). Considering the inherent uncertainty in ASR decoding, we focus on the latter category and briefly review some representative indicators of uncertainty. Recently, there are some works [9] suggesting measuring uncertainty by the predicted variance from Monte Carlo Dropout 42 utilizing aleatoric uncertainty by encouraging intra-domain consistency [48]. performing pseudo- denoising using soft label correction [87) and introducing self self-entropy descent mechanism to find a threshold for pseudo-labeling be dated back for decades, starting by using lattices and confusion networks [18 61 for HMM-based systems. Improved confidence estimation can be achieved by model-based approaches, such as condi- tional random fields 766,, recurrent neural networks [40, 774 and graph neural networks [52]. More recent effforts (60 77 focus on predicting uncertainty for regressegee decoding of attention-based models, however, they have not been applied in the source-free UDA.
[54). It is worth noting that, confidence estimation for ASR systems can
Summary. Given the large amount of data used to pre-train the speech foundation models, it is difficult to define the scope of its source domain and keep the source data for re-training. Therefore we believe it is necessary to directly adapt speech foundation models to target domains for UDA for speech tasks. The proposed STAR method aims to assess the quality of pseudo labels produced by the auto-regressive decoding process, which leads to an instructive and effective self-training process. Since STAR can remove the need for keeping and retraining with source data and considerably reduce the performance difference between using ground truth and pseudo labels for adaptation with target
3

---

domain data samples, it has the potential to fulfil the goal of source- free UDA for the ASR task and achieve user-friendly deployment for real-world speech-based artificial intelligence products.
3 Methodology
3.1 Problem Setup
ASR Formulation. An end-to-end ASR system relies on neural model f to recognize the input speech x E & into the corresponding text transcription y E IR2, where 7 and denote the lengths of the input waveform and output text sequences respectively. During training, the model f is optimized by teacher-forcing [46] with cross-entropy loss:
LASR (2r,
log Joglooullun :
(I)
where VI.I. denotes the tokens in ground-truth labels y,, and 0 denotes the trainable parameters in
UDA Setting. Given source ASR model f((*) trained on labelled source domain data
D(*), domain adaption in ASR aims transfer the learned knowledge and obtain model f(() that performs well target domain D(4), i.e., rt) x² - yxtt, UDA is required if ground-truth labels
()))) are not available. Source-free UDA [99 [51 posts more challenging but practical scenario, where the source data {x((s) ()(())))())))))))))))))))))))))) used to pre-train the ASR is no longer available in adaptation. That is, only speech inputs (²()) is available when adapting the source model f(+) to the target domain D(t), Self-training Strategy. source-free UDA, since a source model itself typically generates pseudo-labels, some previous works (80) have referred to this learning approach semi-supervised learning. To distinguish it from unsupervised domain adaptation, in this paper, we refer to the approach for addressing source-free free UDA as self-training, consistent with the terminology used in studies [92]. Specifically, we adopt the pipeline of psendo-labeling and informed finetuning, First, \N(() Ji=1 fed into source model f(() to generate the
((()) unlabeled speech segments (t) {x
pseudo labels corresponding to each of them, which are denoted as ))()) = {v/ J ii=1
( Then, the
paired dataset with the speech inputs and their newly-generated pseudo labels (x((1) ))(t) are used
to finetune the source model to the target domain based on the self-training loss LSt:
Σ (a .00 (0)))
i=1
where the ASR loss LASR follows the definition in Eq.
Summary. Since self-generated pseudo labels [63] do not introduce extra supervised information to the ASR source model, simply repeating this process unlikely to yield performance improve- ments (75). However, if high-quality pseudo labels selected domain-specific exemplars to inform the speech foundation model, it would then update in a direction beneficial to the target propose critical research question: How
domain performance. Therefore, quality of pseudo labels using an indicator that can also guide the model's 's update? The subsequent content of this section will delve into detailed discussion from both token and utterance levels.
assess the
3.2 Token-level Assessment and Re-weighting
The auto-regressive decoding in ASR can provide step-wise information predicted tokens, which can be used for loken-level uncertainty assessment (7). More importantly, this information can guide the subsequent training process: assigning different weights to each token when calculating the CE loss in (2. namely informed finetuning.
Why is confidence not a good indicator? The confidence score denotes the highest value among the posterior probability predicted by a neural model. In auto-regressive decoding, the I-th step of token confidence score C₁ can be denoted as:
Ci max P(úi (1011111:111 1:1, O"))
(3)

---

Confidence Score
Atleniive Soore
0.27
0.60
0:40
0.31
0.08
dence scom
Atlendive
Variance
CEP
11.0, 10, 1,12 05 02,02 02 M, 12 M, 12,
12. 12, 12 12, 12 11, 12 02 1.0 12
Auz.
[14, 19,21, 18,1,,8,, 18,12 04 03,04
12,
10, 14, 13.
0..
0.800 08,0 0.0 08, 0.0 0.0. 0.04
Pseuda label:
Phose
who
Wooe
For the
red
and
bobe
board
Bhat
thore
Tree
not
been
Brasstentia
Kasa
of housing
this your
0.2
0.03
Figure 2: (Left): An example of pseudo label, ground-truth transcription, confidence scores, attention matrix and attentive scores. (Right-Up): Confusion matrix of confidence and attentive scores, where the y-axis denotes the pseudo token is correct or wrong, and the x-axis denotes the corresponding score is high or low (with 1 as the threshold, more analysis is in Fig. 6). that the diagonal values indicate the score's reliability in assessing the quality of pseudo-label. (Right-Down): Variance of the scores of correct and wrong pseudo tokens.
By preserving the C for each token during pseudo-labeling, we can perform informed finetuning with a re-weighting loss as follows:
(E, Σ log
Cy.
However, substantial body of existing research 8 indicates that confidence does not accurately reflect predictive accuracy, especially in auto-regressive decoding [60]. In Eq. (4) the prediction of the current token is influenced by previously predicted tokens ÿt 1t-1-1 which can easily lead to error accumulation and propagation. We further inspect this claim in Whisper by empirical observation. As shown in Fig. (Right-Up), we employ a confusion matrix to visualize the relationship between confidence score and pseudo-label quality, which shows that 52% correct tokens are assigned low confidence and 60% of wrong tokens are assigned high confidence (more discussion is in Appendix C). Therefore, confidence cannot reliable pscudo-label quality indicator alone, like discussed in [21].
Is attentive score a better indicator? We explore if the self-attention matrix W obtained during auto-regressive decoding can reflect the pseudo-label quality. Unlike C₁ defined in Eq. (3). W has direct association with X and linguistic acceptability [72). which means that it might be less influenced by the variability of speech input (see example Fig.2).
Empirical Observation. Starting from the fourth row and fourth column (first 3 tokens are fixed prompts: "[[en]) ([transcribel) (notimestamps))
for the correctly decoded tokens (black), the attention weights are concentrated on the diagonal and partially fall on other pseudo tokens. However, for wrongly decoded tokens (red), the attention weights almost all fall on the second column that corresponds to the task prompt token "( "[Itranscribel) (highlighted in red boxes). To quantify this finding into numerical metric, defined an "aggregate pattern" indicator called attentive score, which is highlighted in the orange box in Fig. 2 and formulated as:
A: =
W1
Σ W.,s.
(5)
1=1
(=/+1
where At indicates the global semantic correlations between pseudo token d with all tokens { (first 3 tokens are task prompt). Specifically, we add the second term to also consider the attention weights with respect to future tokens, in order to capture the comprehensive global context to better assess the role of current token (see Table ablation study). We compare the values of attentive score A; and confidence score C₁ for this sentence in Fig. 2 (Left). As marked by black boxes, C provides unreliable assessments for both 'board' (correct but low Ci) and 'year. ([eox))((wrong but high Ct). In comparison, A, can accurately reflect the correctness of these tokens. To avoid randomness, we analyze CHiME-4 test-real and plot a confusion matrix Fig. 2 (Right-Up). evident that. compared to C, our A; more reliably assesses the quality of predicted tokens.
5

---

Despite reliability, A exhibits less numerical stability, e.g., "for" and "housing" are both correct tokens but their A, are distinct (1.8 vs. 0.8). The underlying reason is that their roles in the global context as prepositions and nouns are indeed different 82 62
However, when we try to use this A to guide the ASR loss re-weighting like Eq. these labels are expected to be assigned comparable weights as they are equally correct. We verify this finding with the variance of Ai and C₁ in Fig. 2 Biigh,-Down.. For both correct and wrong tokens, Ar exhibits higher variance, indicating it may not be suitable to guide the finetuning in a re-weighting manner directly.
STAR Indicator: Reliable and Stable. To integrate the advantages of C and Ay, we introduce a new indicator that balances reliability and stability. Specifically, in cases where C and Ay exhibit conflicting values toward pseudo token, we would select A, as an indicator that shows higher reliability. It can be mathematically formulated as:
Stne = |o /C: -
+ +
where a denotes the sigmoid function (3) = 1/(1++-- and here it simulates the step function to capture the cases of conflicting scores. Our definition of conflict is Ai/ /C, larger than a hyper- parameter threshold A. This criterion can be decoupled into two terms, Ar and Ay/Ci. which means large attentive score as well as a large gap between attentive and confidence scores" Similarly, Ci /Ai is another case of conflicting scores, and we add them up to simulate the logical "OR" operation.
On the other hand, if A and C present consistent assessment towards pseudo token, Cj would be used to scale A using its stability. Specifically, we design a soft interpolation strategy inspired by focal loss 56 to integrate them:
S.m = Io(A AP/Ci) (?/A)))))) * A * (Cc--Ao)/
Similarly, we also use Sigmoid function to simulate the non-conflicting cases, where we multiply the two terms to denote logical "AND". Inspired by the smoothing technique in focal loss [56). we propose to leverage the gap between two scores for scaling Ar ( (c)/T where 7 is temperature.
During the subsequent informed finetuning stage, combine the two indicators above to guide the training process in a re-weighting manner, and (4) should be re-written as:
L
EASR (z, 0) = Σ log Po 011 S₁; where S = Sqpnn + S
(8)
1=1
As a result, the STAR scores are both reliable and stable as shown in Fig. 5 which serves as a better quality indicator guide the informed finetuning (see Algorithm I in Appendix for details).
3.3 Utterance-level Filtering
The utterance-level filtering aims to remove those predicted utterances with low overall quality since they probably harmful for subsequent adaptation. We now introduce several existing approaches to assess the utterance-level quality of pseudo labels, which are often used for uncertainty estimation. Notably, high uncertainty usually implicates low quality for the generated sequence.
conduct multiple times of stochastic forward decoding with activated dropout to get a list of predictions [9]. Then the list with large variance is considered to have high uncertainty and should be removed from subsequent training. However, this method does not apply to Whisper it does not use dropout in training. As alternative, introduce similar method for assessing utterance-leved uncertainty. Specifically, given an input speech z, we first implement one forward decoding and set the result ÿ as the base transcription. Then, we randomly disturb the model weights of Whisper with Gaussian noise, and repeat the forward decoding for K times, resulting in list of pseudo transcriptions (ūv } Thereafter, we calculate the edit distance (ED) between pseudo transcription 0k and the base transcription ÿ. which indicates the impact of disturbance on Whisper decoding. Then, the model's robustness in transcribing speech x can be calculated as:
Monte Carlo Sampling
K
1
U 1)
ED U.).
K
'To avoid special cases like two tiny scores where one is many times of another (e.g., 0.01, 0.001).

---

Table 1: Main WER (%) results of the proposed STAR adaptation and baselines in various ASR domains. "Whisper (frozen)" denotes the zero-shot performance without adaptation. "Whisper "elff-train.) is the vanilla self-training scheme consisting of pseudo-labeling and finetuning. Based on that, "UTT adds utterance-level filtering explained in $3.3, and "TOKyownight performs two token-level re-weighting explained in $3.2 "Whisper (real label)" is supervised learning with real (ground truth) labels and can be viewed as the upper-bound performance of source- fareee-free UDA.
Whisper (frozen)
Whisper (self-train.)
reweeeigght Ci
STAR (ours)
Whisper (real label)
Testing Scenario
UTT fiter
At
Background Noise
test-real
6.8
6.9
6.4
6.5
6.2
6.0
11.8%
5.2
CHiME-4
test-simu
9.9
10.1
9.7
9.8
9.5
9.4
5.1%
8.7
dev-real
4.6
4.5
4.3
4.3
4.1
3.9
15.2%
3.2
dev-simu
7.0
7.0
6.6
6.7
6.6
6.4.-865% 8.0%
5.9
babble
40.2
37.6
35.0
33.5
31.3
30.2
24.9%
27.2
LS F--FreSSound
airport
15.6
15.5
15.2
15.3
15.0
14.8
-5.1%
14.5
car
2.9
3.0
2.8
2.8
2.6
2.5
2.4
RATS
radio
46.9
47.2
46.0
45.3
44.9
44.6
4.9%
38.6
Speaker Accents
African
6.0
5.8
5.5
5.4
5.0
4.8 20.0%
4.6
Australian
5.8
5.7
5.6
5.5
5.2
5.1 12.1%%
4.3
Common
Indian
6.6
6.5
6.3
6.4
6.1
6.0 9.1%
5.7
Singaporean
6.5
6.2
5.8
5.8
5.4
5.1 21.8%
4.9
Specific Scenarios
TED-LIUM
TED talks
5.2
4.9
4.7
4.8
4.3
4.1 21.2%
3.6
SwitchBoard
telephone
13.3
13.0
12.7
12.3
11.9
11.7
11.7-12.0%
9.9
LRS2
BBC talks
7.6
7.9
7.0.%
5.6
ATIS
airline info.
3.6
3.5
3.3
3.3
2.9
19.4%
2.0
CORAAL
interview
21.5
21.3
20.8
20.7
20.4
20.1
66.5%
17.9
Table WER (%) results regarding catastrophic forgetting. "Frozen" denotes Whisper zero-shot without adaptation. "Self-train" denotes the self-training baseline. "STAR" model is adapted to CHiME-4 using STAR; then evaluated on other domains. More results are Table
LS-FreeSound airport car
Common Voice
Model
RATS
TED-3
SWBD
ATIS
babble
af
au
sg
Frozen
40.2
15.6
2.9
46.9
6.0
5.8
6.6
6.5
5.2
13.3
3.6
Self-train. STAR
38.2
16.6
2.9
47.3
6.4
5.9
6.7
6.3
5.3
13.7
3.4
33.3
15.7
2.8
46.1
6.1
5.8
6.7
5.6
5.0
13.5
2.9
After obtaining pseudo labels, we can examine their diversity to further assess the model's un- certainty. If there are many repetitions in the list, indicates that the model is more confident in transcribing speech æ. Therefore, we utilize a scaling factor that is equal to the utterance amount after de-duplication. The final utterance-level quality is combined by the numeric multiplication of I and U (zx ú). which is then used to rank the N, pseudo data samples, and top a% samples are re- moved due large data uncertainty. Additionally, we also implement beam search decoding and consensus decoding [61] basclines as alternative utterance-level filtering approaches for comparison, where more experimental results and discussions are presented in Appendix
4 Experimental Setup
4.1 ASR Domains
We introduce STAR in various ASR domains to verify its general effectiveness, including noisy speech, accented speech, and specific scenarios. First, for noisy speech we use the CHiME-4 [83], LibriSpeech--FreeSSee [69), and RATS (26) datasets, which covers wide range of noise types including bus, cafe, pedestrian area, street junctions, babble, car, airport, and the challenging radio communication noises. Second, select four typical accents from the Common Voice [1 dataset,

---

Table 3: Case study of an accented speech CV-in (ID: 197_197353. The wrong tokens are highlighted in red. Variance indicates the stability of different scores. "NCE" denotes normalized cross-entropy, where a higher value indicates better measure quality (more results are in Fig. 5
Metric
Content
Variance
NCE Score
Ground-truth
they are organised by scientific themes.
Pseudo label
they are organised by scientific teams.
CI-I
[0.81, 0.88, 0.98, 1.21, 1.13, 1.17, 0.82]
0.023
0.671
ALLL
[1.47, 1.49, 0.95, 1.20, 0.79, 0.43, 0.67]
0.101
0.146
Sa. (ours)
[1.39, 1.40, 0.91, 1.14, 1.03, 0.41, 0.73]
0.058
0.322
Table 4: WER (%) results of STAR with differ- ent speech foundation models on CHiME-4 reat. More models datasets are evaluated Table and and
Table 5: BLEU results of STAR on speech translation task with FLEURS [4] test sets.
+ En
Baseline
Self-train.
STAR
Real
Model
Baseline
Self-train.
STAR
Real
Ar
21.9
22.1
23.3
24.5
Whisper-V23 V3-1.5B
6.8
6.9
6.0
5.2
33.7
34.0
35.9
36.5
Whisper-Med-O.85
8.9
8.8
10.1%%
7.1
Fs
23.9
24.1
24.8
26.4
W3.S--V31-11 1OB
8.4
8.1
7.5-10.75
6.5
Fa
16.6
16.3
17.641.0
19.0
Canary-1 1,0B
8.2
8.0
7.2
6.4
22.4
22.5
23.4.10
24.4
Borakeet-TDT--L..
8.0
7.8
7.0
12.5%
6.2
16.3
16.3
17.140.
17.9
including African, Australian, Indian, and Singaporean accents. Finally, we also evaluate our approach under some specific scenarios, including BBC talks (LRS2 [13), TED talks (TED-LIUM 3 (29]).
telephone conversation (SwitchBoard (25]), interview conversation (CORAAL 43), ) and airline
information consultation (ATIS (28)) More details about the datasets are presented in Appendix F
4.2
Configurations
We use the Whisper-Large V3 model for main experiments, which contains 1.5 billion parameters trained on 680k-hour web-scale data. It is fine-tuned using Adam optimizer [45] with an initial learning rate 1e for 2 epochs. The batch size is set with 16 gradient accumulation steps. For hyper-parameters, the threshold À is set to 2 and the temperature is 10. In addition, the percentile of utterance-level filtering is 20, which shows consistent effectiveness across di different datasets.
Results and Analysis
5.1 Effectiveness of STAR
To examine the effectiveness of STAR, we conduct comparative experiments across various domains and report the WER results in Table
Main Results. From noise adaptation results on CHiME-4, LS LS-FreeSound, and RATS, we observe that: (i) STAR enhances Whisper in all noise scenarios, reducing the WER up to 24.9% relatively. Specifically, on the challenging RATS dataset with pseudo labels 46.9% WER, our STAR can still produce a 4.9% relative improvement. (ii) For some domains, e.g., "airport" and "car", STAR can even approach the upper-boure bound performance by supervised learning. This demonstrates that even with unlabeled data only, our method can effectively adapt Whisper specific target domains. From results on other domains, we observe that: (i) STAR consistently improves the accented ASR to approach the supervised upper bound. (ii) Whisper does not perform well in some colloquial scenarios (SwitchBoard and CORAAL) as the spoken language tends to be informal and less grammatically correct, which leads to poor-quality pscudo labels and then influences our adaptation performance.
Analysis of Catastrophic Forgetting. Table 2 analyzes the potential forgetting issue of our method by evaluating the WHiME-4-finetune model on other datasets. Surprisingly, contrary to the common catastrophic forgetting issue that commonly happens in traditional source-free ASR adaptation, our STAR approach can even improve the performance in other domains. We speculate that under the self-training scheme, the pseudo label is generated by the model itself, so that it may avoid the model from over-fitting to samples with vastly different data distributions [10] Furthermore, compared to vanilla self-training, STAR can better highlight the high-quality pseudo tokens for informed finetuning, which may help improve the model's general ASR ability. More detailed analysis are in S
8

---

CHIME-4 (lest-real)
CV (African)
TED-LIUM 3
ATIS
6.0
0.
3.6
58
4.6
6.4
0.36
52
0.31 h
0.82 h
4.0
0.
100
200
500
1K
2<
9.0k
100 200
500
OK
7.5k
100
200
500
100
200
500
4k
Figure 3: WER (%) results with different numbers of unlabeled training samples. The minimum required data amount (in hours) to obtain the best performance is highlighted in the star mark.
Analysis of Indicators. Table also presents the performance of different indicators in the informed finetuning. First, we observe that the utterance- filtering yields some effects by removing bad training samples. Then, for token-level re- weighting, the two pseudo-label quality indicators both improve the performance, where the attentive score performs better due to higher reliability. Our proposed STAR indicator achieves the best result by integrating the strengths of both scores. We also use a case in Table to illustrate, where exists a wrong pseudo token "teams". The confidence score fails to reflect this error while the attentive score succeeds, but the latter suffers from less numerical stability (i.c., large variance). By integrating their strengths, our STAR score achieves both reliability and stability in assessing the pseudo-label's quality. In addition, we also calculate the NCE metric to show the better quality of our proposed STAR and attentive scores than traditional confidence scores.
5.2 Generality of STAR
Generalization to Different Speech Foundation Models. To further evaluate the generalization abil- ity of our approach, extend STAR to different foundation models, including OWSM, Canary and Parakeet-TDDT that take top places in the HuggingFace ASR leader-board Consistent performance gains (i.e., over 10% relative WER reduction) these models has verified the excellent generality of STAR. In addition, it also works well on relatively small models like Whisper-Medium.coe en-.0.88..
Generalization to Speech Translation (ST) Task. Apart from ASR, we also investigate another widely studied speech task, the ST task, to further verify the generality of STAR adaptation. As shown in Table results various FLEURS X->En tracks illustrate average of over 1.2 BLEU improvements (2.2 BLEU for De Dn) It shows the good potential of our STAR adaptation on other sequence-to se-suuene tasks besides ASR, which could lead to more extensions for future work.
5.3
Ablation Study
In this section, conduct ablation studies to analyze STAR from perspectives of data (Fig. 3 model (Table and finetuning approaches (Table which provide a constructive reference for deploying ASR foundation models in practical scenarios using STAR. More analysis are in Appendix
Data Efficiency. We explore the requirement of unlabeled data amount (Nv:) for STAR adaptation. Fig. 3 shows the WER results on four datasets with different numbers of training utterances. Surpris- ingly, only 200 to 500 sentences (less than 1-hour unlabeled speech data) are required to achieve the optimal effects, which cost around 0.8-hour training time on single NVIDIA-A NVIDIA-A100-40GE GPU. This remarkable data efficiency significantly saves the labours in real-world applications: not only is there no need for manual labelling, but the collection of unlabeled data also requires less than one hour.
Model Size. Table @able/reporte reports the performance on CHiME-4 lest-real that applies STAR to the Whisper family with different model sizes. Results show that our STAR adaptation works well on difference scales of foundation models. Specifically, the promising performance gains on light model (base.en) conditions, such as mobile devices.
implicates the potential of STAR in practical resource-construine
Finetuning Approach. Considering that adapting speech foundation models with small amount of data might risk over-fitting, we explore the impact of different finetuning approaches in Table TO We observe that both regular finetuning (full, encoder-only, decoder-only) and efficient finetuning methods (LoRA) yield similar effectiveness, which provides flexible choices under different settings.
http:://wuulliggeeccc/////////tp leaderbbbad

---

6 Conclusion
We propose STAR, a nource-free UDA method that effectively adapts the speech foundation models to various target domains with unlabeled data. Specifically, STAR introduces a novel indicator to assess the pseudo-label quality and then instructively guide the finetuning of the model. Our experiments verify STAR's is efficacy on ASR tasks across wide range of target domains including noise, accent, and specific scenarios, and it even approaches the upper-bound performance of supervised adaptation on some corpora. Furthermore, we observe that STAR can avoid the catastrophic forgetting problem that is often suffered by models adapted without recalling only requires less than one hour of unlabeled data to achieve average of 13.5% relative WER reduction across 14 domains, and it also shows seamless generality to speech translation tasks. This enables to deploy speech systems in real-world scenarios rapidly and conveniently.
data. Furthermore, STAR
Acknowledgement
This research is supported by the National Research Foundation, Singapore, under its Al Singapore Programme grant number recommendations expressed in this material are those of the author(s) and do not reflect the views of National Research Foundation, Singapore. The computational work for this article was partially performed resources of the National Supercomputing Centre, Singapore (htps://www.nscc.sg))
Any opinions, findings and conclusions or
References
[1] Rosana Ardila, Megan Branson, Kelly Davis, Michael Henretty, Michael Kohler, Josh Meyer, Reuben Morais, Lindsay Saunders, Francis M Tyers, and Gregor Weber. Common voice: A massively-muttiinguall speech corpus. arXiv preprint arXiv: 191206661 2019.
[2] Alexei Baevski, Yuhao Zhou, Abdelrahman Mohamed, and Michael Auli. wav2vec 2.0:
A framework for self-supervised learning of speech representations. Advances in Neural Information Processing Systems, 33: 12449-12460, 2020.
[3] Janet M Baker, Li Deng, James Glass, Sanjeev Khudanpur, Chin- Hui Lee, Nelson Morgan, and Douglas O' Shaughhesss. Developments and directions speech recognition and understanding. Part 1 [DSP Education). IEEE Signal Processing Magazine, 26(3):75-B) 2009.
[4] Loic Barrault, Yu-An Chung. Mariano Coria Meglioli, David Dale, Ning Dong, Mark Dup- penthaler, Paul-Ambroise Duquenne, Brian Ellis, Hady Elsahar, Justin Haaheim, et al. Seamless: Multilingual expressive and streaming speech translation. arXiv preprint arXiv:23 2023.
[5] Peter Bell, Joachim Fainberg, Ondrej Klejch, Jinyu Li, Steve Renals, and Pawel Swietojanski. Adaptation algorithms for neural network-bart based speech recognition: An overview. IEEE Open Journal of Signal Processing, 2:33 66, 2020.
[6] David Chan, Austin Myers, Sudheendra Vijayanarasimhan, David Ross, and John Canny. Ic3: Image captioning by committee consensus. In Proceedings of the 2023 Conference Empirical Methods Natural Language Processing, pages 8975-9003, 2023.
[7] Chen Chen, Nana Hou, Yuchen Hu, Shashank Shirol, and Eng Siong Chng. Noise-robust speech recognition with 10 minutes unparalleled in-domain data. In Proc. ICSSP, pages 4298-4302, 2022.
[8] Chen Chen, Yuchen Hu, Chao-Har Huck Yang. Sabato Marco Siniscalchi, Pin- Yu Chen, and Ensiong Chng. Hyporadise: An open baseline for generative speech recognition with large language models. In seirth- Conference Neural Information Processing Systems Datasets and Benchmarks Track, 2023.
[9] Cheng Chen, Quande Liu, Yueming Jin, Qi Dou, and Pheng-Ann Heng. Source-free domain adaptive fundus image segmentation with denoised pseudo-labeling In Proc. MICCAI, pages 225-235 2021.
10

---

[10] Sanyuan Chen, Yutai Hou, Yiming Cui, Wanxiang Che. Ting Liu, and Xiangzhan Yu. Recall and learn: Fine-tuning deep pretrained language models with less forgetting, arXiv preprint arXiv:2004. 12651, 2020.
[11] Sanyuan Chen, Chengyi Wang, Zhengyang Chen, Yu Wu, Shujie Liu, Zhuo Chen, Jinyu Li, Naoyuki Kanda, Takuya Yoshioka, Xiong Xiao, et al. Wavlm: Large-scale self- sule-sueereiee pre- training for full stack speech processing. IEEE Journal of Selected Topics in Signal Processing, 1666):1505-1518, 2022.
[12] Chung-Cheng Chiu, Tara N. Sainath, Yonghui Wu, Rohit Prabhavalkar, Patrick Nguyen, Zhifeng Chen, Anjuli Kannan, Ron J Weiss, Kanishka Rao, Ekaterina Gonina, al. State-of-the-art speech recognition with sequence-to-sequeece models. In Proc. ICASSP, pages 4774-4778, 2018.
[13] Joon Son Chung, Andrew Senior, Oriol Vinyals, and Andrew Zisserman. Lip reading sentences in the wild. In Proc. CVPR, pages 3444-3453, 2017.
[14] Alexis Conneau, Min Ma, Simran Khanuja, Yu Zhang, Vera Axelrod, Siddharth Dalmia, Jason Riesa, Clara Rivera, and Ankur Bapma. Fleurs: Few-shot learning evaluation of universal representations of speech. In Proc. SLT, pages
2023.
[15] Jun Deng, Zixing Zhang, Florian Eyben, and Bjöm Schuller. Autoencoder- bassd unsupervised domain adaptation for speech emotion recognition. IEEE Signal Processing Letters, 21(9))1068- 1072, 2014.
[16] Ning Ding. Yixing Xu, Yehui Tang, Chao Xu, Yunhe Wang, and Dacheng Tao. Source-free domain adaptation via distribution estimation. In Proc. CVPR, pages 7212-7222, 2022.
[17] Fengning Du. Student perspectives of self-directed language learning: Implications for teaching and research. International Journal for the Scholarship of Teaching and Learning, 7())24, 2013.
[18) Gunnar Evermann and Phil Woodland. Posterior probability decoding, confidence estimation and system combination. In Proc. S7W, pages 78-81, Baltimore, 2000.
[19] Yuqi Fang, Pew-Thian Yap, Weili Lin, Hongtu Zhu, and Mingxia Liu. Source-free unsupervised domain adaptation: A survey. Neural Networks, page 106230, 2024.
[20] Francois Fleuret et al. Uncertainty reduction for model adaptation in semantic segmentation. In Proc. CVPR, pages 9613-9623, 2021.
[21] Marina Fomicheva, Shuo Sun, Lisa Yankovskaya, Frédéric Blain, Francisco Guzmán, Mark Fishel, Nikolaos Aletras, Vishrav Chaudhary, and Lucia Specia. Unsupervised quality estimation for neural machine translation. Transactions of the Association for Computational Linguistics, 8:539-555, 2020.
[22] Frederic Font, Gerard Roma, and Xavier Serra. Freesound technical demo. In Proc. ACM MM, pages 41 411- 2013.
[23] Carlos Franzreb and Tim Polzehl. Domain adversarial training for German accented speech recognition. In Proc. DAGA, pages 1413-1416, 2023.
[24] Yaroslav Ganin and Victor Lempitsky. Unsupervised domain adaptation by backpropagation. In Proc. ICML, pages 1180-1189, 2015.
[25] John J Godfrey, Edward C Holliman, and Jane McDaniel. SwitchBoard: Telephone speech corpus for research and development. In Proc. ICASSP, pages 517-520, 1992.
[26] David Graff, Kevin Walker, Stephanic M Strassel, Xiaoyi Ma, Karen Jones, and Ann Sawyer. The RATS collection: Supporting HLT research with degraded audio data. In Proc. LREC, pages 1970-1977, 2014.
[27] John Hansen and Taufiq Hasan. Speaker recognition by machines and humans: tutorial review. IEEE Signal Processing Magazine, 32(6):74-99, 2015.

---

[28]
Charles T. Hemphill, John J. Godfrey, and George R. Doddington. The ATIS spoken language systems pilot corpus, In Proc. WSNL, Hidden Valley, 1990.
[29] François Hernandez, Vincent Nguyen, Sahar Ghannay, Natalia Tomashenko, and Yannick Esteve. TED-LIUM 3: Twice as much data and corpus repartition for experiments on speaker adaptation. In Proc. SPECOM, pages 198-208, 2018.
[30] Geoffrey Hinton, Li Deng. Dong Yu, George E Dahl, Abdel-rahman Mohamed, Navdeep Jaitly, Andrew Senior, Vincent Vanhoucke, Patrick Nguyen, Tara N. Sainath, al. Deep neural networks for acoustic modeling in speech recognition: The shared views of four research groups.
IEEE Signal Processing Magazine, 29(6):82-97, 2012.
[31] Ehsan Hosseini-Asl, Yingbo Zhou, Caiming Xiong, and Richard Socher. A multi-discriminaoor CycleGAN for unsupervised non-parallel speech domain adaptation. In Proc. Interspeech, pages 3758-3762, 2018.
[32] Wei-Ning Hsu, Yu Zhang, and James Glass. Unsupervised domain adaptation for robust speech recognition via variational autoencoder-based data augmentation. In Proc. ASRU, pages 16-23, 2017.
[33] Hu Hu, Xuesong Yang, Zeynab Racesy, Jinxi Guo, Gokce Keskin, Harish Arsikere, Ariya Rastrow, Andreas Stolcke, and Roland Maas. reDAT: Accent-invariant representation for end- to-end asr by domain adversarial training with relabeling. In Proc. ICASSP, pages 6408-6412, 2021.
[34] Yuchen Hu, Chen Chen, Chao-Han Huck Yang, Ruizhe Li, Chao Zhang, Pin-Yu Chen, and EnSiong Chng. Large language models are efficient learners of noise-robust speech recognition. arXiv preprint 4rX4v:2401.10446, 2024.
[35] Jiaxing Huang, Dayan Guan, Aoran Xiao, and Shijian Lu. Model adaptation: Historical contrastive learning for unsupervised domain adaptation without source data. Advances in Neural Information Processing Systems, 34:3635 3649, 2021.
[36] Qidong Huang, Xiaoyi Dong, Pan Zhang, Bin Wang, Conghui He, Jiaqi Wang, Dahua Lin, Weiming Zhang, and Nenghai Yu. Opera: Alleviating hallucination in multi-modal large language models via over-trust penalty and retrospection-allocatie In Proc. CVPR, 2024.
[37] Shuangping Huang, Yu Luo, Zhenzhou Zhuang, Jin-Gang Yu, Mengchao He, and Yongpan Wang. Context-aware selective label smoothing for calibrating sequence recognition model. In Proc. ACM MM, pages 4591-4599 2021.
[38] Dongseong Hwang, Ananya Misra, Zhouyuan Huo, Nikhil Siddhartha, Shefali Garg, David Qiu, Khe Chai Sim, Trevor Strohman, Françoise Beaufays, and Yanzhang He. Large-scale ASR domain adaptation using self-and semi-supervised learning. In Proc. ICASSP, pages 6627 6631, 2022.
[39] Renaud Jardri, Delphine Pins, Maxime Bubrovszky, Pascal Despretz, Jean-Pierre Pruvo, Mare Steinling, and Pierre Thomas. Self awareness and speech processing: An fMRI study. Neurolm- age, 35(4):1645-1653, 2007.
[40] Kaustubh Kalgaonkar, Chaojun Liu, Yifan Gong, and Kaisheng Yao. Estimating confidence scores on asr results using recurrent neural networks. 2015 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 4999-5003. IEEE, 2015.
[41] Uday Kamath, John Liu, James Whitaker, Uday Kamath, John Liu, and James Whitaker. Transfer learning: Scenarios, self-taught learning, and multitask learning. Deep Learning for NLP and Speech Recognition, pages 463-493, 2019.
[42] Alex Kendall and Yarin Gal. What uncertainties do we need Bayesian deep learning for computer vision? Advances in Neural Information Processing systems, 30:1-11, 2017.
[43] Tyler Kendall and Charlie Farrington. The corpus of regional African American language. version 1.77 eugene, or: The online resources for african american language project, 2021.
12

---

[44] Sameer Khurana, Niko Moritz, Takaaki Hori, and Jonathan Le Roux. Unsupervised domain adaptation for speech recognition via uncertainty driven self-training. In Proc. ICASSP, pages 6553 6557, 2021.
[45] Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In Proc. ICLR, pages 1-13, 2015.
[46] John F. Kolen and Stefan C. Kremer. A field guide dynamical recurrent networks. John Wiley & Sons, 2001.
[47] Jogendra Nath Kundu, Naveen Venkat, R Venkatesh Babu, et al. Universal source-free domain adaptation. In Proc. CVPR, pages 4544-4553, 2020.
[48] JoonHo Lee and Gyemin Lee. Feature alignment by uncertainty and self-training for source-free unsupervised domain adaptation. Neural Networks, 161:682-692, 2023.
[49] Jinyu Li, Li Deng, Yifan Gong, and Reinhold Haeb-Umbach. An overview of noise-robust automatic speech recognition. IEEE/ACM Transactions Audio, Speech, and Language Processing, 22(4):745-777, 2014.
[50] Jinyu Li, Michael L Seltzer, Xi Wang, Rui Zhao, and Yifan Gong, Large-scale domain adaptation via teacher-student learning. Interspeech 2017, 2017.
[51] Qi Li, Jinsong Zheng, Qiru Zhou, and Chin-Hui Lee. Robust, real-time endpoint detector with energy normalization for asr in adverse environments. Proc. ICASSP, volume 1, pages 233-236, 2001.
[52] Qiujia Li, PM Ness, Anton Ragni, and Mark JF Gales. Bi-directional lattice recurrent neural networks for confidence estimation. In ICASSP 2019-2019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 6755-6759, IEEE, 2019.
[53] Qiujia Li, David Qiu, Yu Zhang, Bo Li, Yanzhang Hc, Phil Woodland, Liangliang Cao, and Trevor Strohman. Confidence estimation for attention- based sequence-to-sequene models for speech recognition. In Proc. ICASSP, pages 3388-6392 2021.
[54] Xianfeng Li, Weijie Chen, Di Xie, Shicai Yang, Peng Yuan, Shiliang Pu, and Yueting Zhuang. free lunch for unsupervised domain adaptive object detection without source data. In Proc. AAAI, volume 35, pages 8474-8481, 2021.
[55] Jian Liang, Dapeng Hu, and Jiashi Feng. Do we really need access the source data? Source hypothesis transfer for unsupervised domain adaptation. In Proc. ICML, pages 6028-6039, 2020.
[56] Tsung-Yl Yi Lin, Priya Goyal, Ross Girshick, Kaiming He, and Piotr Dollár. Focal loss for dense object detection. In Proc. ICCV, pages 2980-2988, 2017.
[57] Xinyu Liu and Yixuan Yuan. A source-free domain adaptive polyp detection framework with style diversification flow. IEEE Transactions Medical Imaging, 41(7):1897-1908, 2022
[58] Yi Luan, Daisuke Saito, Yosuke Kashiwagi. Nobuaki Minematsu, and Keikichi Hirose. Semi- supervised noise dictionary adaptation for exemplar-based noise robust speech recognition. In Proc. ICASSP, pages 1745-1748, 2014.
[59] Han Ma, Qiaoling Zhang, Roubing Tang, Lu Zhang, and Yubo Jia. Robust speech recognition using teacher-student learning domain adaptation. IEICE Transactions on Information and Systems, 105(12)2112-2111 2022.
[60] Andrey Malinin and Mark Gales. Uncertainty estimation in autoregressive structured prediction. In Proc. ICLR, pages 1-31, 2021.
[61] Lidia Mangu, Eric Brill, and Andreas Stolcke. Finding consensus in speech recognition: word error minimization and other applications of confusion networks. Computer Speech &L Language, (4(4):3334400 2000.
13

---

[62] David Mareček, Hande Celikkanat, Mikka Silfverberg, Vinit Ravishankar, and Jörg Tiedemann. Are multilingual neural machine translation models better at capturing linguistic features? The Prague Bulletin of Mathematical Linguistics, pages 143-162, 2020.
[63] Zhong Meng, Zhuo Chen, Vadim Mazalov, Jinyu Li, and Yifan Gong. Unsupervised adaptation with domain separation networks for robust speech recognition. In Proc. ASRU, pages 214-221, 2017.
[64] Zhong Meng, Jinyu Li, Yifan Gong, et al. Adversarial feature-mapping for speech enhancement. In Proc. Interspeech, pages 2259-3263, 2018.
[65] Zhong Meng, Jinyu Li. Yong Zhao, and Yifan Gong. Conditional teacher-student learning, In 2019.
Proc. ICASSP, pages
[66] Gaurav Kumar Nayak, Konda Reddy Mopuri, Saksham Jain, and Anirban Chakraborty. Mining data impressions from deep models as substitute for the unavailable training data. IEEE Transactions Pattern Analysis and Machine Intelligence,
2021.
[67] Vassil Panayotov, Guoguo Chen, Daniel Povey, and Sanjeev Khudanpur. Librispeech: An ASR corpus based on public domain audio books. In Proc. ICASSP, pages 5206-5210, 2015.
[68] Yifan Peng, Jinchuan Tian, William Chen, Siddhant Arora, Brian Yan, Yui Sudo, Muhammad Shakeel, Kwanghee Choi, Jiatong Shi, Xuankai Chang, et al. OWSM v3.1: Better and faster open whisper-style speech models based on e- e-branchformer. arXiv preprint arXiv:2401.16658, 2024.
[69] Archiki Prasad, Preethi Jyothi, and Rajbabu Velmurugan. An investigation of end-to-end models for robust speech recognition. In Proc. ICASSP, pages 6893-688 2021.
[70] Graham Pullin and Shannon Hennig. 17 ways to say yes: Toward nuanced tone of voice in AAC and speech technology. Augmentative and Alternative Communication, 31(2):170-180, 2015.
[71] Krishna C. Puvvada, Piotr Zelasko, He Huang, Oleksii Hrinchuk, Nithin Rao Koluguri, Somshubra Majumdar, Elena Rastorgueva, Kunal Dhawan, Zhehuai Chen, Vitaly Larukhin, Jagadeesh Balam, and Boris Ginsburg. New standard for speech recognition and translation from the nvidia nemo canary model. 2024.
[72] Randolph Quirk and Jan Svartvik. Investigating linguistic acceptability. 2019.
[73] Alec Radford, Jong Wook Kim, Tao Xu, Greg Brockman, Christine McLeavey, and Ilya Sutskever. Robust speech recognition via large-scale weak supervision. International Conference Machine Learning, pages 28492-28518. PMLR, 2023.
[74] Anton Ragni, Qiujia Li, Mark J.F. Gales, and Yongqiang Wang. Confidence estimation and deletion prediction using bidirectional recurrent neural networks. In Proc. SLT,, pages 204-211, 2018.
[75] Rajat Raina, Alexis Battle, Honglak Lee, Benjamin Packer, and Andrew Y Ng. Self-taught learning: Transfer learning from unlabeled data. In Proc. ICML, pages 759-766, 2007.
[76] Matthew Stephen Seigel, Phil Woodland, et Combining information sourees for confidence estimation with CRF models. In Proc. Interspeech, pages 905-908, 2011.
[77] Yuxin Shi and Yuhong Sheng. Uncertain quantile autoregressive model. Communications in Statistics-Simulatie and Computation, pages 1-21, 2023.
[78] Serban Stan and Mohammad Rostami. Domain adaptation for the segmentation of confidential medical images. arXiv preprint arXiv:2101.00522. 2021.
[79] Sining Sun, Binbin Zhang, Lei Xie, and Yanning Zhang. An unsupervised deep domain adaptation approach for robust speech recognition. NeuroComputing, 257:79-87, 2017.
14

---

[80] Samuel Thomas, Michael L. Seltzer, Kenneth Church, and Hynek Hermansky. Deep neural network features and semi-supervised training for low resource speech recognition. In Proc. ICASSP, pages 67046670000 6708. 2013.
[81] Juozas Vaicenavicius, David Widmann, Carl Andersson, Fredrik Lindsten, Jacob Roll, and Thomas Schön. Evaluating model calibration in classification. In Proc. ICAIS, pages 3459-3467, 2019.
[82] Jesse Vig and Yonatan Belinkov. Analyzing the structure of attention in a transformer language model. In Proceedings of the 2019 ACL Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP, pages 63-76, 2019.
[83] Emmanuel Vincent, Shinji Watanabe, Jon Barker, and Ricard Marxer. The 4th chime speech sep- aration and recognition challenge. URL: http://spandh. dcs. shef. ac. ukfchime cactiime__aalleeellass accessed on August, 2018), 2016
[84] Shinji Watanabe, Takaaki Hori, Suyoun Kim, John R Hershey, and Tomoki Hayashi. Hybrid etc/attention architecture for end-to-end speech recognition. IEEE Journal of Selected Topics in Signal Processing, 1188):240--253, 2017.
[85] Shinji Watanabe, Michael Mandel, Jon Barker, Emmanuel Vincent, Ashish Arora, Xuankai Chang, Sanjeev Khudanpur, Vimal Manohar, Daniel Povey, Desh Raj. et al. Chime-6 challenge: Tackling multispeaker speech recognition for unsegmented recordings. In CHiME 2020-6th International Workshop on Speech Processing in Everyday Environments, 2020.
[86] Shannon Wotherspoon, William Hartmann, Matthew Snover, and Owen Kimball. Improved data selection for domain adaptation in ASR. In Proc. ICASSP, pages 7018-7022, 2021.
[87] Zhe Xu, Donghuan Lu, Yixin Wang, Jie Luo, Dong Wei, Yefeng Zheng, and Raymond Kai-yu Tong. Denoising for relaxing: Unsupervised domain adaptive fundus image segmentation without source data. In Proc. MICCAI, pages 214-224, 2022.
[88] Chao-Han Huck Yang, Yile Gu, Yi-Chieh Liu, Shalini Ghosh, Ivan Bulyko, and Andreas Stoleke. Generative speech recognition error correction with large language models and task-activating prompting. In 2023 IEEE Automatic Speech Recognition and Understanding Workshop (ASRU), pages 1-8. IEEE, 2023.
[89] Shiqi Yang, Joost van de Weijer, Luis Herranz, Shangling Jui, et al. Exploiting the intrinsic neighborhood structure for foorce-free domain adaptation. Advances in Neural Information Processing Systems, 34:29393-29405, 2021.
[90] David Yarowsky. Unsupervised word sense disambiguation rivaling supervised methods. In Proc. ACL, pages 189-196, 1995.
[91] Kai Yu, Mark Gales, Lan Wang, and Phil Woodland. Unsupervised training and directed manual transcription for LVCSR. Speech Communication, 52(7-8):652-66 2010.
[92] Shuai Zhang, Meng Wang, Sijia Liu Liu, Pin-Yu Yu Chen Chen, and Jinjun Xiong. How does unlabeled data improve generalization in self-training? a one-hidden-layer theoretical analysis. In the Tenth International Conference Learning Representations (ICLR), 2022.
[93] Han Zhu, Gaofeng Cheng, Jindong Wang, Wenxin Hou, Pengyuan Zhang, and Yonghong Yan. Boosting cross-domain speech recognition with self-supervision. IEEE/ACM Transactions Audio, Speech, and Language Processing, 32:471-485, 2023.
A Additional Discussions on the Design of the STAR Framework
Question 1: Is the proposed STAR method limited the Whisper model only?
STAR is a general source-free UDA method that can be compatible with any attention- speech foundation model. To validate this, we also several other models in our experiments, including
15

---

OWSM-V3.1-1.0B 688
Canary-1.OB Parakeet-TDT-I 1.1 and SeamlessM4T- V2---2 4
Table f verify the effective generalization ability of our STAR adaptation method on various state-of- the-art ASR foundation models Furthermore, we also employ the speech translation foundation model SeamlessM4T to verify our generality. presenss the WER results on CHiME-4 test sets with SeamlessM4T, where the STAR adaptation achieves significant improvements over zero-shot and self-training baselines, which even approaches the supervised upper bound. We observed that although the performance of SeamlessM4T-Laue V2 is slightly worse than Whisper Large-V V3 on ASR task, STAR can still achieve up to a 30.7% WER reduction that improves its noise robustness.
Table 6: WER (%) results of STAR with latest speech foundation model, SeamlessM4T-Large- V2 on CHiME-4 test sets.
Test Set
Baseline
Self-train.
STAR (ours)
Real label
test-real
12.3
11.5
9. 1 26.0%
8.7
test-simu
15.2
15.0
13.2 13.2%
13.0
dev-real
8.8
8.3
6.1-30.7% 30.7%
5.8
dev-simu
11.4
11.0
9.2-19.399
19.3%
9.0
Question 2: Can STAR be applied to smaller streaming ASR models like WavLM, RNN-T?
STAR requires large speech models to possess universal robustness across various domains fulfill their role as a reliable pseudo-labeler, where domain -ppciifcc ASR models like WavLM and RNN-T may not work well. To illustrate, the WayLM-Conformer ASR baseline [[T] achieves the state-of- the-art result on the LibriSpeech test-clean dataset (with a WER of only 1.8%). However, when facing domain shifts, such as the CHiME-4 noisy dataset, its zero-shot performance drops 14.4%, and it exceeds 20% on the Common Voice accented dataset. Under these circumstances, it is nearly impossible to use only unlabeled data to adapt this model to the Whisper's 's zero-shot performance (5 7% WER). Therefore, we argue that adding such baselines is not of much reference value. As more general-purpose speech foundation models are released, we prefer to focus our research on studying their decoding behaviors to adapt them more efficiently and conveniently to specific task domains. AableAandFo provide more results on CHiME-4 dataset with more speech foundation models (i.e., OWSM, Canary, Parakeet, M44) to show the good generality of STAR.
Question 3: Is the proposed STAR method limited the ASR task?
Our proposed STAR approach is compatible with any tasks using attention- based encoder-decoder architecture with auto-regressive decoding, the most prevalent framework in many areas not limited to speech and language. Therefore, we believe this work provides useful insights to researchers from other communities who use similar model architectures and need assess the quality of auto- regressive decoders, such as speech translation, audio/image captioning. Table presents the strong results of STAR on speech translation task, which verifies its task generality. However, since this work focuses on ASR task, we would like to leave the evaluation on more tasks to future work. In addition, recent research on LLMs (36) also focuses on the unsmooth self-attention matrix like Fig. Z where they successfully alleviate the LLMs' hallucination problem in auto-regressive decoding through this observation. This evidence indicates the potential impact of our method on other communities.
Question 4: What the difference between STAR with the existing self-training method in ASR?
We summarize the vital difference in the following two points:
Prior works focus on adapting from one source domain to single target domain (e.g., clean to noisy (58)), whereas STAR leverages the universality of Whisper to explore one-to-mant domain adaptation. Although the domain mismatch issue in the latter approach is less severe than in the former, we argue that the baseline performance of the latter is significantly better than that of the former, making improvements more challenging to achieve.
http:///gaggaaaoooood///aaaeeeea v2--ge-
http::/wggggggggeecccco/c///tttt lew..eerrrdddddddrbrbrrrrrddrddd leaderboard
16

---

Although both of them are auto-regressive processes, the decoding of speech foundation models exhibits partially distinct characteristics compared with vanilla ASR decoders in previous works, such as over-confidence phenomena 37 Therefore, STAR adopts an empirical indicator of quality derived from the decoding features of speech foundation models, which can more effectively guide the subsequent finetuning process.
Question 5: What is the scope of efficacy when applying STAR adaptation?
As shown in Table [ STAR can improve the performance with zero-shot WER ranging from 2.9% (car) to 46.9% (radio). However, we observe that the WER improvement in RATS mainly stems from the adjustment of some prepositions and articles, which may not improve the comprehensibility of recognition results fundamentally. Therefore, collecting labeled data for supervised learning is inevitable when the scenarios are quite challenging.
Question 6: Can STAR be involved in adapting test data?
We observe that some works perform unsupervised domain adaptation using unlabeled test data. However, STAR only accesses test data once, and the unlabeled target domain data is drawn from the training of corresponding corpus. We argue this setting more closely aligns with practical scenarios: developers cannot access the test set during the adaptation process. However, they can conveniently collect small amount (see data efficiency Fig. 3 of unlabeled target-domain speech for adaptation, and then deploy the adapted model in testing environments.
Question 7: What about the broader impacts of this work?
This work supports rapid and convenient deployment of ASR applications in real- scenarios, which poses positive societal impact. During training process, we only use publicly available data and pre-trained models, so that our work will not pose explicit negative impact. One thing worth noting that, our algorithm shows good generality and thus might cause abuse in some special occasions (e.g., confidential), we will release code carefully under strict licenses and rules to avoid negative impact.
Question 8: What about the limitations of this work?
Our approach is designed specifically for Transformer-baset speech foundation models, that it may not handle the cases beyond the foundation models, e.g., the unseen languages, unseen tasks, extremely adverse conditions, etc.
B Visualization of Speech Domains Distinction
4 visualizes the spectrograms of parallel clean and noisy speech samples. We can observe clear speech patterns the clean spectrogram (i), while they are significantly contaminated by noise in the noisy spectrograms (ii) and (iii). Specifically, the babble noise corrupts speech signals more than airport noise, where speech patterns are almost completely removed. The reason babble noise contains human speech and thus of sample type significant corruption. These two kinds of noisy speech are reported in Table Overall, the domains of clean and noisy speech are quite distinct from the perspective of pattern recognition.
original speech signal, resulting in
200
100
180
100
(i) Clean
(ii) Airport OdB
(iii) Babble OdB
Figure 4: Spectrograms of parallel clean and noisy speech samples, where we select two noise types for visualization, i..., airport station and babble (used in our experiments). The speech samples are selected from the L.S L.S-FreSSund test and the sample ID is
17

---

C More Discussions of Pseudo-label Quality Indicators
Fig. presents more investigations of the quality indicators. First, from the confusion matrix can observe the higher reliability of the attentive score over the confidence score, which has been discussed in Section 5.1 and proposed STAR score maintains the high reliability of the attentive score by sophisticated integration. Then, from the variance statistics we can observe that the attentive score suffers from less numerical stability, while our proposed STAR score benefits from the high stability of the confidence score. As a result, the STAR score provides a both reliable and stable indicator of the quality of the pseudo label. Furthermore, Whisper model can produce higher-quality pseudo labels, which not only verifies its effectiveness but also explains its potential for iterative adaptation (see Section
note that after STAR adaptation, the
Confusion Matrix
Confusion Matrix
0.48
0.52
0.73
0.27
0.74
0.26
0.54
0.46
0.83
0.17
0.84
0.16
Confidence Score Attentive Score
STAR Score
Confidence Score Attentive Score
STAR Score
Variance
Variance
0.47
0.45
0.4
0.36
0.4
0.35
0.26
0.24
0.20
0.08
0.03
0.01
0.05
Correct
Wrong
Correct
Wrong
NCE
NCE
0.8
0.8
0.77
0.56
0.53
0.46
0.4
0.17
Whisper Baseline
(ii) STAR-Adapted
Figure 5: Confusion matrix, variance and normalized cross-entropy (NCE) of confidence score (orange), attentive score (blue), and our STAR score (green), in terms of the Whisper baseline and our STAR-adapted model. For the confusion matrix, the y-axis denotes the correctness of pseudo tokens, i.e., correct and wrong, and the x-axis denotes whether the corresponding score is high low. Since all scores are normalized via divided by mean value, we set as the threshold to separate them into large and small groups, where more thresholds are analyzed in Fig. 6 NCE a statistical metric to measure the quality of confidence measure, where higher value indicates better measurement.
D More Discussion on Utterance-level filtering
Beam search decoding is widely used strategy in sequence-to-sequeece decoding, which expands the search space to obtain an N-best hypotheses list. Recent research on speech foundation models shows that Nebest results contain rich information 88), where the diversity can reflect the final WER performance [4] 6). Specifically, we utilize the beam search decoding as replacement Gaussian disturbance and obtain an N-best hypotheses list for diversity caleulation. The best hypotheses and beam size N are respectively viewed as base transcription and K, and the quality of utterance is calculated in the same manner as (q. Furthermore, we also introduce an earlier method from conventional ASR, consensus decoding [61], an alternative to investigate the role of
18

---

0.69
0.31
0.58
0.42
0.48
0.52
0.40
0.60
0.33
0.67
Confidence Score
0.76
0.24
0.69
0.31
0.60
0.40
0.29
0.71
0.08
0.92
0.89
0.11
0.83
0.17
0.73
0.27
0.27
0.73
0.05
0.95
Attentive Score
0.42
0.58
0.37
0.63
0.31
0.69
0.27
0.73
0.23
0.77
0.8
0.9
1.0
1.1
1.2
Figure 6: Confusion matrix of confidence score and attentive score terms of different thresholds for separating large and small groups.
Table 7: Ablation study on employing different pseudo tokens to calculate the attentive score in Eq. using CHiME-4 test-real data.
0.70
0.30
0.65
0.35
0.73
0.27
0.33
0.67
0.39
0.31
0.69
Metric
History tokens
Future tokens
Both
History tokens
Future tokens
Both
NCE
0.42
0.37
0.46
Figure 7: Confusion matrix of attentive score calculated using different pseudo tokens.
WER (%)
6.4
6.5
6.2
utterance-level filtering approach, where the utterance quality is estimated similar to the beam search decoding.
Table presents the ablation study of utterance-level filtering strategy. First, compare the K- hypotheses by Gaussian disturbance with the N-best hypotheses by beam search decoding and consensus decoding, where observe that the former performs little better than the latter two. Therefore, we employ Gaussian disturbance for utterance-level filtering in the main experiments. Finally, investigate the role of utterance-level filtering in the entire STAR adaptation, where the results demonstrate its contribution in the final performance gains (second last column).
Table 8: Ablation study of utterance- level filtering in terms of Gaussian disturbance, beam search decoding, and consensus decoding [61]
Whisper (frozen)
Whisper (self-training)
UTT Siter Beam
STAR (ours)
Testing Scenario
Real label
Gaussian
Consensus
w/o UTT
UTT
6.8
6.4
6.6
6.2
6.0-11.8%
11.8%
5.2
CHIME-4
test-sinuu
9.9
10.1
9.7
9.8
9.7
9.7
9.4 6.1%
8.7
dev-real
4.6
4.5
4.3
4.3
4.4
4.0
3.9 15.2%
3.2
dev-sinus
6.6
6.4 6.4_8.8%
5.9
E
Additional Ablation Study
Following Section 5.3 here we would like to present more details about the ablation study on model size and finetuning approaches. In addition, also observe that STAR method can further improve the performance with multiple-round iterative adaptation.
Model Size. Table the performance CHiME-4 test-real that applics STAR to the Whisper family with different model sizes. We can observe consistent and significant improvements on different scales of Whisper models. Specifically, although the light model (base.en) exhibits poor noise robustness, our STAR can bring 45.4% relative improvement. It indicates the potential value of STAR adaptation in practical sesource-constrainee conditions, such mobile devices.
Finetuning Approach. Considering that training large speech models with small amount of data might risk over-fitting, we explore the impact of different tuning approaches for the informed finetuning process this experiment. From Table TO we observe that (i) freezing part of parameters not improve the performance of STAR adaptation. However, finetuning the decoder only is the
19

---

Table 10: WER (%) results of different finetuning
Table 9: WER (%) results of different model sizes CHiME-4 test-real set. "# Param." is the number of model parameters. "Real" denotes Whisper with real label finetuning.
methods on CHiME-4 testrrell. is the number of trainable parameters. "Full" is full finetuning.
"Fnc/Decconly" is encoder/deeooder-only finetune.
Approuch
# Param. *
Baseline
STAR
Real
Model Size
# Param.
Baseline
STAR
Real
Regular Finetuning
large-v3
large-v2
1,550 M
6.8
7.7
6.9 6..41
6.0%1118%
5.2
Full
1550 M
6.0 11.8%
5.2
Enc-only
635 M
6.8
6.3_7.48 7.4%
5.0
large
7.5
7.0 6.75%
6.8
Dec-only
907 M
6.1 6.1_1003%
4.4
medium.cn
769 M
8.9
8.0 1..1
7.1
Parameter-Eainn Efficient Finetuming
small.en
244 M
12.7
10.6-10.8%
9.0
LoRA Reprogram.
16 M
6.0 11 8%
5.1
base.en
74 M
32.4
17.7-15.49
16.1
6.8
0.4
6.7 6.6% 1.5%
6.7
optimal strategy for supervised adaptation. (ii) Using less trainable parameters, LoRA tuning shows commendable results with full finetuning. Nevertheless, LoRA introduces further (e.g., rank) and we found it is sensitive to the learning rate. Considering it slightly decreases the inference efficiency, LoRA tuning is recommended only in situations with limited training resources.
Analysis of Catastrophic Forgetting. Following Table 2 we present more results regarding the forgetting issue on Whisper-Medium.en--e that our STAR can prevent the catastrophic forgetting problem on different-scale speech foundation models. As for the potential reasons, we analyze from three points. First, we observe that the vanilla self-training scheme can also well mitigate the forgetting problem. We can gain some inspiration from previous work [T0] for analysis. Since the pseudo label is generated by the model itself, it may not force the model heavily to over-fit any external data distributions, so that self- leff-training does not degrade the out-of- performance. On the other hand, current prevalent foundation models are usually trained by musk learning, where not all model capacities are used for ASR. Therefore, specific self-training for ASR task may help mitigate the forgetting problem within ASR though with different domains. Furthermore, compared to vanilla self-training, our STAR shows even better performance. The core contribution of STAR is the novel quality indicator that can better highlight the high-quality pseudo tokens for informed finetuning. Considering ASR tasks in different domains, the high-quality pseudo tokens usually correspond to speech frames that are relatively high-quality and easy to recognize. Therefore, highlighting the weights of such pseudo tokens may avoid bringing in low-quality knowledge that may conflict with the existing knowledge embedded in the parameters of the pre-tri rreitraieed speech foundation models, and thus prevent the catastrophic forgetting problem. More study is expected for a deeper understanding of the mechanism behind Considering the focus of this work as well as the space limit, would like to leave this study to future work.
and SeamlessM4TT-2222.55 in Table [1] We observe
Iterability of STAR. As self-training approach, STAR is iterable by repeating the process of pseudo-labeling and informed finetuning. iterations using different model sizes and test sets. In most test sets, multiple iterations of STAR result in further performance improvements. This indicates that while learning from pseudo labels, errors also accumulate, thereby limiting the upper-bound of self-training, Additionally, the enhancement of iteration is relatively larger in smaller models, e.g., 0.7% further WER reduction Whisper-base.
the WER results with different numbers of
F
Dataset Domain Details
For dataset selection, our goal is to cover common scenarios of ASR tasks, which can be grouped into three categories, i.c., background noise, speaker accents, and specific scenarios. Consequently, we collect and employ the following datasets with evident domain characteristics to evaluate our proposed approach. In addition, considering our proposed STAR adaptation is built on self-attention matrix that focuses on global contextual correspondence, we filter out short utterances (i.e., with less than 5 tokens) in some datasets for better and more efficient evaluation.
All the data used in this paper are publicly available and under the following licenses: the Creative Commons BY-NC-ND 3.0 License, Creative Commons BY-NC-ND 4.0 License, Creative Commons BY NC-SA 4.0 License, Creative Commons Attribution 4.0 International License, Creative Commons (CCO) License, the LDC User Agreement for Non-Memhers, the TED Terms of Use, the Yubeu Terms of Service, and the BBC's Terms of Use.
20

---

Table 11: WER (%) results regarding catastrophic forgetting with SeamlessM4T-V2 and Whisper- Medium.en as foundation models. "Frozen" denotes zero-shot performance, "Self-train." denotes the self-training baseline. "STAR" denotes that model is adapted to CHiME-4 dataset using STAR and then evaluated on other domains. This study is an extension of Table
LS-FreeSound airport
Common Voice
Model
RATS
TED-3
SWBD
ATIS
babble
car
af
au
sg
Seamles.M4T
Frozen
54.0
30.1
5.1
92.7
1.8
1.7
0.8
1.1
16.3
25.9
4.3
Self-train.
52.6
30.5
5.4
2.2
1.8
15.5
26.5
4.3
STAR
44.3
28.5
5.1
88.1
1.8
1.7
1.6
1.6
10.7
21.4
3.5
Whisper
Whisspe-MMedimm. Medium. en-0.88
Frozen Self-train.
38.0
22.0
4.4
8.0
6.4
7.4
11.5
14.0
5.6
37.8
21.0
4.5
57.7
8.2
6.3
8.6
7.4
9.8
14.4
5.9
STAR
36.1
19.8
3.8
53.4
7.8
6.0
8.0
7.1
6.4
11.6
4.3
Whisper-Snin Smmlllen-O.
Frozen Self-train.
56.1
32.0
8.1
69.3
8.8
7.4
9.7
8.7
17.4
23.5
6.7
55.3
31.0
7.2
68.3
8.9
7.8
10.1
9.2
15.4
20.2
6.1
STAR
50.7
27.4
4.6
63.0
8.9
7.7
10.1
9.6
7.0
13.8
4.5
Whisper-
WWhispe-Baxeeem--Oe
Frozen
73.7
62.7
17.0
97.4
12.1
11.7
18.0
28.0
42.6
34.2
7.8
Self-train.
69.6
58.8
14.5
96.8
12.5
11.4
17.8
25.1
37.7
28.8
7.0
STAR
62.0
46.8
7.9
94.0
12.6
10.9
17.4
16.5
27.2
17.6
4.9
Table WER (%) results of iterative STAR using different sizes of the Whisper ASR models. "# and STAR adaptation.
Iterations" denotes the number of iterations of
Iterations
Real label
Model
Test set
0
1
2
3
4
5
large-v3 medium.en small.en base.en
6.8
6.0
5.9
5.7
5.7
5.7
5.2
8.9
8.0
7.9
7.9
7.8
7.8
7.1
test-real
12.7
10.6
10.3
10.3
10.3
10.3
9.0
34.4
17.7
17.2
17.2
17.0
17.0
16.1
test-simu
9.9
9.4
9.3
9.0
8.9
8.9
dev-real
4.6
3.9
3.9
3.8
3.8
3.8
3.2
dev-simu
7.0
6.4
6.4
6.4
6.3
6.3
5.9
large-v3
af
6.0
4.8
4.8
4.7
4.7
4.7
au
5.8
5.1
5.0
4.6
4.5
4.5
4.3
in
6.6
6.0
5.8
5,8
5.8
5,8
5.7
sg
6.5
5.1
5.1
5.1
5.1
5.1
F.1 Background Noise
CHiME-4 a popular dataset far-field noisy speech recognition. includes real and simulated noisy recordings in four noisy environments, i.e., bus, cafe, pedestrian area, and street junction. We use its tr05-real split (9,600 utterances) as the target-domain unlabeled training data, as well as the test-real (1,320 utterances), test-simu (1,320 utterances), dev-real (1,640 utterances) and dev-simu(1,66 utterances) splits for testing.
CHiME-4 83
[69] LibriSpeech-Freete FreeSound is a simulated noisy speech dataset for robust speech recognition, which mixes the clean speech data from LibriSpeech train-clean- 100 split [67] and noise data from FreeSound dataset [2 at SNRs of 0, 5, 10, 15, 20, and 25 dB to simulate the noisy speech data. We randomly select 5,000 long utterances (i.e., with more than 5 tokens) from them as the target-domain unlabeled training data. For test set, they select 118 clean speech samples from LibriSpeech test-clean split and mix them with FreeSound noise SNRs of 0, 5, 10, 15, and 20 dB, where select three noise types (i.c., babble, airport, car) at O dB for main experiments.
SbbrSpeech-FreeeSne
21

---

The Robust Automatic Transcription of Speech (RATS) dataset contains radio- communication speech in the ultra high-frequency data category that is extremely noisy and chal- lenging for ASR tasks. Its training data contains 43, 112 noisy speech utterances, where we filter out the low-quality samples and randomly select 5,000 long samples as training set. Its test set contains 7,591 utterances, where we randomly select 1,000 long samples for higher evaluation efficiency.
RATS 26:
F.2 Speaker Accents
Common Voice 5.1 is a freely available dataset for speech recognition. contains speech recordings from diverse speakers in over 60 languages. In this work, we employ the English speech data four different accents, including African, Australian, Indian, and Singaporean. Specifically, we randomly select 7,885 long samples from its train-en split with accent labels, where the training set contains 7,485 samples
Common M
for each accent) and the test set contains 400 samples (100 for each accent). In our experiments, the model is finetuned on the entire training set and then evaluated on each accent individually.
F.3 Specific Scenarios
TED-LIUM 3 [29]: TED-LIUM is dataset of speech recorded from TED Talks in multiple languages. It contains diverse range of background noise, speaker accents, speech topics, etc. To better evaluate our method, we randomly select 6,000 long samples from its train split for our main experiments, where the training set contains 5,000 samples and the test contains 1,000 samples.
SwitchBoard [25]: The SwitchBoard corpus a telephone speech dataset collected from conver- sations between pairs of speakers. It focuses North American English and involves over 2.4k conversations from approximately 200 speakers. We randomly select 5,000 long samples from its Irain split as the training data, and its eva/2000 split the test data.
LRS2 [3:
Lip Reading Sentences (LRS2) a large-scale publicly available labeled audio-visual dataset, which consists of 224 hours of video clips from BBC programs. We randomly select 6,000 long samples from its train split for our main experiments, where the training set contains 5,000 samples and the test set contains 1,000 samples.
ATIS 28]:
Airline Travel Information System (ATIS) is a dataset comprising spoken queries for air travel information, such as flight times, prices, and availability, It contains 3,964 samples in the training set and 809 samples in the test set, which are recorded from over 500 speakers.
CORAAL 43 The Corpus of Regional African American Language (CORAAL) is the first public corpus AAL data. includes audio recordings along with the time-aligned orthographic transcription from over 150 sociolinguistic interviews. We randomly select 2,950 long samples as the training set and 500 samples as the test
Ignore
Unfamilar Speech
Hearing
Learning
Figure 8: Self-training process of humans. "V" denotes the self-generates transcription with high subjective confidence that will thus be selected for subsequent learning.
G Algorithm of STAR Adaptation
22

---

Algorithm Self-Taught Recognizer (STAR) adaptation.
Input: pre-trained speech foundation model f(+), target-domain unlabeled data X {
Output: Target domain ASR model f())
Generate pseudo label ( from x2(0) using f(+). repeat
for = 1 to (9) do
Collect confidence score {C.)-, using Eq.
Calculate attentive score (Ai)f= using Eq. (5).
Calculate STAR indicator St using Eq. (8)
Finetune f(()) with {ax b. and S, using Eq.
8
end for until ASR model / is converged.
NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: Section
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation as long as is clear that these goals attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: Appendix A
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested few datasets with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails handle technical jargon.
23

---

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairesss.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community, Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
Justification: Our work does not include theoretical results.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Justification: Section aadd Appendix F and G
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, descríbing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how replicate the results, access to hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm.
24

---

(b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is new model (e.g.. a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, described in supplemental material?
Answer: [Yes]
Justification:
We have provided the code in supplemental material.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips.cm cc/pu
blic/ Balcc/guies//Cae http:///ooeeummmmiiiioooooooeooe
for more details.
While we encourage the release of code and data, we understand that this might be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https
for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing much information possible in supplemental material (appended to the paper) recommended, but including URLs to data and code is permitted.
6. Experimental Setting/eetails
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: Section andd Appendix G
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented the core the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
25

---

Answer: [No]
Justification:
Our experimental results are stable and do not need such statistical significance.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results accompanied by enror bars, confi- dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification: Section 4 and
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't 't make it into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics
Answer: [Yes]
Justification:
We have reviewed and conformed with the NeurIPS Code of Ethics.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
the authors answer No, they should explain the special cireumstances that require a deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there special consid- eration due to laws or regulations in their jurisdiction).
10. Broader Impacts
26

---

Question:
Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: Appendix A
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, is not needed to point out that generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Question:
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Answer: [NA]
Justification: Our work poses no such risks.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (c.g., code, data, models), used in the paper, properly credited and the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: Section
Guidelines:
The answer NA means that the paper does not existing assets.
27

---

The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset. For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, pnperswithcode com /adtaasseess has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's s creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification:
We have well documented the introduced assets in supplemental material.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification: This paper does involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Bthies, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: This paper does not involve crowdsourcing research with human subjects.
28

---

Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with
human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent)
may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such the institution conducting the review.
29


Reviews:
[{"review_id": "gOKNwkeftY", "reviewer": "Reviewer_vnSx", "summary": "The paper proposes STAR, a novel ASR domain adaptation technique that requires no labeled data and only a few unlabeled samples. STAR utilizes the confidence score and self-attention score obtained during decoder inference to calculate the reliability score (STAR indicator) for each token. The score of each token is then used as a multiplier to adjust the fine-tuning loss, which is based on generated pseudo-labels. In addition, STAR employs utterance-level filtering to remove noisy predictions. Extensive experiments across various ASR models, datasets, and fine-tuning techniques demonstrate that STAR achieves significant accuracy improvements compared to the baseline self-training approach.", "questions": "* The attentive score A (Equation 5) seems to be affected by the total length of the generated transcript. The longer the transcript, the (potentially) higher the attentive score. In contrast, confidence scores are bounded to [0, 1]. Maybe missing a normalization term in Equation 5?\n* Any intuitive reasons not to incorporate cross-attention scores?\n* Comparing Table 1 and 2, the numbers of “Frozen” are the same but “Self-train” and “STAR” are different. For example, TED3 WER is (5.2, 4.9, 4.1) in Table 1 but (5.2, 5.3, 5.0) in Table 2. What’s the difference?\n* How many beams are used in the beam search? Are the beam search-based pseudo-labels also used for self-training baselines?\n* It would strengthen the paper’s claim if the pseudo-label and STAR score could be cross-transferred between different models (for example, Whisper-Large generates training resources for Whisper-Tiny).", "limitations": "* The paper adequately addresses the limitations.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "* The logical flow of this paper is very interesting, and the authors provide empirical evidence for each step. The motivations behind the research and method are inspiring.\n* The authors have conducted comprehensive experiments using multiple datasets, models, and noisy conditions. I appreciate the authors’ effort on this.\n* STAR does not seem to suffer from catastrophic forgetting, and this is a very important advantage.", "weaknesses": "* The proposed method is designed for a Transformer-based model with auto-regressive generative decoder architectures. As such, it may not be easy to adopt STAR for CTC or RNN-T-based ASR models (as the authors also discussed in Appendix A).\n* It would be important to discuss the differences and similarities between STAR and noisy student-based training [1][2]. NST also employs pseudo-labeling, heavy filtering, and iterative refinement.\n  * [1] Pushing the Limits of Semi-Supervised Learning for Automatic Speech Recognition\n  * [2] Improved Noisy Student Training for Automatic Speech Recognition", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely appreciate Reviewer vnSx for valuable and constructive comments. Your suggestion is also instructive for further analysis and our future work.\n\nPlease find detailed responses below:\n\n\n- ***Q1: Appling STAR for CTC or RNN-T-based ASR models.***\n\n   Thanks for your feedback, As the illustration in section 3, applying STAR to CTC or RNN-T, auto-regressive Whisper is still employed as a pseudo-labeler to provide pseudo transcription, as well as token-level or utterance-level indicators. Therefore, the RNN-T model can be adapted based on the provided information. We acknowledge that the process requires compositional efforts, but this connection to popular ASR variants also has high potential application value.\n\n- ***Q2: Discussion of the differences and similarities between STAR and noisy student-based training [1][2].***\n\n   Thanks for your feedback. We will add a paragraph of related discussion to the paper. Although both STAR and NST contain a pseudo labeling and filtering process, they have several distinctions listed as follows:\n\n   - **Motivation:** STAR focuses on domain adaptation, and NST focuses on leveraging abundant unlabeled data to improve general ASR accuracy, where the unlabelled data are from similar domains (LibriSpeech and LibriLight datasets).\n   - **Data:** STAR only requires **1 hour** of unlabeled data from any target domain, NST requires both labeled data (960 hours) and abundant unlabeled data (60k hours). \n   - **Method:** STAR focuses on the token-level indicators for informed finetuning (which could be attributed to the strong linguistic ability of the Whisper decoder), and NST focuses on utterance-level data filtering.\n\n- ***Q3: Missing a normalization term in Equation 5.***\n\n   Thanks for your valuable question. When we obtain the attention matrix, it is normalized according to the sequence length. We will clarify  this point with a note. \n\n- ***Q4: Why not use the cross-attention matrix?***\n\n   Thank you for this discussion. As we described in the paper, self-attention in the decoder better reflects the **linguistic acceptability** of the predicted transcription. As for cross-attention, it is more directly linked to the acoustic representation–vulnerable to variable speech domain. In contrast, self-attention is text-based and Whisper's multi-task pre-training also includes text-only learning objectives. Therefore, self-attention potentially gives more linguistic knowledge. Our preliminary results also confirmed that the cross-attention matrix could not provide a reliable indicator.\n\n- ***Q5: Difference between Table 1 and Table 2.***\n\n   Thanks for your question. Table 1 shows the main results where Whisper is both finetuned and tested on each domain individually. Table 2 aims to explore the forgetting issue of STAR, where Whisper is finetuned on the CHiME-4 dataset and then tested on other datasets. Therefore, Table 2 presents worse results than Table 1. We will clarify this in the captions of the tables.\n\n- ***Q6: How many beams are used in the beam search? Are the beam search-based pseudo-labels also used for self-training baselines?***\n\n   Thanks for your question. For beam search, we set beam$=5$. Our preliminary results show that beam search-based pseudo labels show almost the same self-training performance as the greedy search ones. Therefore, for higher efficiency, we use greedy search only.\n\n- ***Q7: ​​Can Whisper-Large generate training resources for Whisper-Tiny?***\n\n   Thanks for the question. Yes, we confirm that STAR can distill knowledge from large models to smaller models (but not vice versa). We will include this advantage in the next version.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "PyZdAEUb5q", "reviewer": "Reviewer_LswZ", "summary": "This paper investigates the use of audio-only data to enhance ASR performance for domain adaption in Speech Foundation Models. The approach is straightforward: recognition results are used to compute a confidence score for each token (e.g., BPE in this paper), which then weights the loss function, as shown in Equation (4). The key innovation is the computation of token-level confidence scores by adopting attention scores instead of the softmax output for confidence estimation. Unlike traditional methods that operate at the utterance level, this approach focuses on token-level confidence. The concept is simple: use the attention weight as the confidence score rather than the predicted next-word probability. The proposed STAR method combines information from both token-level probabilities and the attention matrix. ASR performance appears promising, especially when compared to the strong baseline model Whisper large v3.", "questions": "I have listed my questions in the above section.", "limitations": "N/A", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. I find the proposed confidence estimation using attention scores intriguing.\n\n2. The authors conducted extensive experiments on various ASR corpora to demonstrate the effectiveness of the proposed STAR algorithms, along with comprehensive ablation studies.\n\n3. Significant performance improvements are reported over the strong baseline, which is Whisper large v3.", "weaknesses": "1. Several details are missing. How is the attention matrix calculated given multiple layers and heads? Are the attention weights from cross-attention or self-attention used?\n2. Do you need to tune the threshold for each corpus, or simply use a fixed threshold for all datasets? What is the specific value for the threshold lambda?\n3. The definition of conflict is A_l^2/C_l, which seems intuitive. Is there any particular reason for using this formulation?\n4. Many studies have attempted to use entropy from the output layer instead of probability for confidence estimation in end-to-end ASR systems.\n5. I am interested in whether the informed fine-tuning shown in Equation (4) is proposed in this paper. The confidence score is used to filter utterances and weigh the importance of each word. Do you have any ablation studies demonstrating the effectiveness of this approach?\n6. It would be better to combine Equations (6) and (7) into an equation array to clearly state the formulation.\n7. In Equation (5), the attention score is computed using the attention weight for the current word to the previous words and from future words to the current word. Is there any particular reason for this choice? Is it acceptable for the A_l to be larger than 1?\n\nOverall, this paper seems to rely heavily on empirical or intuitive settings, making it difficult to extend to a theoretically sound approach. Additionally, many key details are missing. It would be appreciated if the authors could provide more details and present the information more clearly.\n\nI am open to reconsidering my rating based on the authors' rebuttal.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate Reviewer LswZ for valuable and constructive comments, and we believe our detailed responses below can solve your concerns on missing details. Please let us know if you have further questions or recommendations. \n\n- ***Q1: Several missing details: Are the attention weights from cross-attention or self-attention used? How is the attention matrix calculated given multiple layers and heads?***\n\n   Thanks for your question. We use self-attention weight as we introduced in the paper (**line 65** and **line 180**). \n\n   In our experiments, the attention matrix is obtained by averaging all heads in the last transformer layer, as the high layers learned more linguistic semantics. We also found that the performance is not sensitive to the selection of layers or heads. The last two layers, with a single head, can also achieve comparable performance.  More details can be found in our submitted code.\n\n- ***Q2: Do you need to tune the threshold for each corpus, or simply use a fixed threshold for all datasets? What is the specific value for the threshold lambda?***\n\n   Thanks for your question. We apply a fixed threshold of lambda$=2$ for all corpus and it works well generally, as mentioned in **line 264**.\n\n- ***Q3: The definition of conflict is $A_l^2/C_l$, which seems intuitive. Is there any particular reason for using this formulation?***\n\n   Thanks for your question. The design of $A_l^2/C_l$ can be decoupled into two terms, $A_l/C_l$ and $A_l$, which means to match the “conflict” criterion, we not only require $A_l$ to be much larger than $C_l$, but the absolute value of $A_l$ should also be large. This is to avoid special cases with two small scores where one is many times the other (e.g., $A_l = 0.1, 0.01$). We have mentioned this particular design in **line 212 to 214** and **footnote 2**.\n\n   The specific equation definition of “conflict” is not unique, if only the above condition is met roughly.\n\n- ***Q4: Many studies have attempted to use entropy from the output layer instead of probability for confidence estimation in end-to-end ASR systems.***\n\n   Thanks for your question. Entropy is indeed an alternative choice for confidence estimation. However, in our case, since the probability $P$ cannot reliably indicate pseudo-label quality (Figure 2, Figure 5, and Figure 6), the minus entropy $P*logP$ that is positively correlated to $P$ cannot serve as a reliable indicator either. Our preliminary results have confirmed this, but considering $P$ is a more typical and intuitive choice, we only select $P$ as the confidence score baseline.\n\n- ***Q5: I am interested in whether the informed fine-tuning shown in Equation (4) is proposed in this paper. The confidence score is used to filter utterances and weigh the importance of each word. Do you have any ablation studies demonstrating the effectiveness of this approach?***\n\n   Thanks for your question. The informed finetuning with confidence score in Eq. (4) is followed from prior works as discussed in the Introduction (**line 59 to 62**). However, we observe that the confidence score is unreliable in assessing pseudo labels so we propose a reliable attentive score based on the self-attention matrix. Finally, we combine conventional confidence and proposed attentive score as our final STAR indicator, which guides the token-level reweighting. On the other hand, utterance-level filtering is implemented by Monte-Carlo Sampling introduced in Section 3.3.\n\n   For experiments, Table 1 presents the ablation study of utterance-level filtering, and token-level reweighting (confidence score, attentive score, STAR score).\n\n- ***Q6: It would be better to combine Equations (6) and (7) into an equation array to clearly state the formulation.***\n\n   Thanks for your suggestion and careful check, we will revise it in the next version.\n\n- ***Q7: In Equation (5), the attention score is computed using the attention weight for the current word to the previous words and from future words to the current word. Is there any particular reason for this choice? Is it acceptable for the A_l to be larger than 1?***\n\n   Thanks for asking this question. For the choice of history and future tokens in calculating the attentive score (Eq. (5)), we use both of them to capture the comprehensive global context to better assess the role of the current token, in terms of its semantic significance in the entire sentence. Table 7 presents an ablation study on this choice, indicating that both history and future tokens are helpful. The $A_l$ will be normalized using sequence length after calculation so that the absolute value of $A_l$ does not matter, we will add these details in the next version.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "bFlcAPMttI", "reviewer": "Reviewer_zXq9", "summary": "The paper proposes STAR, a novel algorithm for unsupervised domain adaptation (UDA) of speech foundation models (e.g., one that has a decoder like Whisper). The UDA setting is the semi-supervised setting where unlabeled data from the target domain is available, and the speech foundation model is available. The unlabeled data is first pseudo-labeled and then used to train the ASR model. The paper specifically proposes the STAR Indicator, which combines the advantages of the auto-regressive confidence score and the attentive score (the attentive score is self-defined by the authors and derived from attention scores) to finally produce a score that can improve the fine-tuning process on pseudo-labels (named as \"informed fine-tuning\" by the authors and a method prevalent in literature where you guide the training process in a re-weighting manner).", "questions": "The algorithm is simple and sound. The evaluation setup is sufficient except the Whisper part. I might have further questions during the rebuttal period after I see other perspectives. But overall, the paper is well done (again, I do not find it too exciting but very sound) and I am inclined towards accepting. I hope the authors can respond to my points in Weaknesses.", "limitations": "- I don't see the Limitations of this paper discussed in the main paper. Can the authors please add it to the main paper? Limitations is a very important section of a NeurIPS paper. Additionally, can the authors please elaborate the limitations section with more details like: Latency and resource requirement over plain UDA fine-tuning, high noise scenarios where UDA may fail, etc.", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The strengths of the paper are as follows:\n- The paper is well-written and easy to follow. The illustrations are also well-made.\n- The algorithm proposed for uncertainty estimation is novel, to the best of my knowledge.\n- The evaluation setup chosen is sufficient, and no further additions can be made to the best of my knowledge.\n- I like the pre- and post-result analysis of the paper. Provides intuition into the proposed algorithm and results.\n- Section 5.2 and 5.3 is interesting. These kind of analysis is important for speech papers and should be promoted (which is generally not found in other papers).\n- The Appendix is well done and provides nice extra details", "weaknesses": "- As also claimed by the authors, the overall methodology of using uncertainty estimation for improving UDA is not novel. Only the estimation score proposed is novel. However, these works are properly cited. This makes the paper sound, but not very exciting to me. However, I also acknowledge that this is not a primary reason to reject.\n- I am not convinced why only Whisper (and some other models in Table 4) was employed for evaluation (a model that does not have a lot of training details revealed). I have generally found methods evaluated solely on Whisper to not work in a real-world setting. For example, what would have happened if we had adapted a purely LibriSpeech-trained model to the domains mentioned in the paper with the STAR algorithm? Though I understand that the paper title says \"Speech Foundation Models,\" for me, this makes the applicability of STAR a bit less to me. I would leave this decision to other authors\n- Some key baselines are missing: What happens to STAR without Utterance-level Filtering? Why was the LM-rescoring baseline not used (LM-rescoring is the most traditional method for UDA where the language model is further trained on target domain texts, may be you can fine-tune the LM on pseudo transcripts? (open to discussion if this is not valid)?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely appreciate Reviewer zXq9 for your valuable and constructive comments. Please find detailed responses below:\n\n- ***Q1: References to previous work***\n\n   We sincerely appreciate your feedback. While this is not a completely new topic, our proposed method only requires a downloadable foundation model and one hour of unlabeled speech to enhance the model in this domain without catastrophic forgetting. With the introduction of more speech foundation models, we believe this is a meaningful exploration.\n\n- ***Q2: What would have happened if we had adapted a purely LibriSpeech-trained model to the domains?***\n\n   Thanks for your question. Allow us to point out that the robustness of a model trained on LibriSpeech is insufficiently robust to serve as a pseudo-labeler. For instance, using the ATIS test set as an example, the LibriSpeech ASR model cannot recognise many city names. In comparison, Whisper, which is a widely used foundation model trained on much larger and more diverse datasets, is a lot more robust and can achieve one-to-many adaptation to accommodate different speech domains. Related discussions are included in Appendix A (Q1 & Q2).\n\n- ***Q3: What happens to STAR without Utterance-level Filtering?***\n\n   Thanks for your question, the ablation study without Utterance-level Filtering is in Table 8, and discussed in Appendix D. Since Table 1 can no longer accommodate additional columns, and Utterance-level Filtering is merely a process of removing particularly difficult-to-recognize samples. The effect is similar across different methods, so we have placed it in the appendix.\n\n- ***Q4: Why was the LM-rescoring baseline not used?***\n\n   Thanks for your suggestion. We need to point out that in the STAR experiments, each domain used only about one hour of unlabeled speech, which is equal to a few hundred utterances (Figure 3). This amount of text is insufficient to directly train or even adapt a useful LM for LM rescoring. Furthermore, all of our experiments were based on a single end-to-end model, while the additional LM-rescoring stage will increase the complexity making it no longer a purely end-to-end system.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "928b7hNBmS", "reviewer": "Reviewer_8Y8C", "summary": "This paper proposed a Self-TAught Recognizer (STAR) that leverages unlabeled data to enhance the robustness of automatic speech recognition (ASR) systems in diverse target domains. The proposed method includes a novel indicator that empirically integrates step wise information during decoding to assess the token-level quality of pseudo labels without ground truth and utterance level pseudo labels filtering . The STAR seems to have sufficient novelty for publication. Sufficient experiments are presented to show the effectiveness of the proposed method. The work should be reproducible since the code will be released.", "questions": "Some minor concerns or suggestions:\n\n1. Maybe I missed the sentence in the paragraph, but I don't find the description of which layer or head of the attention weights are used for the indicator A. It is worth a mention in the paragraph especially in Equation 5.\n2. Continuing the indicator A, did the authors compare the confusion matrix for other layers? In other words, does the attentive score confusion matrix behave similarly in Figure 2 for specific layers or for all layers (head)?", "limitations": "The limitation of the method can be restricted to certain use cases. However, the proposed method is still helpful to the community.", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The strengths of the papers are:\n1. The proposed method is novel.\n2. Sufficient experiments and good analysis.\n3. Good presentation and the paper are easy to follow.", "weaknesses": "Major concerns about the paper.\n\n1. It was shown in Figure 3 that the UDA converged with about 1 hour speech data for the target domain and the gain is very small with more data. Given that the performance of the STAR is still a lot worse (some are close) than the true-label model (The upper bound of STAR), I am concerning about the usage of STAR in the real scenario. One or several hours data transcription is sometimes affordable for some organizations, so that they might get better numbers than STAR.\n\n2. In other words, UDA has an upper bound while true-label model not. Did authors have the experiments to show how many hours of true-label data finetuning is equivalent to the performance of STAR? This might help the readers understand better on selecting UDA or human-transcription in different scenarios.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely appreciate Reviewer 8Y8C for the valuable and constructive comments. Please find detailed responses below:\n\n- ***Weakness 1 & 2***\n\n   Thanks for your feedback. We clarify that the true-label models use all labeled data from the training set of each domain. Therefore, how good this upper-bound is also depends on the size of their training set, with more training details provided in Appendix F.\n\n   Furthermore, fitting a 1B model to a few hours of labeled data can lead to overfitting issues. Without corresponding measures to mitigate this, the model's performance in other domains would significantly degrade. However, the training labels of STAR are self-generated, and reweighting is merely an attempt to selectively update without forcing the model to overfit to a new data distribution. This is also why STAR avoids catastrophic forgetting.\n\n- ***Question 1 & 2: Which layer or head of the attention weights are used for the indicator A.***\n\n   We appreciate your suggestion. In our experiment, the self-attention matrix is drawn from the last transformer layer of the foundation model, averaging on all heads. We also found that the performance is not sensitive to the selection of layers or heads. The last two layers, with a single head, can also achieve comparable performance. \n\n   For the confusion matrix, we confirm that the attentive score from the last five layers is very similar to Figure 2. The earlier layers can also show similar patterns but are somewhat less indicative of the pseudo-label quality, as the high layers learned more linguistic semantics.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper is one of those just above the acceptance threshold. The questions from the reviewers are fair. In particular, the novelty is somewhat lacking (questioned by vnSx and zXq9). Given that the approach is simple and the improvement is sizable, I am recommending an acceptance for this one.

I recommend the authors revise the parts of the paper that causes confusion, in particular, the parts that reviewers raise questions. Though the confusion was later resolved in the discussion, that means the paper requires revision.
Decision: Accept (poster)