Self-Taught Recognizer: Toward Unsupervised Adaptation for Speech Foundation Models

Authors: yuchen  hu
chen  chen
chao-han  huck
chengwei  qin
pin-yu  chen
eng  siong chng
chao  zhang
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-14 16:36:19

Abstract: We propose an unsupervised adaptation framework, Self-TAught Recognizer (STAR), which leverages unlabeled data to enhance the robustness of automatic speech recognition (ASR) systems in diverse target domains, such as noise and accents. STAR is developed for prevalent speech foundation models based on Transformer-related architecture with auto-regressive decoding (e.g., Whisper, Canary). Specifically, we propose a novel indicator that empirically integrates step-wise information during decoding to assess the token-level quality of pseudo labels without ground truth, thereby guiding model updates for effective unsupervised adaptation. Experimental results show that STAR achieves an average of 13.5% relative reduction in word error rate across 14 target domains, and it sometimes even approaches the upper-bound performance of supervised adaptation. Surprisingly, we also observe that STAR prevents the adapted model from the common catastrophic forgetting problem without recalling source-domain data. Furthermore, STAR exhibits high data efficiency that only requires less than one-hour unlabeled data, and seamless generality to alternative large speech models and speech translation tasks. Our code aims to open source to the research communities.

Main paper: 
 introduction: 
 Human speech, characterized by its inherent acoustic nuances [70] and variability across speakers [27] , is further complicated by the diverse and unpredictable environments. These factors contribute to significant domain distinctions in the speech signal, with differences in accent, speaking style, and background noise (visualized in Appendix B). Consequently, this diversity poses significant challenges in the field of automatic speech recognition (ASR), especially under diverse conditions [51] .In recent years, advancements in ASR technology [30, 84, 12, 73] have been boosted, primarily by the use of deep neural models and supervised learning with high-quality datasets. In particular, end-to-end ASR models pre-trained on industry-scale datasets have been made publicly available to the research community, such as OpenAI Whisper [73] , Meta SeamlessM4T [4] and NVIDIA Canary [71] . Considering the high diversity of speech domains, even a well-trained ASR foundation model usually performs less satisfactorily when encountering a domain shift problem [49, 83, 85] . This performance degradation stems from a critical dilemma: collecting and labelling sufficient training data in the target domain is immensely time-consuming and labour-intensive, thus hindering the domain adaptation process of ASR models. Some existing efforts [32, 44] focus on leveraging labelled source domain and unlabeled target domain data to enhance the ASR performance, as shown † Equal Contribution. in Fig. 1 (i). This solution is generally known as unsupervised domain adaptation (UDA) [24, 32, 44] and has been widely explored in both machine learning and speech processing communities.In the context of the UDA problem in ASR, the human "self-directed" ability [39, 17] when encountering an unfamiliar speech domain is first illustrated. Despite the unawareness of the ground truth labels of our heard speech, individuals can learn speech-to-text mapping from their self-directed transcriptions, particularly when they have high confidence (see Fig. 8 ). This learning mechanism has a parallel in machine learning, known as "self-training" [75, 90, 41] , which typically involves two stages. First, a pre-trained model generates the pseudo labels on target-domain data. Then, these data with pseudo labels, along with the associated confidence levels, are used to adapt the model.Meanwhile, unlike approaches in existing ASR literature, which often require the source data (data used to pre-train the ASR model in source domains) to achieve UDA [63, 15, 5] , humans, as the gold standard of speech communication, can address UDA issues in ASR without requiring any source data. Considering the exhibited generality of the speech foundation models with Transformer-related architectures based on the attention mechanism, it is the opportune moment to centre the attention mechanism on addressing the source-free UDA problem within the realm of ASR. Specifically, we study to adapt the pre-trained Whisper model using a small amount of unlabeled data from the target domain to become a domain-specific speech recognizer in different scenarios without using any source data, based on the process analogous to the human speech recognition as shown in Fig. 1 (ii).We hereby highlight the significant potential value that research on source-free UDA contributes to general ASR applications [3] : (i) It circumvents the extensive computational resources by adapting the ASR models without using any source data. (ii) It can considerably improve ASR performance in the target domain using only a small amount of speech samples without ground-truth labels.In this work, we propose a source-free UDA approach called Self-TAught Recognizer (STAR), which aims to enhance the performance of speech foundation models in specific target domains with unlabeled data. Based on the typical self-training scheme [92] , STAR delves deeply into a general issue: Given the absence of ground-truth labels, how do we assess the quality of pseudo labels for guiding self-training? Unlike humans who can intuitively gauge their confidence in listening, the decoding "confidence scores" from attention-based ASR models are typically approximated by the pseudo posterior probabilities from softmax function [53] , which may be unreliable due to the well-known over-confident issue of softmax [37] . Traditionally with HMM-based ASR, the confidence scores can be estimated based on lattice and confusion network data structures [18, 61, 91] , which, however, are difficult to obtain effectively in the end-to-end ASR framework.In pursuit of a better quality indicator, we explore the self-attention matrix obtained during autoregressive decoding, as it is not only grounded on speech input but also focuses on linguistic acceptability [36] . Specifically, we find the aggregated attention weights can be a more reliable indicator for measuring the quality of ASR-decoded tokens than the confidence scores. However, such an attentive score suffers from numerical instability, as recent findings [82, 62] from linguistic perspectives, it is normal for equally correct words (e.g., prepositions and nouns) to receive different semantic roles in a text sentence. This leads to the sub-optimality of using attentive scores alone to guide the fine-tuning process. We first substantiate these observations experimentally and then, in our STAR method, propose a novel integration approach based on their distinct characteristics, resulting in a both stable and reliable STAR indicator. Finally, it is employed to guide the subsequent finetuning process in a re-weighting manner, making a specific form of instructive adaptation.Our experiments evaluate the proposed STAR in various practical scenarios, including background noise, speaker accents, and specific scenarios (e.g., interviews and talks). Comprehensive results show the significant gains from STAR that enhances Whisper by an average of 13.5% relative word error rate (WER) reduction across 14 target domains. On some corpora, unsupervised STAR even approaches the upper bound of supervised adaptation using real labels. We also surprisingly observe that with informed finetuning, STAR prevents the adapted models from the common catastrophic forgetting problem without recalling source-domain data. Furthermore, we demonstrate that STAR enjoys: (i) remarkable data efficiency: it requires less than one hour of unlabeled data to adapt Whisper to its best performance on target domains; (ii) seamless generality: it is applicable to many prevalent speech foundation models and can be easily extended to the speech translation task.In general, our contributions are summarized as follows:• We direct our focus on source-free UDA in ASR with as one setting closed to real-world applications, where only a pre-trained speech foundation model and unlabeled speech samples are required to adapt to specific target domains. • We present a score-based self-training approach called STAR that includes a novel indicator to evaluate the pseudo-label quality and achieve informed finetuning, which significantly enhances the domain-specific capabilities of speech foundation models across a wide range of target domains, including noise, accent, and specific scenarios. • Intensive experiments demonstrate that STAR effectively avoids the common catastrophic forgetting problem in adaptation. Our further analysis of data efficiency and generality shows its potential for real-world applications, such as incremental updates for voice assistant. 
related work: 
 Unsupervised Domain Adaptation in ASR. Since acquiring the ground truth speech transcriptions is often prohibitively expensive in the target domain, many existing efforts bootstrap from available out-of-domain data to build an improved target domain model [79, 59, 93] . Besides directly simulating the target domain speech [31, 7] , adversarial learning is frequently utilized to learn invariant representations to mitigate domain shifts [33, 23] , which is also applied for front-end speech enhancement [64] . Meanwhile, teacher-student learning provides an alternative solution for efficient adaptation [50, 65] . These methods are also semi-supervised [86] , since labels from the source domain are available. More recently, self-supervised pre-trained models (e.g. wav2vec2 [2] ) have been used for pseudo-labelling to achieve unsupervised adaptation [44, 38] .Source-free Unsupervised Domain Adaptation. Given the potential presence of sensitive information in the source data [78] , there is a high demand for source-free UDA methods that transfer a pre-trained source model to the unlabeled target domain without any source data [47, 66, 16] . As a long-discussed machine learning issue, the mainstream solutions include self-supervised knowledge distillation [57] , contrastive learning [35] , hidden structure mining [89] , and uncertainty-guided adaptation [20] . Considering the inherent uncertainty in ASR decoding, we focus on the latter category and briefly review some representative indicators of uncertainty. Recently, there are some works [9] suggesting measuring uncertainty by the predicted variance from Monte Carlo Dropout [42] , utilizing aleatoric uncertainty by encouraging intra-domain consistency [48] , performing pseudo-labeling denoising using soft label correction [87] , and introducing self-entropy descent mechanism to find a threshold for pseudo-labeling [54] . It is worth noting that, confidence estimation for ASR systems can be dated back for decades, starting by using lattices and confusion networks [18, 61] for HMM-based systems. Improved confidence estimation can be achieved by model-based approaches, such as conditional random fields [76] , recurrent neural networks [40, 74] and graph neural networks [52] . More recent efforts [60, 77] focus on predicting uncertainty for auto-regressive decoding of attention-based models, however, they have not been applied in the source-free UDA.Summary. Given the large amount of data used to pre-train the speech foundation models, it is difficult to define the scope of its source domain and keep the source data for re-training. Therefore we believe it is necessary to directly adapt speech foundation models to target domains for UDA for speech tasks. The proposed STAR method aims to assess the quality of pseudo labels produced by the auto-regressive decoding process, which leads to an instructive and effective self-training process.Since STAR can remove the need for keeping and retraining with source data and considerably reduce the performance difference between using ground truth and pseudo labels for adaptation with target domain data samples, it has the potential to fulfil the goal of source-free UDA for the ASR task and achieve user-friendly deployment for real-world speech-based artificial intelligence products. 
methodology: 
 3.1 Problem Setup ASR Formulation. An end-to-end ASR system relies on a neural model f to recognize the input speech x ∈ R T into the corresponding text transcription y ∈ R L , where T and L denote the lengths of the input waveform and output text sequences respectively. During training, the model f is optimized by teacher-forcing [46] with cross-entropy loss:EQUATIONwhere y 1:L denotes the tokens in ground-truth labels y, and θ denotes the trainable parameters in f .UDA Setting. Given a source ASR model f (s) trained on labelled source domain data {X (s) , Y (s) } ∈ D (s) , domain adaption in ASR aims to transfer the learned knowledge and obtain a model f (t) that performs well on target domain D (t) , i.e., f (t) : X (t) → Y (t) . UDA is required if ground-truth labels Y (t) are not available. Source-free UDA [19, 55] posts a more challenging but practical scenario, where the source data {X (s) , Y (s) } used to pre-train the ASR is no longer available in adaptation. That is, only speech inputs X (t) is available when adapting the source model f (s) to the target domain D (t) . Self-training Strategy. In source-free UDA, since a source model itself typically generates pseudo-labels, some previous works [80] have referred to this learning approach as semi-supervised learning. To distinguish it from unsupervised domain adaptation, in this paper, we refer to the approach for addressing source-free UDA as self-training, consistent with the terminology used in studies [92] . Specifically, we adopt the pipeline of pseudo-labeling and informed finetuning. First, N (t) unlabeled speech segmentsX (t) = {x (t) i } N (t)i=1 are fed into source model f (s) to generate the pseudo labels corresponding to each of them, which are denoted as Ŷ(t) = {ŷ(t) i } N (t)i=1 . Then, the paired dataset with the speech inputs and their newly-generated pseudo labels {X (t) , Ŷ(t) } are used to finetune the source model to the target domain based on the self-training loss L ST :EQUATIONwhere the ASR loss L ASR follows the definition in Eq. (1).Summary. Since self-generated pseudo labels [63] do not introduce extra supervised information to the ASR source model, simply repeating this process is unlikely to yield performance improvements [75] . However, if high-quality pseudo labels are selected as domain-specific exemplars to inform the speech foundation model, it would then update in a direction beneficial to the target domain performance. Therefore, we propose a critical research question: How can we assess the quality of pseudo labels using an indicator that can also guide the model's update? The subsequent content of this section will delve into a detailed discussion from both token and utterance levels.The auto-regressive decoding in ASR can provide step-wise information on predicted tokens, which can be used for token-level uncertainty assessment [60, 77] . More importantly, this information can guide the subsequent training process: assigning different weights to each token when calculating the CE loss in Eq.( 2), namely informed finetuning.Why is confidence not a good indicator? The confidence score denotes the highest value among the posterior probability predicted by a neural model. In auto-regressive decoding, the l-th step of token confidence score C l can be denoted as: By preserving the C for each token during pseudo-labeling, we can perform informed finetuning with a re-weighting loss as follows:EQUATIONEQUATIONHowever, a substantial body of existing research [81] indicates that confidence does not accurately reflect predictive accuracy, especially in auto-regressive decoding [60] . In Eq. ( 4), the prediction of the current token is influenced by previously predicted tokens ŷl-1:1 , which can easily lead to error accumulation and propagation. We further inspect this claim in Whisper by empirical observation. As shown in Fig. 2 (Right-Up), we employ a confusion matrix to visualize the relationship between confidence score and pseudo-label quality, which shows that 52% of correct tokens are assigned low confidence and 60% of wrong tokens are assigned high confidence (more discussion is in Appendix C). Therefore, confidence cannot be a reliable pseudo-label quality indicator alone, like discussed in [21] .Is attentive score a better indicator? We explore if the self-attention matrix W obtained during auto-regressive decoding can reflect the pseudo-label quality. Unlike C l defined in Eq. ( 3), W has a direct association with X and linguistic acceptability [72] , which means that it might be less influenced by the variability of speech input (see example in Fig. 2 ).Empirical Observation. Starting from the fourth row and fourth column (first 3 tokens are fixed prompts: "⟨|en|⟩⟨|transcribe|⟩⟨|notimestamps|⟩"), for the correctly decoded tokens (black), the attention weights are concentrated on the diagonal and partially fall on other pseudo tokens. However, for wrongly decoded tokens (red), the attention weights almost all fall on the second column that corresponds to the task prompt token "⟨|transcribe|⟩" (highlighted in red boxes). To quantify this finding into a numerical metric, we defined an "aggregate pattern" indicator called attentive score, which is highlighted in the orange box in Fig. 2 and formulated as:EQUATIONwhere A l indicates the global semantic correlations between pseudo token ŷl with all tokens {ŷ l } L l=4 (first 3 tokens are task prompt). Specifically, we add the second term to also consider the attention weights with respect to future tokens, in order to capture the comprehensive global context to better assess the role of current token (see Table 7 for ablation study). We compare the values of attentive score A l and confidence score C l for this sentence in Fig. 2 (Left). As marked by black boxes, C l provides unreliable assessments for both 'board' (correct but low C l ) and 'year . ⟨|eos|⟩'(wrong but high C l ). In comparison, A l can accurately reflect the correctness of these tokens. To avoid randomness, we analyze CHiME-4 test-real and plot a confusion matrix in Fig. 2 (Right-Up). It is evident that, compared to C l , our A l more reliably assesses the quality of predicted tokens.Despite reliability, A l exhibits less numerical stability, e.g., "for" and "housing" are both correct tokens but their A l are distinct (1.8 vs. 0.8). The underlying reason is that their roles in the global context as prepositions and nouns are indeed different [82, 62] . However, when we try to use this A l to guide the ASR loss re-weighting like Eq.( 4), these labels are expected to be assigned comparable weights as they are equally correct. We verify this finding with the variance of A l and C l in Fig. 2 (Right-Down). For both correct and wrong tokens, A l exhibits higher variance, indicating it may not be suitable to guide the finetuning in a re-weighting manner directly.STAR Indicator: Reliable and Stable. To integrate the advantages of C f and A f , we introduce a new indicator that balances reliability and stability. Specifically, in cases where C f and A f exhibit conflicting values toward a pseudo token, we would select A f as an indicator that shows higher reliability. It can be mathematically formulated as:EQUATIONwhere σ denotes the sigmoid function σ(x) = 1/(1 + e -x ), and here it simulates the step function to capture the cases of conflicting scores. Our definition of conflict is A 2 l /C l larger than a hyperparameter threshold λ. This criterion can be decoupled into two terms, A l and A l /C l , which means a large attentive score as well as a large gap between attentive and confidence scores1 . Similarly, C 2 l /A l is another case of conflicting scores, and we add them up to simulate the logical "OR" operation.On the other hand, if A f and C f present consistent assessment towards a pseudo token, C f would be used to scale A f using its stability. Specifically, we design a soft interpolation strategy inspired by focal loss [56] to integrate them:EQUATIONSimilarly, we also use Sigmoid function to simulate the non-conflicting cases, where we multiply the two terms to denote logical "AND". Inspired by the smoothing technique in focal loss [56] , we propose to leverage the gap between two scores for scaling A l * e (C l -A l )/τ , where τ is temperature.During the subsequent informed finetuning stage, we combine the two indicators above to guide the training process in a re-weighting manner, and Eq.( 4) should be re-written as:EQUATION)As a result, the STAR scores are both reliable and stable as shown in Fig. 5 , which serves as a better quality indicator to guide the informed finetuning (see Algorithm 1 in Appendix for details).The utterance-level filtering aims to remove those predicted utterances with low overall quality since they are probably harmful for subsequent adaptation. We now introduce several existing approaches to assess the utterance-level quality of pseudo labels, which are often used for uncertainty estimation. Notably, high uncertainty usually implicates low quality for the generated sequence.Monte Carlo Sampling [42] conduct multiple times of stochastic forward decoding with activated dropout to get a list of predictions [9] . Then the list with a large variance is considered to have high uncertainty and should be removed from subsequent training. However, this method does not apply to Whisper as it does not use dropout in training. As an alternative, we introduce a similar method for assessing utterance-level uncertainty. Specifically, given an input speech x, we first implement one forward decoding and set the result ŷ as the base transcription. Then, we randomly disturb the model weights of Whisper with Gaussian noise, and repeat the forward decoding for K times, resulting in a list of pseudo transcriptions {ŷ k } K k=1 . Thereafter, we calculate the edit distance (ED) between pseudo transcription ŷk and the base transcription ŷ, which indicates the impact of disturbance on Whisper decoding. Then, the model's robustness in transcribing speech x can be calculated as:EQUATIONTable 1 : Main WER (%) results of the proposed STAR adaptation and baselines in various ASR domains. "Whisper (frozen)" denotes the zero-shot performance without adaptation. "Whisper (self-train.)" is the vanilla self-training scheme consisting of pseudo-labeling and finetuning. Based on that, "UTT filter " adds utterance-level filtering explained in §3.3, and "TOK reweight " performs two token-level re-weighting explained in §3.2. "Whisper (real label)" is supervised learning with real (ground truth) labels and can be viewed as the upper-bound performance of source-free UDA. After obtaining k pseudo labels, we can examine their diversity to further assess the model's uncertainty. If there are many repetitions in the list, it indicates that the model is more confident in transcribing speech x. Therefore, we utilize a scaling factor l that is equal to the utterance amount after de-duplication. The final utterance-level quality is combined by the numeric multiplication of l and U (x, ŷ), which is then used to rank the N t pseudo data samples, and top α% samples are removed due to large data uncertainty. Additionally, we also implement a beam search decoding and a consensus decoding [61] baselines as alternative utterance-level filtering approaches for comparison, where more experimental results and discussions are presented in Appendix D.4 Experimental SetupQuestion: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof? Answer: [NA] Justification: Our work does not include theoretical results. Guidelines:• The answer NA means that the paper does not include theoretical results.• All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced. • All assumptions should be clearly stated or referenced in the statement of any theorems.• The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition. • Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material. • Theorems and Lemmas that the proof relies upon should be properly referenced. 
asr domains: 
 We introduce STAR in various ASR domains to verify its general effectiveness, including noisy speech, accented speech, and specific scenarios. First, for noisy speech we use the CHiME-4 [83] , LibriSpeech-FreeSound [69] , and RATS [26] datasets, which covers a wide range of noise types including bus, cafe, pedestrian area, street junctions, babble, car, airport, and the challenging radio communication noises. Second, we select four typical accents from the CommonVoice [1] dataset, including African, Australian, Indian, and Singaporean accents. Finally, we also evaluate our approach under some specific scenarios, including BBC talks (LRS2 [13] ), TED talks (TED-LIUM 3 [29] ), telephone conversation (SwitchBoard [25] ), interview conversation (CORAAL [43] ), and airline information consultation (ATIS [28] ). More details about the datasets are presented in Appendix F.We use the Whisper-Large-V3 model for main experiments, which contains 1.5 billion parameters trained on 680k-hour web-scale data. It is fine-tuned using Adam optimizer [45] with an initial learning rate of 1e -5 for 2 epochs. The batch size is set to 1 with 16 gradient accumulation steps. For hyper-parameters, the threshold λ is set to 2 and the temperature τ is 10. In addition, the percentile α of utterance-level filtering is 20, which shows consistent effectiveness across different datasets.5 Results and AnalysisQuestion: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)? Answer: [Yes] Justification: Section 3 and 4, Appendix F and G. Guidelines:• The answer NA means that the paper does not include experiments.• If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not. • If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable. • Depending on the contribution, reproducibility can be accomplished in various ways.For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed. • While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example (a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm. 
effectiveness of star: 
 To examine the effectiveness of STAR, we conduct comparative experiments across various domains and report the WER results in Table 1 .Main Results. From noise adaptation results on CHiME-4, LS-FreeSound, and RATS, we observe that: (i) STAR enhances Whisper in all noise scenarios, reducing the WER up to 24.9% relatively. Specifically, on the challenging RATS dataset with pseudo labels of a 46.9% WER, our STAR can still produce a 4.9% relative improvement. (ii) For some domains, e.g., "airport" and "car", STAR can even approach the upper-bound performance by supervised learning. This demonstrates that even with unlabeled data only, our method can effectively adapt Whisper to specific target domains.From results on other domains, we observe that: (i) STAR consistently improves the accented ASR to approach the supervised upper bound. (ii) Whisper does not perform well in some colloquial scenarios (SwitchBoard and CORAAL) as the spoken language tends to be informal and less grammatically correct, which leads to poor-quality pseudo labels and then influences our adaptation performance.Analysis of Catastrophic Forgetting. Table 2 analyzes the potential forgetting issue of our method by evaluating the CHiME-4-finetuned model on other datasets. Surprisingly, contrary to the common catastrophic forgetting issue that commonly happens in traditional source-free ASR adaptation, our STAR approach can even improve the performance in other domains. We speculate that under the self-training scheme, the pseudo label is generated by the model itself, so that it may avoid the model from over-fitting to samples with vastly different data distributions [10] . Furthermore, compared to vanilla self-training, STAR can better highlight the high-quality pseudo tokens for informed finetuning, which may help improve the model's general ASR ability. More detailed analysis are in §E. Analysis of Indicators. Table 1 also presents the performance of different indicators in the informed finetuning. First, we observe that the utterance-level filtering yields some effects by removing bad training samples. Then, for token-level re-weighting, the two pseudo-label quality indicators both improve the performance, where the attentive score performs better due to higher reliability. Our proposed STAR indicator achieves the best result by integrating the strengths of both scores. We also use a case in Table 3 to illustrate, where exists a wrong pseudo token "teams". The confidence score fails to reflect this error while the attentive score succeeds, but the latter suffers from less numerical stability (i.e., large variance). By integrating their strengths, our STAR score achieves both reliability and stability in assessing the pseudo-label's quality. In addition, we also calculate the NCE metric to show the better quality of our proposed STAR and attentive scores than traditional confidence scores.Generalization to Different Speech Foundation Models. To further evaluate the generalization ability of our approach, we extend STAR to different foundation models, including OWSM, Canary and Parakeet-TDT that take top places in the HuggingFace ASR leader-board2 . Consistent performance gains (i.e., over 10% relative WER reduction) on these models has verified the excellent generality of STAR. In addition, it also works well on relatively small models like Whisper-Medium.en-0.8B.Generalization to Speech Translation (ST) Task. Apart from ASR, we also investigate another widely studied speech task, the ST task, to further verify the generality of STAR adaptation. As shown in Table 5 , results on various FLEURS X→En tracks illustrate an average of over 1.2 BLEU improvements (2.2 BLEU for De→En). It shows the good potential of our STAR adaptation on other sequence-to-sequence tasks besides ASR, which could lead to more extensions for future work.In this section, we conduct ablation studies to analyze STAR from perspectives of data (Fig. 3 ), model (Table 9 ), and finetuning approaches (Table 10 ), which provide a constructive reference for deploying ASR foundation models in practical scenarios using STAR. More analysis are in Appendix E.Data Efficiency. We explore the requirement of unlabeled data amount (N t ′ ) for STAR adaptation. Fig. 3 shows the WER results on four datasets with different numbers of training utterances. Surprisingly, only 200 to 500 sentences (less than 1-hour unlabeled speech data) are required to achieve the optimal effects, which cost around 0.8-hour training time on single NVIDIA-A100-40GB GPU. This remarkable data efficiency significantly saves the labours in real-world applications: not only is there no need for manual labelling, but the collection of unlabeled data also requires less than one hour.Model Size. Table 9 reports the performance on CHiME-4 test-real that applies STAR to the Whisper family with different model sizes. Results show that our STAR adaptation works well on difference scales of foundation models. Specifically, the promising performance gains on light model (base.en) implicates the potential of STAR in practical resource-constrained conditions, such as mobile devices.Finetuning Approach. Considering that adapting speech foundation models with a small amount of data might risk over-fitting, we explore the impact of different finetuning approaches in Table 10 . We observe that both regular finetuning (full, encoder-only, decoder-only) and efficient finetuning methods (LoRA) yield similar effectiveness, which provides flexible choices under different settings. 
conclusion: 
 We propose STAR, a source-free UDA method that effectively adapts the speech foundation models to various target domains with unlabeled data. Specifically, STAR introduces a novel indicator to assess the pseudo-label quality and then instructively guide the finetuning of the model. Our experiments verify STAR's efficacy on ASR tasks across a wide range of target domains including noise, accent, and specific scenarios, and it even approaches the upper-bound performance of supervised adaptation on some corpora. Furthermore, we observe that STAR can avoid the catastrophic forgetting problem that is often suffered by models adapted without recalling source-domain data. Furthermore, STAR only requires less than one hour of unlabeled data to achieve an average of 13.5% relative WER reduction across 14 domains, and it also shows seamless generality to speech translation tasks. This enables us to deploy speech systems in real-world scenarios rapidly and conveniently.OWSM-V3.1-1.0B [68] 3 , Canary-1.0B4 , Parakeet-TDT-1.1B 5 , and SeamlessM4T-V2-2.3B [4] 6 . Table 4 verify the effective generalization ability of our STAR adaptation method on various state-ofthe-art ASR foundation models 7 . Furthermore, we also employ the speech translation foundation model SeamlessM4T to verify our generality. Table 6 presents the WER results on CHiME-4 test sets with SeamlessM4T, where the STAR adaptation achieves significant improvements over zero-shot and self-training baselines, which even approaches the supervised upper bound. We observed that although the performance of SeamlessM4T-Large-V2 is slightly worse than Whisper Large-V3 on ASR task, STAR can still achieve up to a 30.7% WER reduction that improves its noise robustness. STAR requires large speech models to possess universal robustness across various domains to fulfill their role as a reliable pseudo-labeler, where domain-specific ASR models like WavLM and RNN-T may not work well. To illustrate, the WavLM-Conformer ASR baseline [11] achieves the state-ofthe-art result on the LibriSpeech test-clean dataset (with a WER of only 1.8%). However, when facing domain shifts, such as the CHiME-4 noisy dataset, its zero-shot performance drops to 14.4%, and it exceeds 20% on the CommonVoice accented dataset. Under these circumstances, it is nearly impossible to use only unlabeled data to adapt this model to the Whisper's zero-shot performance (5 7% WER). Therefore, we argue that adding such baselines is not of much reference value. As more general-purpose speech foundation models are released, we prefer to focus our research on studying their decoding behaviors to adapt them more efficiently and conveniently to specific task domains. Table 4 and 6 provide more results on CHiME-4 dataset with more speech foundation models (i.e., OWSM, Canary, Parakeet, SeamlessM4T) to show the good generality of STAR.Question 3: Is the proposed STAR method limited to the ASR task?Our proposed STAR approach is compatible with any tasks using attention-based encoder-decoder architecture with auto-regressive decoding, the most prevalent framework in many areas not limited to speech and language. Therefore, we believe this work provides useful insights to researchers from other communities who use similar model architectures and need to assess the quality of autoregressive decoders, such as speech translation, audio/image captioning. Table 5 presents the strong results of STAR on speech translation task, which verifies its task generality. However, since this work focuses on ASR task, we would like to leave the evaluation on more tasks to future work. In addition, recent research on LLMs [36] also focuses on the unsmooth self-attention matrix like Fig. 2 , where they successfully alleviate the LLMs' hallucination problem in auto-regressive decoding through this observation. This evidence indicates the potential impact of our method on other communities. 
broader impacts: 
 Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?Answer: [Yes] Justification: Appendix A.Guidelines:• The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact. • Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations. • The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster. • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology. • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML). 
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?Answer: [NA] Justification: Our work poses no such risks.Guidelines:• The answer NA means that the paper poses no such risks.• Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters. • Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images. • We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.12. Licenses for existing assets Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?Answer: [Yes] Justification: Section F.Guidelines:• The answer NA means that the paper does not use existing assets.• The authors should cite the original paper that produced the code package or dataset.• The authors should state which version of the asset is used and, if possible, include a URL. • The name of the license (e.g., CC-BY 4.0) should be included for each asset.• For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided. • If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset. • For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. • If this information is not available online, the authors are encouraged to reach out to the asset's creators. 
new assets: 
 Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?Answer: [Yes]Justification: We have well documented the introduced assets in supplemental material. Guidelines:• The answer NA means that the paper does not release new assets.• Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc. • The paper should discuss whether and how consent was obtained from people whose asset is used. • At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file. 
crowdsourcing and research with human subjects: 
 Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?Answer: [NA] Justification: This paper does not involve crowdsourcing nor research with human subjects.Guidelines:• The answer NA means that the paper does not involve crowdsourcing nor research with human subjects. • Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper. • According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector. 
institutional review board (irb) approvals or equivalent for research with human subjects: 
 Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?Answer: [NA] Justification: This paper does not involve crowdsourcing nor research with human subjects.Guidelines:• The answer NA means that the paper does not involve crowdsourcing nor research with human subjects. • Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper. • We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution. • For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review. 
 
 Appendix: We summarize the vital difference in the following two points:• Prior works focus on adapting from one source domain to a single target domain (e.g., clean to noisy [58] ), whereas STAR leverages the universality of Whisper to explore one-to-many domain adaptation. Although the domain mismatch issue in the latter approach is less severe than in the former, we argue that the baseline performance of the latter is significantly better than that of the former, making improvements more challenging to achieve.• Although both of them are auto-regressive processes, the decoding of speech foundation models exhibits partially distinct characteristics compared with vanilla ASR decoders in previous works, such as over-confidence phenomena [37] . Therefore, STAR adopts an empirical indicator of quality derived from the decoding features of speech foundation models, which can more effectively guide the subsequent finetuning process.Question 5: What is the scope of efficacy when applying STAR adaptation?As shown in Table 1 , STAR can improve the performance with zero-shot WER ranging from 2.9% (car) to 46.9% (radio). However, we observe that the WER improvement in RATS mainly stems from the adjustment of some prepositions and articles, which may not improve the comprehensibility of recognition results fundamentally. Therefore, collecting labeled data for supervised learning is inevitable when the scenarios are quite challenging.Question 6: Can STAR be involved in adapting test data?We observe that some works perform unsupervised domain adaptation using unlabeled test data. However, STAR only accesses test data once, and the unlabeled target domain data is drawn from the training set of corresponding corpus. We argue this setting more closely aligns with practical scenarios: developers cannot access the test set during the adaptation process. However, they can conveniently collect small amount (see data efficiency in Fig. 3 ) of unlabeled target-domain speech for adaptation, and then deploy the adapted model in testing environments.This work supports rapid and convenient deployment of ASR applications in real-world scenarios, which poses positive societal impact. During training process, we only use publicly available data and pre-trained models, so that our work will not pose explicit negative impact. One thing worth noting is that, our algorithm shows good generality and thus might cause abuse in some special occasions (e.g., confidential), we will release code carefully under strict licenses and rules to avoid negative impact.Our approach is designed specifically for Transformer-based speech foundation models, so that it may not handle the cases beyond the foundation models, e.g., the unseen languages, unseen tasks, extremely adverse conditions, etc.Fig. 4 visualizes the spectrograms of parallel clean and noisy speech samples. We can observe clear speech patterns in the clean spectrogram (i), while they are significantly contaminated by noise in the noisy spectrograms (ii) and (iii). Specifically, the babble noise corrupts speech signals more than airport noise, where speech patterns are almost completely removed. The reason is babble noise contains human speech and thus of sample type as an original speech signal, resulting in more significant corruption. These two kinds of noisy speech are reported in Table 1 . Overall, the domains of clean and noisy speech are quite distinct from the perspective of pattern recognition. C More Discussions of Pseudo-label Quality Indicators Fig. 5 presents more investigations of the quality indicators. First, from the confusion matrix we can observe the higher reliability of the attentive score over the confidence score, which has been discussed in Section 5.1, and our proposed STAR score maintains the high reliability of the attentive score by sophisticated integration. Then, from the variance statistics we can observe that the attentive score suffers from less numerical stability, while our proposed STAR score benefits from the high stability of the confidence score. As a result, the STAR score provides a both reliable and stable indicator of the quality of the pseudo label. Furthermore, we note that after STAR adaptation, the Whisper model can produce higher-quality pseudo labels, which not only verifies its effectiveness but also explains its potential for iterative adaptation (see Section E).Confidence Score Attentive Score Figure 5 : Confusion matrix, variance and normalized cross-entropy (NCE) of confidence score (orange), attentive score (blue), and our STAR score (green), in terms of the Whisper baseline and our STAR-adapted model. For the confusion matrix, the y-axis denotes the correctness of pseudo tokens, i.e., correct and wrong, and the x-axis denotes whether the corresponding score is high or low. Since all scores are normalized via divided by mean value, we set 1 as the threshold to separate them into large and small groups, where more thresholds are analyzed in Fig. 6 . NCE is a statistical metric to measure the quality of confidence measure, where higher value indicates better measurement.Beam search decoding is a widely used strategy in sequence-to-sequence decoding, which expands the search space to obtain an N-best hypotheses list. Recent research on speech foundation models shows that N-best results contain rich information [8, 88] , where the diversity can reflect the final WER performance [34, 6] . Specifically, we utilize the beam search decoding as a replacement for Gaussian disturbance and obtain an N-best hypotheses list for diversity calculation. The best hypotheses and beam size N are respectively viewed as base transcription ŷ and K, and the quality of utterance U is calculated in the same manner as Eq.( 9). Furthermore, we also introduce an earlier method from conventional ASR, consensus decoding [61] , as an alternative to investigate the role of utterance-level filtering approach, where the utterance quality is estimated similar to the beam search decoding.Table 8 presents the ablation study of utterance-level filtering strategy. First, we compare the Khypotheses by Gaussian disturbance with the N -best hypotheses by beam search decoding and consensus decoding, where we observe that the former performs a little better than the latter two. Therefore, we employ Gaussian disturbance for utterance-level filtering in the main experiments. Finally, we investigate the role of utterance-level filtering in the entire STAR adaptation, where the results demonstrate its contribution in the final performance gains (second last column). Following Section 5.3, here we would like to present more details about the ablation study on model size and finetuning approaches. In addition, we also observe that our STAR method can further improve the performance with multiple-round iterative adaptation.Model Size. Finetuning Approach. Considering that training large speech models with a small amount of data might risk over-fitting, we explore the impact of different tuning approaches for the informed finetuning process in this experiment. From Table 10 , we observe that (i) freezing part of parameters can not improve the performance of STAR adaptation. However, finetuning the decoder only is the optimal strategy for supervised adaptation. (ii) Using less trainable parameters, LoRA tuning shows commendable results with full finetuning. Nevertheless, LoRA introduces further hyper-parameters (e.g., rank) and we found it is sensitive to the learning rate. Considering it slightly decreases the inference efficiency, LoRA tuning is recommended only in situations with limited training resources.Analysis of Catastrophic Forgetting. Following Table 2 , we present more results regarding the forgetting issue on Whisper-Medium.en-0.8B and SeamlessM4T-V2-2.3B in Table 11 . We observe that our STAR can prevent the catastrophic forgetting problem on different-scale speech foundation models. As for the potential reasons, we analyze from three points. First, we observe that the vanilla self-training scheme can also well mitigate the forgetting problem. We can gain some inspiration from previous work [10] for analysis. Since the pseudo label is generated by the model itself, it may not force the model heavily to over-fit any external data distributions, so that self-training does not degrade the out-of-domain performance. On the other hand, current prevalent foundation models are usually trained by multi-task learning, where not all model capacities are used for ASR. Therefore, specific self-training for ASR task may help mitigate the forgetting problem within ASR though with different domains. Furthermore, compared to vanilla self-training, our STAR shows even better performance. The core contribution of STAR is the novel quality indicator that can better highlight the high-quality pseudo tokens for informed finetuning. Considering ASR tasks in different domains, the high-quality pseudo tokens usually correspond to speech frames that are relatively high-quality and easy to recognize. Therefore, highlighting the weights of such pseudo tokens may avoid bringing in low-quality knowledge that may conflict with the existing knowledge embedded in the parameters of the pre-trained speech foundation models, and thus prevent the catastrophic forgetting problem. More study is expected for a deeper understanding of the mechanism behind it. Considering the focus of this work as well as the space limit, we would like to leave this study to future work.As a self-training approach, STAR is iterable by repeating the process of pseudo-labeling and informed finetuning. Table 12 reports the WER results with different numbers of iterations using different model sizes and test sets. In most test sets, multiple iterations of STAR result in further performance improvements. This indicates that while learning from pseudo labels, errors also accumulate, thereby limiting the upper-bound of self-training. Additionally, the enhancement of iteration is relatively larger in smaller models, e.g., 0.7% further WER reduction on Whisper-base.For dataset selection, our goal is to cover common scenarios of ASR tasks, which can be grouped into three categories, i.e., background noise, speaker accents, and specific scenarios. Consequently, we collect and employ the following datasets with evident domain characteristics to evaluate our proposed approach. In addition, considering our proposed STAR adaptation is built on self-attention matrix that focuses on global contextual correspondence, we filter out short utterances (i.e., with less than 5 tokens) in some datasets for better and more efficient evaluation.All the data used in this paper are publicly available and under the following licenses: the Creative SwitchBoard [25] : The SwitchBoard corpus is a telephone speech dataset collected from conversations between pairs of speakers. It focuses on North American English and involves over 2.4k conversations from approximately 200 speakers. We randomly select 5,000 long samples from its train split as the training data, and use its eval2000 split as the test data. • The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size. • If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness. • While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.Justification: Our experimental results are stable and do not need such statistical significance.Guidelines:• The answer NA means that the paper does not include experiments. Guidelines:• The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.• If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics. • The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).To avoid special cases like two tiny scores where one is many times of another (e.g., 0.01, 0.001).https://huggingface.co/spaces/hf-audio/open_asr_leaderboardhttps://huggingface.co/espnet/owsm_v3.1_ebfhttps://huggingface.co/nvidia/canary-1bhttps://huggingface.co/nvidia/parakeet-tdt-1.1bhttps://huggingface.co/facebook/seamless-m4t-v2-largehttps://huggingface.co/spaces/hf-audio/open_asr_leaderboard 


Decision: This paper is one of those just above the acceptance threshold. The questions from the reviewers are fair. In particular, the novelty is somewhat lacking (questioned by vnSx and zXq9). Given that the approach is simple and the improvement is sizable, I am recommending an acceptance for this one.

I recommend the authors revise the parts of the paper that causes confusion, in particular, the parts that reviewers raise questions. Though the confusion was later resolved in the discussion, that means the paper requires revision.

Reviews: review: paper_summary: paper_summary: The paper proposes STAR, a novel ASR domain adaptation technique that requires no labeled data and only a few unlabeled samples. STAR utilizes the confidence score and self-attention score obtained during decoder inference to calculate the reliability score (STAR indicator) for each token. The score of each token is then used as a multiplier to adjust the fine-tuning loss, which is based on generated pseudo-labels. In addition, STAR employs utterance-level filtering to remove noisy predictions. Extensive experiments across various ASR models, datasets, and fine-tuning techniques demonstrate that STAR achieves significant accuracy improvements compared to the baseline self-training approach.
strength_weakness: strength_weakness: * The proposed method is designed for a Transformer-based model with auto-regressive generative decoder architectures. As such, it may not be easy to adopt STAR for CTC or RNN-T-based ASR models (as the authors also discussed in Appendix A).
* It would be important to discuss the differences and similarities between STAR and noisy student-based training [1][2]. NST also employs pseudo-labeling, heavy filtering, and iterative refinement.
  * [1] Pushing the Limits of Semi-Supervised Learning for Automatic Speech Recognition
  * [2] Improved Noisy Student Training for Automatic Speech Recognition
questions: questions: * The attentive score A (Equation 5) seems to be affected by the total length of the generated transcript. The longer the transcript, the (potentially) higher the attentive score. In contrast, confidence scores are bounded to [0, 1]. Maybe missing a normalization term in Equation 5?
* Any intuitive reasons not to incorporate cross-attention scores?
* Comparing Table 1 and 2, the numbers of “Frozen” are the same but “Self-train” and “STAR” are different. For example, TED3 WER is (5.2, 4.9, 4.1) in Table 1 but (5.2, 5.3, 5.0) in Table 2. What’s the difference?
* How many beams are used in the beam search? Are the beam search-based pseudo-labels also used for self-training baselines?
* It would strengthen the paper’s claim if the pseudo-label and STAR score could be cross-transferred between different models (for example, Whisper-Large generates training resources for Whisper-Tiny).
limitations: limitations: * The paper adequately addresses the limitations.
paper_summary: paper_summary: This paper investigates the use of audio-only data to enhance ASR performance for domain adaption in Speech Foundation Models. The approach is straightforward: recognition results are used to compute a confidence score for each token (e.g., BPE in this paper), which then weights the loss function, as shown in Equation (4). The key innovation is the computation of token-level confidence scores by adopting attention scores instead of the softmax output for confidence estimation. Unlike traditional methods that operate at the utterance level, this approach focuses on token-level confidence. The concept is simple: use the attention weight as the confidence score rather than the predicted next-word probability. The proposed STAR method combines information from both token-level probabilities and the attention matrix. ASR performance appears promising, especially when compared to the strong baseline model Whisper large v3.
strength_weakness: strength_weakness: 1. Several details are missing. How is the attention matrix calculated given multiple layers and heads? Are the attention weights from cross-attention or self-attention used?
2. Do you need to tune the threshold for each corpus, or simply use a fixed threshold for all datasets? What is the specific value for the threshold lambda?
3. The definition of conflict is A_l^2/C_l, which seems intuitive. Is there any particular reason for using this formulation?
4. Many studies have attempted to use entropy from the output layer instead of probability for confidence estimation in end-to-end ASR systems.
5. I am interested in whether the informed fine-tuning shown in Equation (4) is proposed in this paper. The confidence score is used to filter utterances and weigh the importance of each word. Do you have any ablation studies demonstrating the effectiveness of this approach?
6. It would be better to combine Equations (6) and (7) into an equation array to clearly state the formulation.
7. In Equation (5), the attention score is computed using the attention weight for the current word to the previous words and from future words to the current word. Is there any particular reason for this choice? Is it acceptable for the A_l to be larger than 1?

Overall, this paper seems to rely heavily on empirical or intuitive settings, making it difficult to extend to a theoretically sound approach. Additionally, many key details are missing. It would be appreciated if the authors could provide more details and present the information more clearly.

I am open to reconsidering my rating based on the authors' rebuttal.
questions: questions: I have listed my questions in the above section.
limitations: limitations: N/A
paper_summary: paper_summary: The paper proposes STAR, a novel algorithm for unsupervised domain adaptation (UDA) of speech foundation models (e.g., one that has a decoder like Whisper). The UDA setting is the semi-supervised setting where unlabeled data from the target domain is available, and the speech foundation model is available. The unlabeled data is first pseudo-labeled and then used to train the ASR model. The paper specifically proposes the STAR Indicator, which combines the advantages of the auto-regressive confidence score and the attentive score (the attentive score is self-defined by the authors and derived from attention scores) to finally produce a score that can improve the fine-tuning process on pseudo-labels (named as "informed fine-tuning" by the authors and a method prevalent in literature where you guide the training process in a re-weighting manner).
strength_weakness: strength_weakness: - As also claimed by the authors, the overall methodology of using uncertainty estimation for improving UDA is not novel. Only the estimation score proposed is novel. However, these works are properly cited. This makes the paper sound, but not very exciting to me. However, I also acknowledge that this is not a primary reason to reject.
- I am not convinced why only Whisper (and some other models in Table 4) was employed for evaluation (a model that does not have a lot of training details revealed). I have generally found methods evaluated solely on Whisper to not work in a real-world setting. For example, what would have happened if we had adapted a purely LibriSpeech-trained model to the domains mentioned in the paper with the STAR algorithm? Though I understand that the paper title says "Speech Foundation Models," for me, this makes the applicability of STAR a bit less to me. I would leave this decision to other authors
- Some key baselines are missing: What happens to STAR without Utterance-level Filtering? Why was the LM-rescoring baseline not used (LM-rescoring is the most traditional method for UDA where the language model is further trained on target domain texts, may be you can fine-tune the LM on pseudo transcripts? (open to discussion if this is not valid)?
questions: questions: The algorithm is simple and sound. The evaluation setup is sufficient except the Whisper part. I might have further questions during the rebuttal period after I see other perspectives. But overall, the paper is well done (again, I do not find it too exciting but very sound) and I am inclined towards accepting. I hope the authors can respond to my points in Weaknesses.
limitations: limitations: - I don't see the Limitations of this paper discussed in the main paper. Can the authors please add it to the main paper? Limitations is a very important section of a NeurIPS paper. Additionally, can the authors please elaborate the limitations section with more details like: Latency and resource requirement over plain UDA fine-tuning, high noise scenarios where UDA may fail, etc.
paper_summary: paper_summary: This paper proposed a Self-TAught Recognizer (STAR) that leverages unlabeled data to enhance the robustness of automatic speech recognition (ASR) systems in diverse target domains. The proposed method includes a novel indicator that empirically integrates step wise information during decoding to assess the token-level quality of pseudo labels without ground truth and utterance level pseudo labels filtering . The STAR seems to have sufficient novelty for publication. Sufficient experiments are presented to show the effectiveness of the proposed method. The work should be reproducible since the code will be released.
strength_weakness: strength_weakness: Major concerns about the paper.

1. It was shown in Figure 3 that the UDA converged with about 1 hour speech data for the target domain and the gain is very small with more data. Given that the performance of the STAR is still a lot worse (some are close) than the true-label model (The upper bound of STAR), I am concerning about the usage of STAR in the real scenario. One or several hours data transcription is sometimes affordable for some organizations, so that they might get better numbers than STAR.

2. In other words, UDA has an upper bound while true-label model not. Did authors have the experiments to show how many hours of true-label data finetuning is equivalent to the performance of STAR? This might help the readers understand better on selecting UDA or human-transcription in different scenarios.
questions: questions: Some minor concerns or suggestions:

1. Maybe I missed the sentence in the paragraph, but I don't find the description of which layer or head of the attention weights are used for the indicator A. It is worth a mention in the paragraph especially in Equation 5.
2. Continuing the indicator A, did the authors compare the confusion matrix for other layers? In other words, does the attentive score confusion matrix behave similarly in Figure 2 for specific layers or for all layers (head)?
limitations: limitations: The limitation of the method can be restricted to certain use cases. However, the proposed method is still helpful to the community.

Comments: title: Thank You for the rebuttal
Regarding the deadline of author-reviewer discussion period
Thank you for feedback
Thank you for feedback
Thank you for feedback
Reply the rebuttal
Thank you for the response
Thank You for the rebuttal
comment: I appreciate the authors' detailed rebuttal addressing the concerns I highlighted in my review. I think the rebuttal addresses some of my questions effectively. I have adjusted my score accordingly.
Dear Reviewer LswZ,

Thank you for your kind efforts in providing initial reviews for our work! We have taken them into careful consideration and provided the responses accordingly.

Since the deadline of author-reviewer discussion period is approaching, could you please take some time to confirm whether our responses have satisfactorily addressed your concerns? If you believe so, may you please consider adjusting your initial score accordingly? Please do not hesitate to reach us if you have any further questions or comments.

Best,

Paper 10759 Authors
Dear Reviewer 8Y8C,

Thank you very much for positive feedback! Please do not hesitate to reach us if you have any further questions or comments.

Best,

Paper 10759 Authors
Dear Reviewer zXq9,

Thank you very much for positive feedback! Please do not hesitate to reach us if you have any further questions or comments.

Best,

Paper 10759 Authors
Dear Reviewer vnSx,

Thank you very much for positive feedback! Please do not hesitate to reach us if you have any further questions or comments.

Best,

Paper 10759 Authors
Thank you for the rebuttal. I am keeping my score which is already an accept. Please make sure the explanations can be reflected in the revised paper.
Thank you for addressing my questions. I am keeping my score toward accept, and I believe this paper can inspire many following studies.
Thank You for the rebuttal. I am keeping my score which is already an accept. The paper takes a decent step towards advancing the domain.