It's Never Too Late: Fusing Acoustic Information into Large Language Models for Automatic Speech Recognition
Authors: CHEN CHEN
Ruizhe Li
Yuchen Hu
Sabato Marco Siniscalchi
Pin-Yu Chen
EngSiong Chng
Chao-Han Huck Yang
Venue: ICLR 2024 poster (2024)
Keywords: Automatic speech recognition
large language model
generative error correction.
Primary area: representation learning for computer vision, audio, language, and other modalities

Abstract: Recent studies have successfully shown that large language models (LLMs) can be successfully used for generative error correction (GER) on top of the automatic speech recognition (ASR) output. Specifically, an LLM is utilized to carry out a direct mapping from the N-best hypotheses list generated by an ASR system to the predicted output transcription. However, despite its effectiveness, GER introduces extra data uncertainty since the LLM is trained without taking into account acoustic information available in the speech signal. In this work, we aim to overcome such a limitation by infusing acoustic information before generating the predicted transcription through a novel late fusion solution termed Uncertainty-Aware Dynamic Fusion (UADF). UADF is a multimodal fusion approach implemented into an auto-regressive decoding process and works in two stages: (i) It first analyzes and calibrates the token-level LLM decision, and (ii) it then dynamically assimilates the information from the acoustic modality. Experimental evidence collected from various ASR tasks shows that UADF surpasses existing fusion mechanisms in several ways. It yields significant improvements in word error rate (WER) while mitigating data uncertainty issues in LLM and addressing the poor generalization relied with sole modality during fusion. We also demonstrate that UADF seamlessly adapts to audio-visual speech recognition.

Published as a conference paper at ICLR 2024
IT's NEVER ToO LATE: FUSING ACOUSTIC INFOR- MATION INTO LARGE LANGUAGE MODELS FOR AU- TOMATIC SPEECH RECOGNITION
Chen Chen Ruizhe Li
Yuchen Hu-
Sabato Marco Siniscalchi
Pin-Yu Chen'
Eng Siong Chnga Chao-Han Huck Yangš*.6
Nanyann Technological University
Univerriity of Aberdeen Georgia Institute of Technology
University of Palermo "NVIDIA Research
MMI-IIBM Waston Al Lab
ABSTRACT
Recent studies have successfully shown that large language models (LLMM) can be successfully used for generative error correction (GER) on top of the automatic speech recognition (ASR) output. Specifically, an LLM is utilized to carry out a direct mapping from the N-best hypotheses list generated by an ASR system the predicted output transcription. However, despite its effectiveness, GER intro- duces extra data uncertainty since the LLM is trained without taking into account acoustic information available in the speech signal. In this work, we aim to over- come such a limitation by infusing acoustic information before generating the predicted transcription through novel late fusion solution termed Uncertainty- Aware Dynamic Fusion (UADF). UADF is a multimodal fusion approach imple- mented into auto-regressive decoding process and works in two stages: (i) It first analyzes and calibrates the token-level LLM decision, and (ii) then dy- namically assimilates the information from the acoustic modality. Experimental evidence collected from various ASR tasks shows that UADF surpasses existing fusion mechanisms in several ways. It yields significant improvements in word eтror rate (WER) while mitigating data uncertainty issues in LLM and addressing the poor generalization relied with sole modality during fusion. We also demon- strate that UADF seamlessly adapts to audio-visual speech recognition.
INTRODUCTION
In recent years, Large Language Models (LLMs) have emerged as an epistemic beacon in the field of natural language processing (NLP), endowing text-based tasks with substantial performance gains through their extensive knowledge repository and remarkable generation capabilities (OpenAl contined 2023 Anil al. 2023 Touvron al. 2023a b). Additionally, the prowess of LLMs
to textual data alone. They have demonstrated the capability to perceive and process non-textual information (Li al., 2023 Lyu et al., 2023 Han al. 2023 Wu al. 20236), bridging the gup between various modalities. This multifaceted understanding allows L.MMs to serve as a universal interface, facilitating an intricate fusion amidst disparate data modalities for multi-modal tasks, e.g-
image description (Wang et al. 2023) and speech translation (Zhang et al. 2023a).
Compared to the semantic-level fusion between images and texts, perceiving speech signals for LLMs remains complex endeavor. The challenge stems from the inherent high sampling rate of acoustic data and the substantial modality gap that exists when transitioning between audio and tex- tual data. Furthermore, prompting LLMs for automatic speech recognition (ASR) tasks presents a particular barrier as ASR requires learning precisely frame-level alignment instead of utterance- level understanding. Previously, the realm of ASR predominantly employed n-grams neural lan- guage models (Yang et al.) (2021a) (LMs) to rescore an N-best hypothesis list, culminating in the selection of the 1-best sentence. A recent development this field, Chen et al. (2023b) introduce a generative error correction (GER) benchmark and prompting methods for LLM-enhanced ASR wherein the N-best hypotheses list provides informative elements directly predict output tran-
"Corresponding authors: CHRNI 43e.. atu.eddu.gg. hucky@nvi di com.

---

Published as a conference paper at ICLR 2024
scription. Notwithstanding its innovation, this methodology inadvertently introduces enhanced level of data uncertainty, primarily because the inherent acoustic information remains agnostic dur- ing LLMs learning. This quandary is further compounded by existing fusion strategies.
Methods such as semantic loken-based early fusion (Fathullah et al. 2023 Deshmukh et al. 2023 or representation post-encoding through cross-attention (termed as mid fusion) (Lei et al.; 2023
Radhakrishnan et al. 2023a) offer potential solutions. However, the early fusion approach tends to bias the model's learning towards a specific modality quite easily, and the mid fusion suffers from the typically longer sequence length of speech signals compared to text sequences (Wu et al. (023a). In addition, their efficacy is often cireumscribed by inherent modality disparity. Furthermore, pre- maturely fusing two modalities may give rise to modality laziness problem (Du et al. 2003aa) which describes the neural network's 's tendency to excessively rely on a specific modality that even multi- modal performance does not surpass that of unimodal performance. This paper tries to shed light upon multimodal fusion for LL.M-ehhanced ASR tasks. Based on the GER-emper emppweeed Hypotheses- to-transcription (H2T) paradigm, LLMs acquire the capacity to provide an independent probability distribution based on text-only modality, allowing us to explore a fundamental fusion strategy for auto-regressive token prediction. In particular, Aware Dynamic Fusion (UADF) that performs step-wise late fusion in the regresseeee decoding process. Key our framework, we leverage the token-level uncertainty estimation to dynamically determine the fusion weight allocated to each modality in each decoding step. This mechanism is consistent with human multimodal perception: when the primary modality equivocal, then we spontancously seek compensatory information from another modality.
devise novel framework named Uncertainty-
In summary, we make several salient contributions. Firstly, we underscore the challenges inherent in fusing acoustic information into LLMs for ASR and implement several feasible fusion strate- gies for comparative analysis. Secondly, we introduce a novel uncertainty- technique. UADF, which dynamically allocates modality weights in the auto-regressive decoding process, thus significantly reducing the likelihood of the occurrence of the modality laziness phe- nomenon. Thirdly, experimental evidence shows UADF achieves remarkable performance gain in terms of word error rate (WER), outperforming a bunch of established GER baselines. Lastly, UADF demonstrates strong generalization other multimodal auto-regressive tasks that are scamlessly in- corporated as a plug-in in for audio-visual speech recognition.
dynamic fusion
RELATED WORK
Language Modeling in ASR. To improve linguistic acceptability. there has been considrrable prior
efforts in applying Language Models (LMs) within ASR system (Jelinek 1976 Jjojee et al.
1999
Mohri et al. 2008 Sak et al.
2010 Chorowski Jaitly (016)] Chen et 2019 Hu et al. 2020
Wang et al.) 2022 Liu et al.
2023a). ASR designs have been firstly explored as acoustic model
(A.M) and a language model (LM), independently trained, within a noisy channel framework (Je-
linek 1976; Dixon & Silverman 1975). The LM could be integrated in an efficient first-pass decod- ing (Kuo et 20022 Mohri et al., [2008: Liu et al. 2023b) and in second rescoring to manage larger L.ss (Ljolje et al. 1999 et al. 2010). Despite the shift to hybrid HMM-DNN models, the basic decoding/rescoring structure persisted and led to new fusion approaches for LM integration un- der the emergence of end-of-end (E2E) ASR models (Chorowski & Jaitly 2016 Sriram al. (017). Further advancements included a two-pass E2E ASR combining streaming and full-context decod- ing (Sainath et al. 2019), and a deliberation network that enhanced output generation by attending to both acoustic representations and first-pass hypotheses (Hu et al.) 2020), showcasing the ongoing evolution and integration of LMs in ASR. With recent advancements in pre-trained language models (PLMs), language models have been playing versatile roles within ASR systems, e.g., bi-directional rescoring (Xu et al. 2022), knowledge distillation (Futami et al. 2020). and error correction (Leng et al.) [023] Chen et al. (2023a). More recently. Chen et al. (2023b); Yang et al. (2023): >; RRadhakrish- nan et al. (2023a) proposed an LLM-enhanced ASR benchmark called generative error correction, which learns
mapping by LLMs with LoRA adapter. In particular, since GER enables LLMs to predict transcription based on text-only modality, this work leverages this capacity and considers integrating audio modality into LLMs.
Fusion based Multimodal Learning. Multimodal fusion is one of the most fundamental topics in multimodal learning, which aims integrate available modalities into a uniform learning frame-
2

---

Published as a conference paper at ICLR 2024
LLM
XR
Tokenizer
N-best List
Encoder
Encoder
N-best List
X
X
snc
MHA KAV
MHA V
Adapter
Decoder
LLM
X
LLM
N-best List
UADF
(i) Early fusion
(ii) Mid fusion
(iii) Late fusion
Figure I: Different fusion strategies: early, mid and late fusions. The the green area indicates where the fusion strategies happened. N-best List is generated by ASR engine with beam search decoding. Left: the speech tokens extracted from the acoustic encoder are directly concatenated with the cor- responding word embeddings of the N-best list before feeding into the LLMs: Middle: the acoustic features from the last layer of the acoustic encoder are integrated into the LLMs decoding process using the cross-attention mechanism; Right: the step-wise fusion happens in the auto-regressive decoding process by integrating both decision-level information.
work (Xu et al. 2021 Yang al. 2021b Zhang et al., 2023b Peng al. 2023 Yi et al. 2023
However, due to inter-modal disparities, unified learning framework often results in imbalances of modalities. This phenomenon is defined modality laziness, referring to situations where multi- modal performance is worse than single-modal performance (Du et al. 2023a). A common solution utilize late fusion to preserve uni-modal learning (Hessel & Lee 2020 Yao Mihalcea [0022)
or to employ knowledge distillation prior to modality fusion (Wang et al. 20200 Peng et al. (0022). In
the ASR task, most efforts on multimodal fusion focus on -viuul speech recognition (Afouras
et al. 201 8a
Hsu & Shi. (022). However, although the modality laziness phenomenon is also re-
ported
(Du et al. | 2023a), limited research focuses on addressing it.
Uncertainty Estimation in Auto-regressive Task. The motivation of uncertainty estimation is to evaluate the reliability of neural model's predictions, which is typically measured by Bayesian neural networks (BNNs) (Neal) (012) and its varieties (Gal & Ghahramani [016] |Han et al.) [2022).
(Malinin Gales| 2020) first develop an ensemble-based uncertainty estimation framework for auto- regressive prediction. In the ASR task, uncertainty estimation is also explored for improving noise- robustness (Tran et al., 2014; Stouten et al., 2006), knowledge distillation (Kim et al.) 202I), and intelligibility prediction (Tu et al.) (2022). Furthermore, Zhang et al. (2023c) theoretically proves that uncertainty estimation can also be applied to modality fusion, where an energy score is utilized determine dynamic weight for each modality. This work extends the theory to the context of autoregressive decoding, which performs step-wise late fusion based on the uncertainty of LLMs' predictions.
2.1 A GENERATIVE FRAMEWORK ASR ERROR CORRECTION
Given the speech signal X E R', the ASR task aims to predict its textual transcription with I' sequen- tial tokens Yr (y1, 3/,- , .r) with a neural network. HyPoradise dataset (Chen et al. 2023b)
provides an informative N-best list consisting of n hypotheses candidates Di, {Y1, Y2, ... Y,, using beam search, and then learn dypotheses-to-transeeee troaassccriptinn (H2T) mapping in auto-regressive manner:
Y- = MHzz (Dn, 01).
P(Yr)Din 01) II P(ynYYe, [=
(1)
where Or denotes a pre-trained LLM with LoRA adapter, and Y=e denotes the history sequence (y1, "",-.. 1)) It is worth noting that such a learning paradigm is text-only, as X is not directly involved in the calculation of P(Y.). Consequently, it introduces extra expected data uncertainty when predicting transcription. In this work, we focus on integrating X or its hidden representation into H2T mapping, which can be written as:
T
P(Yr)Din 0,
II PlynYYce,Din. t=
(2)
3

---

Published as a conference paper at ICLR 2024
More details about the HyPoradise dataset and relevant H2T learning are attached in Appendix A.1
ACOUSTIC INFORMATION FUSION
In this part, we first illustrate different fusion strategies including early, mid, and late fusion. Then we concentrate on late fusion, and introduce the relevant techniques in the proposed UADF.
3.1 FUSION STRATEGY
Considering the long- rangrannge character of the raw speech signals X, we employ neutral network 8g to extract its hidden representation. In this work, we investigate three common speech representations in Transformer-based ASR model named: (i) Xtok: Speech tokens extracted by self-supervised
learning, e.g., Wave2Vec (Schneider al., 2019). (ii).enne acoustic features in the last layer of the acoustic encoder, and (ii) Xdeci acoustic features in the last layer of the ASR decoder. These three kinds representations range from shallow to deep, which corresponds to different fusion approaches shown in F
Early Fusion with Speech Tokens and Language Embedding: An early fusion approach directly concatenates the speech tokens Xtok and word embeddings Yook before feeding into the first self- attention layer LLM decoder, where Xtok requires to be projected to the dimension the Yook ensure compatibility. In practice, we follow the stacking approach introduced in (Fathullah al., 2023) to reduce the length of tok- Subsequently, the N serves as prompt tokens that are fed into the LLM decoder, and perform auto-regressive decoding as follows:
T II P(ye Concaa(Xxass Yet), ==
P(Y+)
Considering the Xtok and Dn are from distinct modalities, prematurely fusing them may lead to modality laziness (Du al. 2023b) due to modality gap, LLMs proficiently handle D, while remains entirely unacquainted with Xtok-
Mid-Fusion thorough Model Attention Merging: model attention merging is one recent neural adapter- based techniques (Lin et al. 2023 Radhakrishhan et al.) 2023a) (i.e., Whispering-LLLaMa that utilizes the cross-attention mechanism the LLM decoder to integrate Xene into the decoding process. Specifically, we utilize Xeno as key K (Xeene) and value V ([Xene) matrices, then perform cross-attention using query Qum in LLM decoder layer. The final layer output is obtained by sum- ming Ò and original self-attention representation with a fixed weight
Ć = soft Qit (K(e)n))) V(Xenc) Nenc)
C = moft Qum (Kum)) Vitm)
(4)
Vdz
Vdk
Two considerations should be addressed when applying mid fusion in LLMs: 1) the Xene requires to be aligned with Qum as the latent dimensions are usually mismatched between the acoustic model and LLMs. A typical solution is adding a trainable adapter to ensure dimensional compatibility, which also serves as a modality converter from audio to text (Chen et al. 2023d] Yang et al. 2023) 2) Since mid fusion happens in each layer of LLMs, the tuning approach is expected balance the perception of new modality with the retention of pre-trained knowledge.
In this paper, we employ residual adapter with a down-up internal structure for modality transfer and dimension alignment. Furthermore, we keep most LLMs parameters frozen for the retention of pre-trained knowledge, and conduct both prompt-tuning and LoRA adapter (Yu et al. 2023) in each
decoder layer to perceive acoustic representations Xene- More details and discussion attached in the Appendix A.3 and Appendix A.55
Late Fusion in Auto-regressive Decoding: A step-wise late-fusion happens in the auto-regressive decoding process that integrates decision-level information to predict the current token. Therefore, decompose the X into f:a=( (X) according to step t, which is sequentially calculated by an independent encoder-decoder ASR model. Moreover, fr (X) can be viewed as logits that indi- cates the probability distribution the vocabulary space V. Although no feature-level alignment
4

---

Published as a conference paper at ICLR 2024
is needed, late fusion requires consistent decoding space between the ASR model and LLMs, and then performing weighting fusion to obtain t-step logits ft:
Jt = Im !!lm (D..) + w "="""""
St (x),,
and pllm "5 E R.V.
(5)
Specifically, considering the pre-existence of fusion techniques between ASR and LM, we herein elucidate the connection between our work and several existing approaches. Both mid fusion and deep fusion (Gulcehre et al.] 2015) involve integration at the level of hidden features. However, mid fusion further capitalizes on the cross-attention mechanism of the LLM decoder. Late fusion and shallow fusion (Kannan et al. 2018) exhibit similarities, yet LLM in late fusion scheme can uniquely leverage the GER to independently predict transcriptions without the assistant of ASR model. Additionally, unlike cold fusion (Sriram et al. 2017), late fusion avoids the subsequent training process after fusing the information in ruto-regressive decoding.
Considering late fusion fuses information in the decision-level stage, it maximally mitigates the emergence of modality laziness problems. The upcoming two chapters will be dedicated to the exploration of two key challenges in late fusion: (i) how to calibrate the token-level logits ft, and (ii) how determine the fusion weights wmm and warr.
3.2 CALIBRATION
The significance of the calibration in late fusion arises from the over-confidence phenomenon (Bai et al. (022) in neural networks, which indicates the confidence score of models is usually higher than their accuracy. In the ASR task, the training process utilizes the Teacher-forcing technique that the Yo in E.q.( 2 are drawn from the ground-truth sequence. However, the trained model has to rely on its own prediction during inference as ground truth is not available. This mismatch leads to exposure bias that exacerbates the over-confidence when calculating the probability of the current token Py. during auto rettrrregeesvve decoding. Furthermore, prior study (Mukhoti et al. 2020) demonstrates that over-confidence can seriously hurt the ensemble performance when integrating the logits of classification models. To alleviate this issue, we adopt temperature scaling approach introduced in (Kumar et al., 2022). Specifically, we establish two temperatures T to match up the confidences of models with their average accuracies small validation set:
Conf(f,7)
max softmax
(6)
Thdre
where Naee denotes all decoding steps accumulated from the samples in the validation set. When + O we obtain a uniform distribution, and when т + oo we obtain Dirac distribution on the most likely output. In practice, we determine the Tı for LLMS and T2 for ASR using binary search algorithm based on token error rate (TER):
Confum (f n) = 1 ( (fllm)) Confasr(f"*) (f"*",T2) ~
TER (faor)
There are several alternative approaches for TER calculation, as the divergence between ASR and LLMs inevitably leads to different history sequences Yer. To unify it, we update the Y<e using greedy strategy based on the calibrated probability ft for token ye. which is written as:
ft = Him softmax
film
+ asr softmax(
St
posr
(8)
3.3 UNCERTAINTY-AWAR DYNAMIC FUSION
An intuitive method to measure um and waer is to estimate two constants according to WER per- formance on the validation set. However, such static fusion strategy leads to higher upper bound of generalization error compared with dynamic fusion, which has been theoretically proven using Rademacher complexity (Bartlett & Mendelson 2002) in multi-modal classification (Zhang et al,, 2023c). More importantly, Zhang et al. (203cc) theoretically identifies the connection between dy- namic multimodal fusion and uncertainty estimation. This connection is relevant to our motivation: we focus on fusing acoustic information to tackle the data uncertainty in the H2T learning of L.Mss. Typically, the uncertainty of a yt by LLMs is given by the entropy of the predictive posterior:
Ultim = -P(x)) logP(w:),
where P(y.) = softmax
5

---

Published as a conference paper at ICLR 2024
A large l means large uncertainty when LLMs predict the current token, which requires more acoustic compensation from fe Since late fusion incorporates only two modalities, and consid- ering the predominant role of LLMs within them, we set the w as I and dynamically modulate we in terms of U, Therefore, the uncertainty-awan dynamic fusion in auto-regressive decoding can be written as:
fllm
past
P(YT) I softmax(softmark t=0
(sigmoid((4"") 3) softmax
(10)
where 3 is parammte with default value of 0.5. From E.q. 10 we observe that if the LLM is extremely confident after calibration (24ttm - 0+)) then the final decision could completely rely on its own decision (sigmoid m 0.5 0+). Otherwise, the weight of the ASR model increases with the increase of mm!.
4 EXPERIMENT
4.1 DATASET
Hy Poradise T (Chen et al. 2023b) generative error correction benchmark for LLM-enhanced ASR task, which contains more than 3 316K bypotheses-transerriiioo pairs collected from mainstream ASR corpus. Specifically, each utterance is equipped with at least 5 hypotheses that are transcribed by Whisper-large-v2 model with beam search decoding. In this work, we employ WSJ and ATIS as clean condition and CHiME-4 as noisy condition from HyPoradise, and more statistic details in Appendix A. and Table In this paper, select the WSJ (Paul & Baker 1992. Garofalo itt al. 2007). ATIS (Hemphill et al. 1990).. CHiME (Vincent et al. 2016), and LRS3 (Afouras et al. 2018b) datasets evaluation the proposed methods. More details can be found in Appendix A.2
4.2 SETUP
H2T Learning. We employ LLaMA-7E from Huggingface as the foundation model for H2T learn- ing. A low-rank adapter is inserted into each layer of LLaMA with the rank of 8. We use a uniform prompt template to transform the N-best list into inputs suitable for LLMs. More training details and typer-parameters can be found in Appendix [.. Additionally, we employ the GER method as baseline and report the results in the next section.
Early and Mid Fusion baselines. YeoR is extracted from raw speech signal by Wav2vec2-large and HuBERT pre-trained models. They have both been trained by CTC loss on the LibriSpeech dataset. Since the dimension of X is 1024, we stack 4 Xtok to align with the dimension of LLaMA's word embedding and reduce the length of Xtok- Xene extracted by Whisper encoder.
Uncertainty Aware Dynamic Fusion (UADF). We employ a Whisper-tiny model to provide Xdec for late fusion. Since it can independently calculate WER, we term it as ASR-only baseline in the experiments. To align the decoding space with I.MMs, we fix the encoder and finetune the decoder using I.LMMA's word embeddings, Tokenizer, and special tokens. We randomly select a small validation set with 200 training examples from the training set to determine and 72 using binary search, as well as selecting the best model. With the same validation set, we also utilize the grid search to find out the best-fixed weight for LLMs and ASR models and perform static late fusion as our baseline. Additionally, the 3 in E.q. 10 is set as default 0.5 for both ATIS and WSJ.
5 RESULT AND ANALYSIS
In this section, we conduct experiments and answer the following questions: (i) What is the perfor- mance of different fusion strategies when integrating acoustic information into LLMs, (ii) Does the proposed UADF method surpass its counterparts (static fusion) using late fusion, and (iii) How does the generalization ability of UADF, and can be seamlessly applied other reR-rel tasks?
'https//huggingfacc.oo https//hhggigghcccooooaasss///ee hwwaa///awwaaaaaaaaaaaaaaaa/aaae
hhtps://huggingffce.c hhtps//huuginngaccccooocooodddd http://gw.....c.ooooooocrerrraaa

---

Published as a conference paper at ICLR 2024
Table 1: WER (%) and WERR results of early, mid, and late fusion on ATIS and WSJ dataset. "W2v." "Hub." and "Whis." indicate Wav2vec2-large, HuBERT and Whisper model, respectively. "Conc.", "Atten.", ", and "Stat." indicate concatenation, cross-attention and static fusion strategies introduced in 3 "GER" denotes the H2T results of LLM that is consistent across the three fusion methods.
Fusion
GER
ASR-only ATIS
WER +
WERR
1 WSJ
Acoustic Info.
where
how
ATIS
WSJ
WS/
ATIS
WSJ
ATIS
Ntok by W2v.
2.16
3.21
-34.2%
-13.4%
early
Cone.
1.61
2.83
by Hub.
2.02
3.11
-25.5%
-9.9%
Xene by Whis.
mid
Atten.
1.61
2.83
1.75
2.59
-8.7%
8.5%
Stat.
1,36
2.55
15.5%
9.9%
dec by ASR
late
1.61
2.83
4.67
UADF
1.24
2.47
23.0%
12.7%
Table 2: Ablation study of WER (%) and WERR results on the ATIS dataset based on UADF using late fusion. The difference between the system ID-1 to ID-3 is the different performance of ASR- only model (Xdec), and the system ID-4 to ID-5 varies based on the different combination of "Cali." and "Dyn.". " "Static" does not utilize either "Cali." " and "Dyn."
System ID
ASR-only (Xdec)
ID-C
Static
UADF
GER
WER
WERR
WER
WERR
Cali.
Dyn.
WER
WERR
12.16
2.41
-49.7%
1,51
6.2%
1.52
5.6%
2
1.61
8.22
1.96
-21.7%
1.45
9.9%
/
/
1.39
13.7%
3
4.67
1.57
2.5%
1.36
15.5%
/
1.24
23.0%
4
/
x
1.33
17.4%
1.61
4.67
1.57
2.5%
1.36
15.5%
5
X
V
1.39
13.7%
We employ the word error rate (WER) and word error rate reduction (WERR) to evaluate the per- formance. A lower WER parameter signifies better performance, while a higher WERR indicates a greater improvement relative to GER.
5.1 EFFECT OF FUSION STRATEGIES
We first report the WER performance on ATIS and WSJ for different fusion strategies in Table T ASR-onle is calculated by Xeec and thus only appears in late fusion. From Table we observe that:
(i) Early fusion performs slightly below the GER baseline in terms of WER, regardless of using Wav2vec2-large
HuBERT tokenizer. The underlying reason is intuitive: when concatenate X token and D,,, the language model has no knowledge of Xtoken but is well-acquainted with the Dn. leading to the occurrence of modal laziness. In other words, the acoustic information Xtoken would be regarded as a form of linguistic "noise" we treat the concatenated Xtoken and the n-best list prefix tokens. (ii) Mid fusion on the WSJ dataset shows better performance than WSJ due to the larger data amount. It indicates that mid fusion requires more training examples to overcome the modal disparities in cross-attention. (iii) Late fusion achieves considerable performance gains compared with GER baseline, where UADF respectively reduces the relative WER by 23.0% and 12.7% on ATIS and WSJ datasets. Surprisingly, despite the ordinary performance of the ASR model, static weight sum approach yields better WER results than GER. Additionally, we observe that when the ratios of um and wer fall within a certain range (e.g., "m / aar the combination produces highly similar results.
2 on ATIS),
To visualize the effect UADF, we conducted case study to show how the UADF performs late fusion to correct LLM's token-level decision. Figure 2 is real case drawing from the decoding process of ATIS test set. this case, LLM predicts the current token as ID-5521 ("how") but with high uncertainty (4 9.91). According to E.q. 10 UADF allocates high weight (z 0.5) to the ASR model, resulting a token ("all") with ID-484 as the final decision that is consistent with ground truth token.

---

Published as a conference paper at ICLR 2024
Table 3: WER (%) and WERR results of Noise robust ASR results on Chime-4 dataset. The best results are in bold.
Noise Type
Static
UADF
ASR-only
GER
WER
WERR
WER
WERR
bus
12.45
8.67
8.05
7.2%
7.98
8.0%
caf
11.48
6.96
6.37
8.5%
6.22
10.6% %
ped
11.36
5.49
4.96
9.8%
4.82
12.2%
str 12.28
5.86
5.28
9.9%
5.28
9.9%
Avg.
11.89
6.75
6.17
8.6%
6.08
9.9%
5.2
EFFECT OF UADF
We then conduct ablation study analyze the effectiveness of UADF in late fusion. There three primary factors that influence the performance of UADF, as shown in Table 2 which are: the performance of ASR-only model (Xdec), the calibration operation ("Cali.") and the uncertainty- based dynamic fusion ("Dyn."). "ID-C" is late based baseline that only utilizes the same calibration method with two same constants "m and wasr for decision in classification (Kumar al. 2022). "Static" is to estimate the value of wtt and wasr according to WER the validation set without calibration.
In T we observe that: (i) As the ASR-only models (ID-1 to ID-3) exhibit progressively better performance, the late fusion results become better in terms of WER. It is noteworthy that even a modest-performing ASR-only model (12.16%) with UADF can yield substantial performance gains (1.61% + 1.52%) compared with GER. (ii) From the performance of "ID-C", is evident that performing calibration in isolation without weight searching does not enhance GER performance. This is primarily due to the complexity of auto-regressive decoding. However, calibration plays crucial role in UADE, since it can mitigate the issue of overconfidence in our models, thereby encouraging diversity in fusion decisions. (iii) UADF can achieve better WER performance than static fusion, as it adaptively assimilates the decision of X terms of uncertainty. Furthermore, compared with static fusion, UADF avoids searching the fusion weight on a validation set.
To support viewpoint (ii) and illustrate the importance of calibration in UADF, we visualize the accuracy, con- fidence, and token distribution on the ATIS test Figure 3 where the histogram denotes the token dis- tribution according to LLM's confidence, and the "x" denotes the actual average accuracy based on each con- fidence interval. In the left part, LLM shows the ob- vious over-confident phenomenon: more than 98.9% of token predictions fall within the confidence interval of 0.9 to 1.0, and their average confidence is 99.9%, which is higher than the true accuracy of 96.5%. In other intervals, the confidence is also higher than the actual accuracy, since all "x" are below the dashed line. After calibration, the overconfidence issue is sig- nificantly alleviated, as shown in the right part of Fig- 3 93.7% tokens fall within the confidence interval of 0.9 to 1.0, while the average confidence has dropped to 97.03% which similar to the accuracy of 96.5%. Additionally, calibration can affect subsequent uncer- tainty estimation, enabling the identification of token decisions where the LLM performs poorly (e.g., the case in Figure], 2 and facilitating the dynamic incor- poration of decision information from the ASR model.
Prob.
1.0
x('all')
UADF X
ASR X
0.8
LLM X
0.6
('all') Decsion
0.4
0.2
('how')
Token ID
x
0
484
5521
11906
Figure 2: Case study on a high uncertainty (4, Im is 9.91) example. Top-2 candidates from LLM are displayed while the "how" is a wrong prediction. UADF corrects the results "all" according the decision of the ASR model.
8

---

Published as a conference paper at ICLR 2024
Acc. "x"
Token Num.
Acc.
"x"
Token Num.
1.0
13k
1.0
X
12k
0.8
12k
0.8
+
+
4k
X
11k
+
400
0.6
0.6
10k
+
100
+
0.4
x
60
0.4
80
x
+
45
60
0.2
30
0.2
40
15
20
0
0.6
0.8
1.0
0
x
0.2
0.4
0.6
0.8
1.0
0
Confidence
Calibration
Confidence
Figure 3: The visualization of before (left) and after (right) calibration for LLM in UADE. The dashed line represents the ideal relationship where confidence and accuracy are perfectly matched. The blue bar indicates the token distribution under different LLM's confidence intervals, and "x" indicates the actual average accuracy based on each confidence interval.
5.3 GENERALIZATION OF UADF
We first consider examining the UADF's generalization on noise-robust ASR task, as background noise can increase the variability among hypotheses in the N-best list, thus leading to higher uncer- tainty for LLMs when predicting transcription. The results of UADF on the CHiME-4 dataset are reported in Table terms of noise categories. We observe that the performance of ASR-only GER slightly drops due to noise interference compared with ATIS and WSJ. However, UADF approach can yield significant performance gains across different noise environments. Furthermore, same clean conditions, UADF outperforms the static baseline due to the sigmoid function, which can normalize the high uncertainty of individual tokens, mitigating over-reliance on the ASR model.
We then validate the effect of UADF in the audio-visual speech recognition (AVSR) task, where noise-invariant visual modality is utilized to provide compensation information for speech recog- nition. worth noting that the modality laziness phenomenon the system tends to overly rely the audio modality due to its higher recognition ease. More in- troduction and discussion about AVSR are attached in Appendix [A.4] With the proposed UADF, perform late fusion AV-HuBERT baseline (Shi al. 2022b) and pre-trained lip-reading model (Shi et al., 2022a). Besides static fusion, we employ MSRL (Chen et al., 2023c) as baseline, which integrates two models in reinforcement lcarning-basse manner. In Table 6 we observe that multimodal AV-HuBERT achieves worse performance than unimodal V-HuBERT due to modality laziness. Accordingly, all three methods can effectively improve noise-robust by reusing the dependent visual modality. Furthermore, UADF surpasses static fusion in all conditions in terms of WER and achieves comparable performance with MSRL. Notably, MSRL requires an extra training process for reinforcement learning while our UADF is training-free.
particularly in AVSR because
CONCLUSION
In this paper, we ask basic yet well-discovered question: how can audio information be integrated into Large Language Models (LLMs) for GER-based speech recognition tasks? After exploring multiple fusion strategies different levels, present simple yet effective solution UADF that performs late fusion in the auto-regressive decoding process. Benefiting from uncertainty estimation of LLM outputs, UADF dynamically assimilates information from the audio modality, leading to more reasonable token-level decisions. Experimental evidence demonstrates that our method can avoid modality laziness, yielding better WER performance gain the GER compared with other fusion strategies. Additionally, UADF seamlessly adapts to noise-robust ASR as well as AVSR.

---

Published as a conference paper at ICLR 2024
REFERENCES
Triantafyllos Afouras, Joon Son Chung, Andrew Senior, Oriol Vinyals, and Andrew Zisserman. Deep audio-visual speech recognition. IEEE transactions on pattern analysis and machine intel- ligence, 44 (4(12)87777-8721 2018a
Triantafyllos Afouras, Joon Son Chung, and Andrew Zisserman. Lrs3-ted: large-scale dataset for visual speech recognition. arXiv preprint arXiv:1809.00496, 2018b.
Rohan Anil, Andrew M Dai, Orhan Firat, Melvin Johnson, Dmitry Lepikhin, Alexandre Passos, Siamak Shakeri, Emanuel Taropa, Paige Bailey, Zhifeng Chen, al. Palm 2 technical report. arXiv preprint 2tti::2335.00000 2023.
Yu Bai, Song Mei, Huan Wang, and Caiming Xiong. Don't just blame over parametrizaaiio for over-confidence: Theoretical analysis of calibration binary classification. In International Con- ference on Machine Learning, pp. 566-576. PMLR, 2021.
Peter L Bartlett and Shahar Mendelson. Rademacher and gaussian complexities Risk bounds and structural results. Journal of Machine Learning Research, 3(Nov):463-482 2002.
Chen Chen, Yuchen Hu, Chao-Han Huck Yang, Hexin Liu, Sabato Marco Siniscalchi, and Eng Siong Chng. Generative error correction for switching speech recognition using large language models. arXiv preprint arXhv:2310.13013 2023a.
Chen Chen, Yuchen Hu, Chao-Han Han Huck Yang. Sabato Marco Siniscalchi, Pin- Yu Chen, and En- siong Chng. Hyporadise: An open baseline for generative speech recognition with large language models. In seiethh Conference on Neural Information Processing Systems Datasets and Benchmarks Track, 2023b.
Chen Chen, Yuchen Hu, Qiang Zhang, Heqing Zou, Beier Zhu, and Eng Siong Chng. Leveraging modality-specific representations for audio-visual speech recognition via reinforcement learning. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 37:11, pp. 12607- 12615, 2023c.
Feilong Chen, Minglun Han, Haozhi Zhao, Qingyang Zhang, Jing Shi, Shuang Xu, and Bo Xu. foreign
X-IIm: Bootstrapping advanced large language models by treating languages. arXiv preprint arXiv:2305.044 1400, 2023d.
Zhehuai Chen, Mahaveer Jain, Yongqiang Wang, Michael L Seltzer, and Christian Fuegen. End-to- end contextual speech recognition using class language models and a token passing decoder. In ICASSP 2019-2019 IEEE International Conference on Acousties, Speech and Signal Processing (ICASSP), pp. 6186 6190. IEEE, 2019.
Zih-Ching Chen, Chin-Lun Fu, Chih- Ying Liu, Shang- Wen Daniel Li, and Hung-yi Lee. Explor- ing efficient-tuning methods self-supervised speech models. In 2022 IEEE Spoken Language Technology Workshop (SLT), pp. 1120-1 127. IEEE, 2023e.
Zih- Zih-Ching Chen, Chao-Han Huck Yang, Bo Li, Yu Zhang, Nanxin Chen, Shou-Yin Chang. Rohit Prabhavalkar, Hung-yi Lee, and Tara N Sainath. How to estimate model transfcrability of pre- trained speech models? Proc. Interspeech, 2023f.
Jan Chorowski and Navdeep Jaitly. Towards better decoding and language model integration in sequence to sequence models. arXiv preprint arXiv: 1el..02622...... 2016.
Soham Deshmukh, Benjamin Elizalde, Rita Singh, and Huaming Wang. Pengi: An audio language model for audio tasks. arXiv preprint 2305v: 1834, 2023.
N Dixon and H Silverman. description of parametrically controlled modular structure for speech processing. IEEE Transactions on Acoustics, Speech, and Signal Processing, 23(1):87-91, 1975.
Chenzhuang Du, Jiaye Teng, Tingle Li, Yichen Liu, Tianyuan Yuan, Yue Wang, Yang Yuan, and Hang Zhao. On uni-modal feature learning in supervised multi-modal learning. In Andreas Krause, Emma Brunskill, Kyunghyun Cho, Barbara Engelhardt, Sivan Sabato, and Jonathan Scar- lett (eds.), Proceedings of the 40th International Conference on Machine Learning, volume 202 of Proceedings of Machine Learning Research, pp. 8632 8656. PMLR, 23-29 Jul 2023a.
10

---

Published as a conference paper at ICLR 2024
Chenzhuang Du, Jiaye Teng, Tingle Li, Yichen Liu, Tianyuan Yuan, Yue Wang. Yang Yuan, and Hang Zhao. On uni-modal feature learning in supervised multi-modal learning. arXiv preprint arXiv:2305.01233, 2023b.
Yassir Fathullah, Chunyang Wu, Egor Lakomkin, Junteng Jia, Yuan Shangguan, Ke Li, Jinxi Guo, Wenhan Xiong, Jay Mahadeokar, Ozlem Kalinli, et al. Prompting large language models with speech recognition abilities. arXiv preprini arXiv:2307. 1195, 2023.
Hayato Futami, Hirofumi Inaguma, Sei Ueno, Masato Mimura, Shinsuke Sakai, and Tatsuya Kawahara. Distilling the knowledge of bert for eequence-t--sqquene asr. arXiv preprint arXiv:2008. 03822, 2020.
Yarin Gal and Zoubin Ghahramani. Dropout bayesian approximation: Representing model uncertainty in deep learning. In international conference on machine learning, pp. 1050-1059.
PMLR, 2016.
John Garofalo, David Graff, Doug Paul, and David Pallett. Csr-i (wsj0) complete. Linguistic Data Consortium, Philadelphia, 2007.
Caglar Gulcehre, Orhan Firat, Kelvin Xu, Kyunghyun Cho, Loic Barrault, Huei-Chi Lin, Fethi Bougares, Holger Schwenk, and Yoshua Bengio. On using monolingual corpora in neural ma- 2015.
chine translation. arXiv preprini arXiv:/503.0r
Jiaming Han, Renrui Zhang, Wenqi Shao, Peng Gao, Peng Xu, Han Xiao, Kaipeng Zhang, Chris Liu, Song Wen, Ziyu Guo, Imagebind-llm: Multi-modality instruction tuning. arXiv preprint arXiv:2309.03900 2023.
Zongbo Han, Zhipeng Liang, Fan Yang. Liu Liu, Langing Li, Yatao Bian, Peilin Zhao, Bingzhe Wu, Changqing Zhang, and Jianhua Yao. Umix: Improving importance weighting for subpopula- tion shift via uncertainty-awee mixup. Advances in Neural Information Processing Systems, 35:
37704 37718, 2022.
Charles T. Hemphill, John J. Godfrey, and George R. Doddington. The ATIS spoken language systems pilot corpus. In Speech and Natural Language: Proceedings of Workshop Held at Hidden Valley, Pennsylvania, June 24-27, 1990, 1990. URL httpsss//sclaancholeeyyy
H9C-1021
Jack Hessel and Lillian Lee. Does my multimodal model learn cross-modal interactions? it's harder to tell than you might think! Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pp. 861-877, 2020.
Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzebski, Bruna Morrone, Quentin De Laroussilhe, An- drea Gesmundo, Mona Attariyan, and Sylvain Gelly. Parameter-efficient transfer learning for nlp. In International Conference on Machine Learning, pp. 2790-2799. PMLR, 2019.
Wei-Ning Hsu and Bowen Shi. u-hubert: Unified mixed-modal speech pretraining and zero-shot transfer to unlabeled modality. Advances in Neural Information Processing Systems, 35:21157 1577 21170, 2022.
Ke Hu, Tara N Sainath, Ruoming Pang, and Rohit Prabhavalkar. Deliberation model based two- pass end-to-end speech recognition. In ICASSP 2020-2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 7799-7803. IEEE, 2020.
Yun-Ning Hung, Chao-Han Huck Yang, Pin- Yu Chen, and Alexander Lerch. Low-resource music genre classification with cross- cross-modal neural model reprogramming. In Proc. of ICASSP, pp. 1-5. IEEE, 2023.
Frederick Jelinek. Continuous speech recognition by statistical methods. Proceedings of the IEEE, 64(4):532-555 1976.
Anjuli Kannan, Yonghui Wu, Patrick Nguyen, Tara N Sainath, Zhijeng Chen, and Rohit Prab- havalkar. An analysis of incorporating external language model into soquence-to-squemme model. 2018 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 1-5828. IEEE, 2018.
11

---

Published as a conference paper at ICLR 2024
Ho- Go-onyynng Kim, Min-Joong Lee, Hoshik Lee, Tae Gyoon Kang, Jihyun Lee, Eunho Yang, and Sung Ju Hwang. Multi-domain knowledge distillation via uncertainty mncertainty-matching for end-to-end models. In INTERSPEECH 2021, pp. 1311-1315 International Speech Communication As- sociation, 2021.
Ananya Kumar, Tengyu Ma, Percy Liang, and Aditi Raghunathan. Calibrated ensembles can mit- igate accuracy tradeoffs under distribution shift. In Uncertainty in Artificial Intelligence, pp- 1041-1051. PMLR, 2022.
Hong-Kwang Jeff Kuo, Eric Fosier-Lussier, Hui Jiang, and Chin-Hui Lee. Discriminative training of language models for speech recognition. In 2002 IEEE International Conference on Acoustics, Speech, and Signal Processing, volume 1, pp. I-325. IEEE, 2002.
Yuxuan Lei, Dingkang Yang, Mingcheng Li, Shunli Wang, Jiawei Chen, and Lihua Zhang. Text- oriented modality reinforcement network for multimodal sentiment analysis from unaligned mul- timodal sequences. arXiv preprint arXiv: 10007.1.200 13205, 2023.
Yichong Leng, Xu Tan, Wenjie Liu, Kaitao Song, Rui Wang. Xiang- Yang Li, Tao Qin, Ed Lin, and Tie- Yan Liu. Softcorrect: Error correction with soft detection for automatic speech recognition. In Proceedings of the AAAI Conference Artificial Intelligence, volume 37:11, pp. 13034-13042, 2023.
Bo Li, Yuanhan Zhang, Liangyu Chen, Jinghao Wang, Fanyi Pu, Jingkang Yang, Chunyuan Li, and Ziwei Liu. Mimic-it: Multi-modal in-context instruction tuning. arXiv preprint arXiv:2306.05425, 2023.
Yan-Bo Lin, Yi-Lin Sung, Jie Lei, Mohit Bansal, and Gedas Bertasius. Vision transformers are parameter-efficient audio-visual learners. In Proceedings of the IEEE/CVF Conference on Com- puter Vision and Pattern Recognition, pp. 2299-2309, 2023.
Hexin Liu, Leibny Paola Garcia, Xiangyu Zhang, Andy WH Khong, and Sanjeev Khudanpur. Enhancing code-switching speech recognition with interactive language biases. arXiv preprint arXiv:2309. 16953, 2023a.
Hexin Liu, Haihua Xu, Leibny Paola Garcia, Andy WH Khong, Yi He, and Sanjeev Khudanpur. Reducing language confusion for code-switching speech recognition with token-level language diarization. In ICASSP 2023-2023 IEEE International Conference Acoustics, Speech and Signal Processing (ICASSP), pp. 1-5, IEEE, 2023b.
Andrej Ljolje, Fernando Pereira, and Michael Riley. Efficient general lattice generation and rescor- ing. In Sixth European Conference on Speech Communication and Technology, 1999.
Chenyang Lyu, Minghao Wu, Longyue Wang, Xinting Huang, Bingshuai Liu, Zefeng Du, Shuming Shi, and Zhaopeng Tu. Macaw-llm: Multi-modal language modeling with image, audio, video, and text integration. arXiv preprint arXiv:2306.0900) 2023.
Andrey Malinin and Mark Gales. Uncertainty estimation in autoregressive structured prediction. arXiv preprint arXiv:2002.07650, 2020.
Mehryar Mohri, Fernando Pereira, and Michael Riley, Speech recognition with weighted finite-state transducers. Springer Handbook of Speech Processing, pp. 559-584, 2008
Jishnu Mukhoti, Viveka Kulharia, Amartya Sanyal, Stuart Golodetz, Philip Torr, and Puneet Doka- nia. Calibrating deep neural networks using focal loss. Advances in Neural Information Process- ing Systems, 13:15288-15299 2020.
Radford M Neal. Bayesian learning for neural networks, volume 118. Springer Science & Business Media, 2012.
R OpenAl. Gpt-4 technical report. arXiv, pp. 2303-08774 2023.
Douglas B Paul and Janet Baker. The design for the wall street journal- corpus. In Speech and Natural Language: Proceedings of a Workshop Held at Harriman, New York, February 23- 26, 1992, 1992.
12

---

Published as a conference paper at ICLR 2024
Ru Peng, Yawen Zeng, and Jake Zhao. Distill the image nowhere: Inversion knowledge distil- lation for multimodal machine translation. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pp. 2379-2390, 2022.
Zhiliang Peng, Wenhui Wang, Li Dong, Yaru Hao, Shaohan Huang, Shuming Ma, and Furu
Wei. Kosmos-2: Grounding multimodal large language models to the world. arXiv preprint TrXiv:2306.14824. 2023.
Srijith Radhakrishnan, Chao-Han Yang, Sumeer Khan, Rohit Kumar, Narsis Kiani, David Gomez- Cabrero, and Jesper Tegnér. Whispering llama: cross-modal generative error correction frame- work for speech recognition. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pp. 10007-10016, 2023a.
Srijith Radhakrishnan, Chao-Han Huck Yang. Sumeer Ahmad Khan, Narsis A Kiani, David Gomez- Cabrero, and Jesper N Tegner. A parameter-cfficient learning approach to arabic dialect identifi- cation with pre-trained general-purpose speech model. Proc. of Interspeech, 2023b.
Tara N Sainath, Ruoming Pang. David Rybach, Yanzhang He, Rohit Prabhavalkar, Wei Li, Mirkó Visontai, Qiao Liang, Trevor Strohman, Yonghui Wu, et al. Two-pass end-to-end speech recogni- tion. arXiv preprint arXiv: 1908. 10992, 2019.
Haşim Sak, Murat Saraclar, and Tunga Güngör. On-the-fly lattice rescoring for real-time automatic speech recognition. In Eleventh annual conference of the international speech communication association, 2010.
Steffen Schneider, Alexei Baevski, Ronan Collobert, and Michael Auli. wav2vee: Unsupervised pre-training for speech recognition. arXiv preprint arXiv:1904.05862 2019.
Bowen Shi, Wei-Ning Hsu, Kushal Lakhotia, and Abdelrahman Mohamed. Learning audio- visual speech representation by masked multimodal cluster prediction. arXiv:2201.0218. 2022a.
arXiv preprint
Bowen Shi, Wei-Ning Hsu, and Abdelrahman Mohamed. Robust self-supervised audio-visual speech recognition. arXiv preprint arXiv:2201001763 2022b.
David Snyder, Guoguo Chen, and Daniel Povey. Musan: A music, speech, and noise corpus. arXiv preprint 150.00844, 2015.
Amuroop Sriram, Heewoo Jun, Sanjeev Satheesh, and Adam Coates. Cold fusion: Training seq2seq models together with language models. arXiv preprint arXiv:1708.0644 2017.
Veronique Stouten, Patrick Wambacq, et al. Model-based feature enhancement with uncertainty decoding for noise robust asr. Speech communication, 48(1 ():1:):::20000 150-11544, 2006.
Yi-Lin Sung, Linjie Li, Kevin Lin, Zhe Gan, Mohit Bansal, and Lijuan Wang. An empirical study of multimodal model merging. arXiv preprint arXiv:2304. /4933, 2023.
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière. Naman Goyal, Eric Hambro, Faisal Azhar, et Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302. /397/, 2023a.
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Niko- lay Bashlykov. Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open founda- tion and fine-tuned chat models. arXiv preprint arXiv:2307.00220 2023b.
Dung T Tran, Emmanuel Vincent, and Denis Jouvet. Fusion of multiple uncertainty estimators and propagators for noise robust asr. In 2014 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 5512-5516. IEEE, 2014.
Zehai Tu, Ning Ma, and Jon Barker. Unsupervised uncertainty measures of automatic speech recog- nition for non-intrusive speech intelligibility prediction. arXiv preprint arXiv:2204. 042888 2022.
Emmanuel Vincent, Shinji Watanabe, Jon Barker, and Ricard Marxer. The 4th chime speech sep- aration and recognition challenge. URL: http://spandh. des. shef. ac. ukfchime challenge LLast Accessed on / August, 2018). 2016.
13

---

Published as a conference paper at ICLR 2024
Chengyu Wang. Suyang Dai, Yipeng Wang, Fei Yang, Minghui Qiu, Kehan Chen, Wei Zhou, and Jun Huang. Arobert: An ast robust pre-trained language model for spoken language understand- ing. IEEE/ACM Transactions on Audio, Speech, and Language Processing, 30: 30:1207-1218, 2022.
Qi Wang. Liang Zhan, Paul Thompson, and Jiayu Zhou. Multimodal learning with incomplete modalities by knowledge distillation. In Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Diseovery Data Mining, pp. 1828-1838, 2020.
Teng Wang. Jinrui Zhang. Junjie Fei, Yixiao Ge, Hao Zheng, Yunlong Tang, Zhe Li, Mingqi Gao, Shanshan Zhao, Ying Shan, et al. Caption anything: Interactive image description with diverse multimodal controls. arXiv preprint urXiv:2305.02677 2023.
Jian Wu, Yashesh Gaur, Zhuo Chen, Long Zhou, Yimeng Zhu, Tianrui Wang, Jinyu Li, Shujie Liu, Bo Ren, Linquan Liu, et al. On decoder-only architecture for speech-to-text and large language model integration. arXiv preprint arXiv:2307.03911 2023a.
Shengqiong Wu, Hao Fei, Leigang Qu, Wei Ji, and Tat-Seng Chua. Next-gpt: Any-to-any multi- modal Ilm. arXiv preprint 20t::3309005/99 2023b.
Liyan Xu, Yile Gu, Jari Kolehmainen, Haidar Khan, Ankur Gandhe, Ariya Rastrow, Andreas Stol- cke, and Ivan Bulyko. Rescorebert: Discriminative speech recognition rescoring with bert. In Proc. ICASSP, pp. 61 6117-11. IEEE, 2022.
Yiheng Xu, Tengchao Lx, Lei Cui, Guoxin Wang, Yijuan Lu, Dinei Florencio, Cha Zhang, and Furu Wei. Layoutxlm: Multimodal pre-training for multilingual visually-rich document understanding. arXiv preprint arXiv:2104.08866 2021.
Chao-Har Han Huck Yang. Linda Liu, Ankur Gandhe, Yile Gu, Anirudh Raju, Denis Filimonov, and Ivan Bulyko. Multi-task language modeling for improving speech recognition of rare words. In Proc. of ASRU, pp. 1087-1093. IEEF, 2021a.
Chao- Han Huck Yang, Yun-Yun Tsai, and Pin-Ys Yu Chen. Voice2series: Reprogramming acoustic models for time series classification. In International conference on machine learning, pp. 11808- 11819. PMLR, 2021b.
Chao-Han Huck Yang, Yile Gu, Yi-Chieh Liu, Shalini Ghosh, Ivan Bulyko, and Andreas Stolcke. Generative speech recognition error correction with large language models and activvati prompting. In Proc. ASRU, pp. 1-8. IEEE, 2023.
Yiqun Yao and Rada Mihalcea. Nodality-specifee learning rates for effective multimodal additive late-fusion. In Findings of the Association for Computational Linguistics: ACL 2022, pp. 1824 1834, 2022.
Xuanyu Yi, Jiajun Deng, Qianru Sun, Xian-Sheng Hua, Joo-Hwee Lim, and Hanwang Zhang. In- variant training 2d-3d joint hard samples for few-shot point cloud recognition. arXiv preprint arXiv:2308.00960 2023.
Yu Yu, Chao-Han Huck Yang, Jari Kolehmainen, Prashanth G Shivakumar, Yile Gu, Sungho Ryu Roger Ren, Qi Luo, Aditya Gourav, I-Fan Chen, Yi-Chieh Liu, al. Low-rank adapta- tion of large language model rescoring for parameter-efffcient speech recognition. In Proc. of ASRU, pp. 1-8. IEEE, 2023.
Dong Zhang, Shimin Li, Xin Zhang, Jun Zhan, Pengyu Wang, Yaqian Zhou, and Xipeng Qiu. Speechgpt: Empowering large language models with intrinsic cross-modal conversational abil- ities. arXiv preprint arXiv: 1000, 2023a.
Hang Zhang, Xin Li, and Lidong Bing. Video-Ilama: An instruction-tuned audio-visual language model for video understanding. arXiv preprint arXiv:2306.022. 02858. 2023b.
Qingyang Zhang, Haitao Wu, Changqing Zhang, Qinghua Hu, Huazhu Fu, Joey Tianyi Zhou, and Xi Peng. Provable dynamic fusion for low-quality multimodal data. arXiv preprint arXiv:2306.0200 020500 2023c.
14

---

Published as a conference paper at ICLR 2024
A APPENDIX
A.1 HYPORADISE AND H2T LEARNING
Hyporadise dataset provides more than 316k sypotheses-transeripioo pairs, and Table shows the statistic information of WSJ and ATIS we used in this paper. Each utterance is equipped with at least 5 hypotheses that are transcribed by a Whisper-large- model with beam search decoding.
Table 4:
Statistics in terms of the number of traantisee pairs and average utterance length on WSJ and ATIS.
Domain
Training Set
# Pairs
Length
Test Set
# Pairs
Length
Source
Category
dev93 eval92
503
16.7
WSJ
Business news
train-si284
37,514
17.5
333
17.3
ATIS
Airline info.
train
3,964
12.4
test
809
11.3
CHiME4
Noise
train
8,738
17.0
test-real
1,320
16.4
H2T learning indicates learning the mapping relationship from the N-best hypotheses list to tran- scription, which requires the linguistic information of LLMs. The motivation behind this method is to harness the token-level information present within the N-best list, while mainstream ASR meth- ods usually only output the first hypothesis and discard others. Take a Confermer-baseed ASR model (WER 1.8) as an example, the second hypothesis has 14% probability of
trained on LibriSpeech" [ having a lower WER than the first hypothesis. Furthermore, given wrong token in the first utter- ance, there is a 34% probability of finding the correct token in the second utterance. Furthermore, we have an oracle re-ranking method to choose the best hypothesis, the WER would 1.0. If we have an oracle compositional method that uses token-level information to predict transcription, the WER would be 0.6. Both 1.0 and 0.6 surpasses state-of- ofate-of-thean the-art performance by a large margin. To perform H2T learning, we utilize instruction-following prompt template shown as follows:
"Below is a best-hypotheses that is transcribed from an automatic speech recognition system. Write response predict the true transcription using the tokens from other--yyotheeee..ee ### best- hypothesis: {1 (1st utterance ### ###l other-hypothesi:: {2nd ~ 5th utterances] ###Response.
We employ LLaMA as foundation model from huggingface, the learning rate is set 1 and the batch size 128. For the low-rank adapter, we implement by peft Fl where the rank configuration of rank is set 8. For early fusion, Wav2vec2-large" and HuBERT" pre-trained models used to extract Xtok from raw speech signals, which have been trained by CTC loss on the LibriSpeech dataset.
To show the effect of H2T learning, we report our reproduced GER results with LLaMA-7b in Table 1-best denotes the WER of the first hypothesis with highest probability in the given N-best list. Notably, this probability is provided by the ASR beam search decoding. The ond: WER of the "best hypothesis" in N-best hypotheses list, which can be viewed as the upper bound performance of any reranking based methods. The compositional oracle method Ocp is the achievable WER using "all tokens" in N-best hypotheses list. We also attach the results of rerank baseline "LM and proposed UADF for comparison.
A.2 INTRODUCTION OF DATASETS
WSJ (Wall Street Journa)) Paul & Baker 1992 Garofalo al. 2007 is widely-used ASR corpus that focuses on the domains business news and financial data. The training set includes 37514
15

---

Published as a conference paper at ICLR 2024
Table 5: WER (%) results of GER and UADF with more baselines. "** denoted the WER repro- duced in this paper.
Oracle
Dataset
1-best
LMPank
GER
UADF
Onh
Ocp
ATIS
8.9
6.9
1.61"
1.29'
5.2
1.1
WSJ
4.5
4.3
2.83*
2.47*
4.1
1.2
CHiME
11.1
11.0
6.75"
6.08
9.1
2.8
utterances from 101 speakers in a clean environment. The test set consists of dev93 (with 503 utterances) and eval92 (with 333 utterances) and we report the average WER results on them.
ATIS (Airline Travel Information System) (Hemphill et al. 1990) is an ASR dataset that concen- trates on the domain of air travel information, such as flight times, prices, and availability. It contains a training set with 3964 utterances and a test set with 809 utterances that are collected from more than 500 different speakers.
ChiME-4 (Vincent et al., 2016) is dataset that is widely used in noise-robust ASR task. It includes real and simulated noisy recordings four noisy environments, i.e., bus, cafe, pedestrian area, and street junction. This work employ the test-real as test set that is recorded in real noisy conditions.
LRS3 (Lip Reading Sentences) (Afouras et al. 20T8b) is the largest public multimodal dataset for audio-visual speech recognition (AVSR) tasks. It contains more than 400 hours of speech data with paired face images of speakers. We utilize LRS3 to demonstrate the generalization ability of UADF on AVSR tasks.
A.3 DETAILS MID FUSION
To build cross-modal fusion during the intermediate transformer layers, introduce neural adapter based mid-fusion, which inspired by multi-modal attention merging (Sung et al.) 2023
Hung al. 2023 Radhakrishnan et al. (003a).. To fine-tune fused models, we integrate two resid- ual adapter modules (Houlsby et al., 2019 Radhakrishnan et al. 2023b Chen et al., (023e) (A}, and Aw, subsequent to the self-attention modules (SA'p) of the stabilized LLaMA model within each layer. The adapter Ai, is the module in layer assigned to refine the LLaMA model, employ- ing scaled dot product attention mechanism. Conversely, the adapter Aiy is used in layer i for the amalgamation of pre-trained Whisper features with the LLaMA model, adhering to wise-layer fused decoder approach. We employ the Adam optimizer and conduct experiments with learning rates of le le-3, and 5e selecting the best development losses over 25 epochs for evaluation. We select a batch size of 32 and apply a weight decay of 1e For the residual adapter's setup, we fix bottleneck dimensions of 32 after ablation studies. A similar design in vision and croustic-base mid-fusion can also he referred to (Lin et al. 2023). where this baseline can be considered as its language and acoustic-hased variant.
A.4 DISCUSSION OF AUDIO-VISUAL SPEECH RECOGNITION
Audio-visual speech recognition (AVSR) represents cutting-edge interdisciplinary task that inte- grates principles from both auditory and visual processing domains to enhance speech recognition capabilities. This task involves the synchronous analysis of audio signals and visual cues, particu- larly lip movements and facial expressions, to accurately decipher spoken language. AVSR systems leverage the complementary nature of audio and visual information to improve recognition accuracy, especially in noisy environments where traditional audio-only systems might struggle. This tech- nology not only holds promise for advancing human-computer interaction but also offers significant improvements in accessibility for individuals with hearing impairments. Mainstream AVSR meth- ods focus on learning modality-invariant invariant representations by integrating audio and visual modalities into common subspace. Typically, they employ separated encoder for speech and image input, and then concatenate the hidden representation after alignment (Afouras et al.] 2018a| Chen et al. 2023f.. This learning pattern can easily lead to modal laziness because the audio modality is much
16

---

Published as a conference paper at ICLR 2024
Table 6: WER (%) and WERR results of AVSR task on the LRS-3 by late fusion. "Babble" the noise drawn from (Snyder et al. 2015). indicates the need for an additional training procedure.
SNR (Babble)
AV-HuBERT audio-visual
V-HuBERT visual-only
Static w WERR
MSRL
UADF
WER
WER
WERR
WER
WERR
-10
30.3
23.4
22.8%
22.3
26.4%
21.8
28. %
13.5
11.4
15.6%
11.3
16.3%
10.7
20.7%
0
4.9
26.9
4.8
2.0%
4.5
8.2%
4.6
6.1%
2.5
-12.0%
8.0%
-4.0%
Avg.
12.8
10.6
17.2%
10.1
21.1%
9.9
22.7%
Clean
1.45
26.9
1.42
2.1%
1.33
8.3
1.36
6.2%
easier to recognize than visual information, causing neural networks to gradually ignore the role of the visual modality.
Recently, Shi et al. (2022a) proposes a self- supervised learning approach learn bet- ter modality-invariant representation for AVSR task. With large amount of pre-training data, the AV-HuBERT achieves remarkable perfor- mance on LRS-3 dataset. However, based on AV-HuBERT, Chen et al. (2023c) still report modality laziness problem for modality- invariant representation shown in Figure the poddality-invariine representation (green line) exhibits susceptibility noise interfer- ence. More importantly. when SNR smaller than threshold a, the multimodal representa- tion perform worse than visual-only represen- tation. To address it, Chen al. (2023c) pro- poses a reinforcement learning based-method reuse the visual modality representation in ruto-regressive decoding process. It construct trainable policy network to predict the final to- ken probability distribution. We add MSRL for comparison results in Table 6 For static fusion baseline, the w of AV-HuBERT (0.5, 0.65, 0.7, 0.75, 0.85} for SNR {-10. -5, 0, 5, clean), and the wight of AV-HuBERT is 1 For UADF implementation, we observe though WER high, the AV-HuBERT still exhibit over-confidence tendency that is higher than actual accuracy. We replace LLM using AV-HuBERT and estimate the uncertainty after calibration, and the 3 is set as {0, 0.4, 0.5, 0.5, 0.5) respectively.
Word Error Rate (%)
Audio modality-specific Visual speciic
Modality-nvariiat (a+v)
Oracle
SNR Level
Figure 4: Modality laziness reported in (Chen ett al. 2023c), where SNR level denotes the qual- ity of speech.
A.5 DISCUSSION OF N-BEST LIST
For fair comparison, we involve the N-best list in early fusion and mid fusion. However, this operation may introduce a potential issue in early fusion, where the LLM learns that it can predict the answer solely based on the n-best list, thus overlooking the acoustic tokens Xtok To this end, try to remove the n-best list and only employ X as prefix tokens in decoding. However, we found that the L.M unable to identify X, through low-rank tuning, and this might be attributed to the limited amount of training data. We leave it as our future work.
17


Reviews:
[{"review_id": "Ibhuxq04ll", "reviewer": "Reviewer_kWJM", "summary": "The authors present an approach they describe as Uncertainty Aware Dynamic Fusion (UADF) that 1. calibrates the LLM score at the token level accounting for the over-confidence of the neural network when doing auto-regressive decoding, 2. a time-changing uncertainty uncertainty that dynamically adjusts the decision-level fusion of text-based LLM and ASR scores. The results show significant improvement over strong baseline models such as Whisper and wav2vec or Hubert on a variety of tasks. Further the authors demonstrate that the results can be applied in an AVSR task with the dynamic uncertainty improving results in noise over a static fusion.", "questions": "Can you re-frame UADF in a Bayesian framework?", "limitations": "", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "Overall the work is well motivated and demonstrates strong empirical results.", "weaknesses": "The point about calibration in using LLMs for rescoring of ASR hypotheses may be novel, however both calibration of hybrid or neural network ASR model scores and the use of strong language models combined with ASR scores are not novel approaches and have been widely used within speech. \n\nThe paper seems to desire establishing the term \"modality laziness\" as a technical way of describing when multimodal system underachieve compared to a unimodal system; in this case I would describe this as a simply a poorly designed system. There are multiple papers in the literature that describe effective ways of combining audio and visual modalities to yield an improve AVSR system over any single modality. Look for the term \"modality drop out\" and work on audio-visual conformers or AV-hubert.\n\nApplying the entropy of the LLM predictive posterior eqn (9)  as a weighting factor for the dynamic fusion neatly correlates with the desired property of: when the LLM is more certain, then the ASR weight of the ASR can be less, and when the LLM is uncertain, then the ASR weight is higher.  However, it doesn't stem from the statistical approach of ASR where \nY_T = argmax Y_n p(X|Y_n; M_am) P(Y_n; M_lm ) where the first term in the argmax is the acoustic score and the second term the text-based LLM score, which could be computed with a LLM. In this light, without a Bayesian motivation, one might consider UADF as a hueristic method of combining LLM scores with ASR.\n\nLastly, the results here are all conducted on speech recognition tasks. Its unclear if such an approach can be applied to non-speech tasks or is of interest to the broader ICLR community.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ec1qg2gFIH", "reviewer": "Reviewer_2SYt", "summary": "Recent studies showed that large language models (LLMs) can be successfully used to map the N-best hypotheses list generated by an ASR system to the predicted output transcription to improve ASR accuracy. But LLM is not trained with acoustic information. This paper proposed a method named Uncertainty Aware Dynamic Fusion (UADF) to infuse acoustic information to the auto-regressive decoding process with ASR decoder and LLM predictions. Specifically, it uses late fusion to combine the probability output by ASR decoder and LLM to predict the recognition results. In this process, temperature scales are applied to the logits of ASR decoder and LLM to match up the model confidence and recognition accuracy. Besides, the ASR and LLM combination weights are decided by LLM entropy so when LLM’s uncertainty is larger, more compensation from the ASR model will be used to decide the recognition results. The proposed method showed obvious WER reduction for several ASR tasks.", "questions": "1.\tDo we have the WER of the top 1 hypothesis in HyPoradise dataset for different tasks? These values may not be useful to validate the proposed method. But it will make the readers understand better why we need to use LLM to refine the N Hypotheses.  \n2.\tAs noted above, tuning of temperature t1 and t2 becomes a key point to make the proposed method work. The questions about this are: \na.\tDid the author observe the same conclusion for other tasks (WSJ,  Chime-4, LRS3)? \nb.\tDo we need to tune these values for different tasks, or we could get them with one task’s data and applied them to other tasks.\nc.\tHow much minimum data should we use to tune these values? \nd.\tWhat’s the optimal value for these two parameters for the tasks in this paper? \n3.\tIn eq(10), the weight of LM and ASR didn’t sum up to 1.0. Will it matter since this means for different t, the range of P(y_t) will be different. \n4.\tIt’s noted that “beta” in eq. (10) is set to 0.5 for the experiments. It didn’t show what’s the performance is if we change it to other values. Will the results be sensitive to this value?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 3, "strengths": "The paper proposed several methods applied to the fusion of ASR decoder and LLM output to improve performance. The paper compared the proposed method with several existing methods and did some ablation study to analyze the effectiveness of the proposed method. It also did experiments to prove the generalization of the proposed method. All the results showed the effectiveness of the proposed method.", "weaknesses": "•\tSome contents are not accurate or clear. \no\tIn eq. (9), the entropy of LLM output is not accurate since for a discrete distribution, the entropy should be the sum of -p(y_t,i)logp(y_t,i). \no\tIn figure 2, it’s clear the probability of “all” from ASR is high. But it didn’t show where is the probability of “all” from LLM ? And what’s the meaning to show the blue cross in the left bottom part and the green cross in the right bottom part? \no\tIn the experiments part, for all late fusion experiments, the ASR model needs to be retrained with the training data. But it’s not clear whether all the training data are used to get a unified model or for each task (WSJ, ATIS, Chime-4), the model is updated only with the training data from this task. \no\tThe results in table 2 showed that without calibration, the proposed UADF did not perform better than the static method (WER 1.39 vs. 1.36). This means calibration is crucial for UADF, so the tuning of temperature t1 and t2 becomes a key to make it work. But the paper didn’t give enough information on these two parameters, such as what’s the optimal value, should we tune them for different tasks and so on.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "fuF73OSo5o", "reviewer": "Reviewer_1EWw", "summary": "This paper introduces a late fusion strategy called \"UADF\" that combines the modalities of LLM and ASR to enhance generative error correction. The research demonstrates promising results across multiple datasets.", "questions": "1. What distinguishes the proposed method from shallow fusion using LLM?\n2. In equation 5, do $f^{llm}$ and $f^{asr}$ need to be of the same length? If they do, how to ensure this?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. Integrating acoustic information with LLM is a promising strategy that enhances the capabilities and potential of LLMs in ASR tasks. This approach leverages the strengths of both acoustic processing and advanced language modeling, providing a more robust framework for GER.\n2. The paper conducts a comprehensive examination of different fusion strategies, delving into detailed analyses of each approach.\n3. The proposed UASF yields promising results across datasets with varying conditions, including ASR and VASR.", "weaknesses": "I believe it's not entirely fair to make comparisons with GER given the use of acoustic information.  It would be better to add some comparisons of the results with LLM scoring.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "vlfmncpXsh", "reviewer": "Reviewer_3ovr", "summary": "This paper builds upon the recently proposed “generative error correction” (GEC) paradigm for speech recognition (ASR). The previous GEC work used a text-only LLM to map from a list of ASR hypotheses to a single transcription. In this paper, the authors propose to add acoustic information to this mapping process by including speech representation in the GEC. They experiment with several such “fusion” strategies (so-called early, mid, and late fusion), and finally show that late fusion with uncertainty awareness works best. Evaluations are conducted on a number of ASR benchmarks, showing significant WER improvements compared to GEC or ASR-only baselines, and the method is also shown to generalize to audio-visual ASR.", "questions": "Some of the proposed methods are specific to particular model choices. For example, (i) “early fusion” assumes that we are using Wav2Vec 2.0 as the encoder, and (ii) late fusion would only work with an encoder-decoder style model. The dominant ASR modeling strategy in the industry is conformer-transducers, which do not have such speech tokens, and their logit space is 3-dimensional (where vocabulary dimension also includes a blank token). Have the authors considered how the proposed method would work with such models?", "limitations": "", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. Using LLMs for ASR error correction is an interesting idea and has shown strong results in the past. ASR models often perform badly on rare words such as named entities, and LLMs can be a useful component to improve recognition of such words. In the earlier GEC work, only the top hypotheses of the ASR model was used for the final transcription. Since these hypotheses may not always contain information from the correct word, it is natural to also use the ASR information more directly. With these considerations, the ideas in this paper are relevant and a logical continuation in this line of work.\n\n2. Incorporating modality-specific uncertainty using the UADF approach is a novel and interesting technique, and seems to provide reasonable improvements compared to static weighting of modalities. For example, in Table 4, we see that particularly on very low SNR conditions, there is a large improvement in WER when using UADF instead of static weights. I also appreciate that the authors show ablation results and analysis for the UADF strategy in Section 5.2.\n\n3. In addition to traditional ASR benchmarks, the authors show that their UADF strategy can be used for audio-visual ASR, where it gives a small improvement over AV-HuBERT.", "weaknesses": "My main concerns about the paper are about the presentation (in terms of relation with prior work) and comparison with baselines in the experiments, which I will detail below.\n\n### This model in the context of prior work\n\nIn order to make this point clear, let me briefly summarize some key events in the history of the use of language models (LMs) in ASR.\n- ASR was formulated as a noisy channel model using the Bayes rule P(W|X) = P(X|W)P(W). The two distributions were named an acoustic model and a language model, respectively. The LM was trained separately on source text, and only used for decoding [1].\nWFST-based decoding provided a way to create a incorporate n-gram LMs into the decoding graph for efficient **first-pass decoding** [2].\n- Incorporating larger n-gram LMs was hard (since the decoding graph explodes), so researchers used them instead in **second-pass rescoring** in both offline and on-the-fly settings [3, 4], by subtracting the original LM scores and adding back the larger LM’s scores.\n- The Bayesian formulation still made sense in the era of hybrid HMM-DNN model, even when the acoustic models were discriminatively trained, since the scores could be interpreted as pseudo-likelihoods by subtracting an appropriate prior, and so the same decoding/rescoring framework carried over.\n- In the era of “end-to-end” ASR, the models are trained to directly estimate P(W|X), and so log-linear interpolation with an external LM is not interpretable in the Bayesian sense anymore. Yet, practitioners still do this and call the process “fusion” (e.g. shallow fusion, cold fusion, etc. [5, 6]) — this is analogous to using LMs in the first-pass decoding.\n- Analogous to the second-pass rescoring, we can obtain candidates using beam search on an ASR model, and then re-rank them with an externally trained LM (e.g. as done in the original LAS paper). So far, this “rescoring” step only manipulates the hypotheses, without considering the acoustic information. Let’s call this model (A).\n- More recently, a two-pass E2E ASR model was proposed which has an encoder shared between a streaming RNN-T model and a full-context LAS decoder [7]. The idea was to perform a first-pass decoding with the RNN-T head, and use the decoded hypotheses along with the encoder representations to rescore using the LAS head in a *acoustic-guided rescoring* technique.\n- In [8], the authors further built upon this two-pass model by proposing a **deliberation network**, which is an LAS decoder that attends to both the acoustic representations as well as the first-pass hypotheses, to generate the output. Let’s call this model (B). \n\nThe Generative Error Correction (GER) model (which this paper extends) is analogous to model (A), with the exception that it formulates the rescoring problem as a many-to-one sequence transduction task, as opposed to a simple re-ranking task. \n\nThe proposed model is similar to model (B), as is clear from equation (2). Its goal is to generate a new sequence given acoustic representations and an N-best list. In fact, the “early” and “mid” fusion methods in the paper are, in my opinion, exactly the same as a deliberation model, with the exception that the decoder is frozen instead of being jointly trained. \n\nSituating late fusion is a little more complicated since the fusion only happens at the scoring stage, and it is quite confusing to me exactly how to think about it. This “late fusion” is, in a way, similar to shallow fusion methods in E2E ASR, with the exception that a GER model is used in place of the external LM. The other view is how the authors present it: GER with additional acoustic information.\n\nWhile it is completely acceptable to use new terminology that is more in keeping with broader advances in LLMs, I think it would be beneficial to situate the proposed method in the context of the above ASR+LM strategies to make the paper more widely accessible. In summary, the use of LMs in ASR can be through either first-pass decoding or second-pass rescoring (this paper falls into the latter). Second pass rescoring may be modeled as re-ranking or sequence generation, and methods may or may not use the original acoustic information. This kind of taxonomy and categorization can be used to clarify the contribution of this work. \n\n[1] Jelinek, Frederick. “Continuous speech recognition by statistical methods.” Proceedings of the IEEE 64 (1976): 532-556.\n\n[2] Mohri, Mehryar et al. “Speech Recognition with Weighted Finite-State Transducers.” (2008).\n\n[3] Ljolje, Andrej et al. “Efficient general lattice generation and rescoring.” EUROSPEECH (1999).\n\n[4] Sak, Hasim et al. “On-the-fly lattice rescoring for real-time automatic speech recognition.” Interspeech (2010).\n\n[5] Chorowski, Jan and Navdeep Jaitly. “Towards Better Decoding and Language Model Integration in Sequence to Sequence Models.” Interspeech (2016).\n\n[6] Sriram, Anuroop et al. “Cold Fusion: Training Seq2Seq Models Together with Language Models.” Interspeech (2017).\n\n[7] Sainath, Tara N. et al. “Two-Pass End-to-End Speech Recognition.” ArXiv abs/1908.10992 (2019): n. pag.\n\n### Evaluation problems\n\nIf we accept the above categorization of models, the paper can be viewed in 2 ways: (i) shallow fusion of ASR and GER, and (ii) GER with acoustic information. As such, the proposed method should be compared against the following baselines: (1) end-to-end ASR, (2) shallow fusion of ASR and LLM, and (3) GER-only. The authors have compared against (1) and (3), but not against (2), which may be important to show how using GER improves compared to simply using a LLM with shallow fusion.\n\nThis is important because training the GER itself requires generating N-best hypotheses for the whole training data, which is computationally expensive, and it can only be used in an offline manner, while simple shallow fusion can be done on-the-fly. As such, we expect to see significant WER gains when using the GER module.\n\nAnother concern, which is a common issue when using LLMs, is the following: *How can we ensure that the test data was not seen by LLAMA?* In particular, the authors show in Table 3 that we get a large improvement on CHiME-4 when using the proposed method. CHiME-4 is essentially WSJ read outdoors, and WSJ consists of text from newspapers, which are in the public domain. It would be more useful to conduct experiments on a closed dataset, such as CHiME-5, which is less likely to have been memorized by LLAMA.\n\n### Other minor comments\n\n1. Some sections of the paper use a lot of flowery language which can be avoided. For example, in the last paragraph in Section 1, extraneous terms such as “a pioneering UADF technique”, “adroitly allocates”, and “conspicuously outperforming” can be made concise. In general, authors should stick to reporting rather than embellishing.\n2. In Section 2 (paragraph 1), the authors mention that language models have been used in ASR for two decades, but the oldest citation is only from 2019. I think they should conduct a more thorough literature review in order to perform correct credit attribution.\n3. The use of the terms “early”, “mid”, and “late” fusion in the paper may be confusing for readers familiar with ASR methods such as shallow, cold, and deep fusion. The authors should consider either changing *fusion* to another word, or making the difference explicit in the paper.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "7GoZLS3BVH", "reviewer": "Reviewer_GWCM", "summary": "To integrate acoustic information into the speech recognition output error correction using large language models (LLMs), the authors of this paper compare three fusion methods: early, mid, and late fusions. To further improve late fusion, this paper investigates adding uncertainty and proposes a new approach, Uncertainty-Aware Dynamic Fusion (UADF), introduced to integrate acoustic information, significantly enhancing word error rate (WER) and showing potential in audio-visual speech recognition.", "questions": "The N-best are from WavLM and Whisper in GER-based H2T neurlPS2023. This paper is actually implementing a system combination, which, of course, will bring accuracy improvement. So, this work can be extended to more wider tasks.", "limitations": "", "rating": 10, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 4, "strengths": "Overall, this paper's structure is easy to follow.\nIt reminds me of previous milestone works about RNN decoders: Shallow fusion, Cold fusion, and Deep fusion. And this paper also has the potential to be a milestone. \n\nThe UADF is novel in this paper, and the improvements are guaranteed shown in Tables 1 and 2.", "weaknesses": "ICLR conference is less specialized for speech researchers than ICASSP and INTERSPEECH. A brief and precise background introduction to the speech recognition framework is needed.\n\nFor the three frameworks in Figure 1, it needs to be clarified: where is N-best from? The N-best and the X_tok, X_enc, and X_dec are from different types of models according to the descriptions. It is better to mention these differences in the figures.\n\n\"language models have been widely utilized in ASR tasks over the past two decades,\" maybe there is a better way to express it; some earlier efforts are ignored.\n\nAdditionally, I believe you want to demonstrate that the proposed method is also applicable to audio-visual tasks. This requires a more detailed description; merely using a single paragraph here is insufficient.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)