Large Language Models are Efficient Learners of Noise-Robust Speech Recognition
Authors: Yuchen Hu
CHEN CHEN
Chao-Han Huck Yang
Ruizhe Li
Chao Zhang
Pin-Yu Chen
EngSiong Chng
Venue: ICLR 2024 spotlight (2024)
Keywords: Large language models
automatic speech recognition
generative error correction
noise-robustness
Primary area: representation learning for computer vision, audio, language, and other modalities

Abstract: Recent advances in large language models (LLMs) have promoted generative error correction (GER) for automatic speech recognition (ASR), which leverages the rich linguistic knowledge and powerful reasoning ability of LLMs to improve recognition results. The latest work proposes a GER benchmark with "HyPoradise" dataset to learn the mapping from ASR N-best hypotheses to ground-truth transcription by efficient LLM finetuning, which shows great effectiveness but lacks specificity on noise-robust ASR. In this work, we extend the benchmark to noisy conditions and investigate if we can teach LLMs to perform denoising for GER just like what robust ASR do, where one solution is introducing noise information as a conditioner into LLM. However, directly incorporating noise embeddings from audio encoder could harm the LLM tuning due to cross-modality gap. To this end, we propose to extract a language-space noise embedding from the N-best list to represent the noise conditions of source speech, which can promote the denoising process in GER. Furthermore, in order to enhance its representation ability of audio noise, we design a knowledge distillation (KD) approach via mutual information estimation to distill the real noise information in audio embeddings to our language embedding. Experiments on various latest LLMs demonstrate our approach achieves a new breakthrough with up to 53.9% correction improvement in terms of word error rate while with limited training data. Analysis shows that our language-space noise embedding can well represent the noise conditions of source speech, under which off-the-shelf LLMs show strong ability of language-space denoising.

Published as a conference paper at ICLR 2024
LARGE LANGUAGE MODELS ARE EFFICIENT LEARN- ERS OF ROBUST SPEECH RECOGNITION
Yuchen Hu
Chen Chen 1.*
Chao-Han Huck Yang2.3.
Ruizhe Li²
Chao Zhang
Pun-Yu Chen
Eng Siong Chng'
!Nanyang Technological
University
2Georgia Institute of Technology IIBM Research
NVIDIA Research
*University of Aberdeen
Tsinghua University
ABSTRACT
Recent advances in large language models (LLMs) have promoted generative error correction (GER) for automatic speech recognition (ASR), which leverages the rich linguistic knowledge and powerful reasoning ability of LLMs to improve recogni- tion results. The latest work proposes a GER benchmark with "HyPoradise" dataset learn the mapping from ASR N-best hypotheses to ground-truth transcription by efficient LLM finetuning, which shows great effectiveness but lacks specificity on roise-robust ASR. In this work, we extend the benchmark to noisy conditions and investigate we can teach L.Mss to perform denoising for GER just like what robust ASR do, where one solution is introducing noise information as a conditioner into LLM. However, directly incorporating noise embeddings from audio encoder could harm the LLM tuning due to cross-modality gap. To this end, we propose extract language-space noise embedding from the N-best list to represent the noise conditions of source speech, which can promote the denoising process GER. Furthermore, in order to enhance its representation ability of audio noise, we design a knowledge distillation (KD) approach via mutual information estimation to distill the real noise information in audio embeddings to our language embedding. Experiments on various latest LLMs demonstrate our approach achieves a new breakthrough with up to 53.9% correction improvement terms of word error rate while with limited training data. Analysis shows that our language-space noise embedding can well represent the noise conditions of source speech, under which off-the-sher the-s LLMs show strong ability of language-spar denoising")
INTRODUCTION
Recent advances large language models (LLMs) have attracted a surge of research interest due
to their representation power of language generation (OpenAl 2022 2023 Touvron et al.] (023a))
which achieve a wide range of success on natural language processing (NLP) tasks (Brown et al.
2020: Wei et al. 2022 Ouyang al. 2022). Powered by LLMs, latest works (Chen et al. 2023b Yang et al. 2023a) propose a generative error correction (GER) framework for automatic speech recognition (ASR), along with "HyPoradise" datase that contains abundant pairs of ASR N-best hypotheses and ground-truth transcription. It has shown great performance in learning the mapping from hypotheses transcription by parameter-efffcient LLM finetuning (Hu et al., 2021). which significantly outperforms typical LM rescoring methods (Mikolov et al. 2010). However, their study lacks specificity on noisy ASR scenarios, which are the most common real world (Li et al. 2015).
In this work. we extend the GER benchmark to noisy conditions, as well as propose Robust HyPoradise (RobustHP) dataset with 113K in common noisy scenarios. Similar to the original benchmark, we also observe eπror correction improvement of L.LM finetuning on noisy ASR, but the performance gain in most noisy conditions is still limited (see Table D It indicates that bassed GER is still prone to source audio noise
pairs from various ASR corpus
contribution. y ourer This work is open sourced at: htt.aaaeweaaaaeeeeeaaaaaoooooann
hucky@nvidia.com com

---

Published as a conference paper at ICLR 2024
Response
Response
Response
enjoy listen to music
X
enjoy listening music
X
enjoy listeming to music
Denoise
Denoise
LLM
Adapter
LLM
Adapter
LLM
Adapter
f
N-best Hypotheses
N-best Hypotheses
N-best Hypotheses enjoys listening to music enjoy listen music
enjoys listening to music enjoy listen music
T enjoys listening music enjoy listen music
Audio Noise
Language Noise
* ASR
* ASR
x ASR
KD
Noisy Speech
Noisy Speech
Noisy Speech
(a) GER
(b) GER Audio Denoising
(c) GER Language Denoising (ours)
Figure 1: Overview of (a) GER (Chen et al.
2023b
Yang et al.
2023a (b) GER with audio-space
denoising (Zhang et al. 2023b) (see details in BB. (c) GER with language-space denoising.
(see our case study in Table 5). Luckily, we draw inspiration from the noise-robust ASR community. Their key idea is map noisy speech features to clean space (i.e., denoise) before recognition (Li
et al. 2014). where speech enhancement denoising (Pandey et al. (022)) is one of the most popular approaches. Therefore, we raise a research question for our case: Can we teach LLMs to denoise the N-best hypotheses for GER, just like whul robust ASR and speech enhancement do?
Inspired by recent works on L.M adaptation [Wu et al. 2023a Fathullah et al. 2023 Gao etal. 2023
general solution here is to incorporate audio noise information as a conditioner into LLM finetuning to make it noise-aware, which also similar to the popular conditional diffusion model (Dhariwal & Nichol 20 However, latest works find that directly introducing other modalities (e.g., audio, vi-
sual) into LLM finetuning could harm its stability and performance due to cross- cross-modality gap (Zhang
ct al. 2023b Li et al. 2023b). Our examination in Table a indicates this limitation.
To this end, we propose to extract a noise embedding in language space to represent the noise conditions of source speech, by measuring the diversity of N-best hypotheses list from ASR decoding. The insight behind that, the worse noisy conditions (more challenging noise type or lower SNR), the higher uncertainty of ASR beam search decoding, and thus results in more diverse N-best hypotheses, which has been illustrated in Table 15 and Fig 6 Extracted from the language space of hypotheses instead of audio space, our noise embedding can be well incorporated into LLM tuning improve GER, which can be viewed as a novel language-spat denoising process. Furthermore, in order to enhance its representation ability of audio noise, design a knowledge distillation (KD) approach via mutual information estimation (Belghazi et al.) 2018) to distill the real noise information in audio embeddings to our extracted language embedding. As a result, it presents stronger noise representativeness (see Fig. (b)) and enhances the denoising performance, Various latest L.MMs (e.g.,
LLaMA-2 (Tuvrron et al. 2023b), LLaMA (Touvron et al., (023a) and Falcon (Penedo et al. 2023) utilized to verify the effectiveness of our approach, and the comprehensive experimental results demonstrate that our model improves the GER performance with up to 53.9% word error rate (WER) reduction on RobustHP test sets while with limited training data.
Our contribution can summarized as follows:
We extend the latest ASR generative error correction benchmark to noise-robust ASR, where a Robust HyPoradise (RobustHP) dataset with 113K prppthesss-transccippe pairs is collected from various ASR corpus common noisy conditions.
We propose RobustGER, a noise-aware generative error correction approach based on LLMs to map N-best hypotheses to true transcription, where an extracted language-space noise embedding with audio distillation is utilized to teach LLMS to perform denoising.
Experiments on various latest LLMs show the proposed approach achieves a new break- through on RobustHP with up 53.9% GER improvement in terms of word error rate
2

---

Published as a conference paper at ICLR 2024
(WER). Analysis verifies the effectiveness of our proposed language-spar space embedding to represent audio noise, under which LLMs show strong ability of language- space denoising.
RELATED WORK
Large Language Models and Parameter-efficient Finetuning. There is recently surge of research interests in Transformer based LLMs, such as ChatGPT (OpenAl) 2022), GPT-4 (OpenAl, (023) and LLaMA (Touvron al., 2023a). Benefiting from giant model size and abundant training data, L.LMs can understand the linguistic structures and semantic meanings behind text, which shows remarkable performance on a wide range of NLP tasks (Brown et al. 2020 Wei et al., 2022) Ouyang al. 2022). To adapt LLMs to downstream tasks, many recent works investigate parameter-efficien LLM linetuning (Hu et al. 202T) considering its huge model size. In order to further exploit the potential of LLMs on multimodal tasks, more recent works investigate to incorporate other modalities into
LLM tuning (Wu al. 2023a Fathullah et al.) 2023 Li et al. 2023a Chen et al., 2023c Zhang et al. 2023a b Gao et al. 2023 Wang et al. 2023 Radhakrishnan et al. 2023). However, the latest
works find that directly introducing other modalities into LLMs could harm the finetuning stability
and performance due to the heterogeneous cross coos-modalltyy gap (Zhang al.) 2023b Li al. (023b).
Therefore, this work proposes to extract a language embedding from the N-best list to represent audio noise, which works well in teaching LLMs perform denoising.
LM Rescoring and ASR Generative Error Correction. LM rescoring has been widely used in ASR decoding improve the linguistic acceptability of recognition results, which achieves stable gains of ASR performance (Arisoy et al. 2015) Shin et al.] (2019] Mikolov et al.) 2010 Yang et al.) Yu al. 2023). Typically, an external LM is deployed rescore the N-best hypotheses list from ASR beam search decoding to rerank out the 1-best candidature. Furthermore, to make full use of all candidatures, recent works use the entire N-best list for error correction (Leng et al. 202 Ma et al.) 2023: Hu et al. 2020: 2023a Guo et al.) 2019: Hu et al. 2022a;; Chen et al. (023a). which outperforms rescoring methods. Powered by LLMss, the latest works propose generative error correction (GER) benchmark (Chen et al.) 2023b) to directly predict the ground- truth transcription from ASR N-best hypotheses. To enable the learning
2021:
woatheesee-t-ttansriiiie also propose a HyPoradise dataset with 316K eepotheses-transcripee GER benchmark to the most common noisy ASR scenarios with a new Robust HyPoradise dataset.
mapping, they pairs. This work extends the
Noise-robust ASR. Neural ASR has achieved human-level performance in recent years but its noise-robustness in the real world remains a challenge (Krishna et al. 2019). Recent noise-robust ASR methods make some progress by mapping noisy speech features to clean space (i.e., denoise) before recognition (Li et al. 2014 Pandey et al. 2021). For instance, speech enhancement serves as denoising front-end (Fu al. 2019 Hu et al.] 2023b) improve speech quality for ASR (Hu et al.
2022b: 2023d c), domain adversarial training aims to learn noise-invariant speech features (Prasad al.] 2021), and the recently popular ASR foundation model proposes to use web-scale data and various preprocessing steps for denoising (Radford et al., (202)). Inspired by them, this work investigates teach LLMs to denoise the N-best hypotheses language space for GER.
BENCHMARK AND DATASET
3.1 GENERATIVE ERROR CORRECTION BENCHMARK
We extend the original generative error correction benchmark (Chen et al., 2023b) to noise-robust ASR. Given input noisy speech Xn, the pre- pre-trained ASR model first transcribes it into -esest hypotheses Vv (Yi, ...., Y by beam search decoding. The goal of GER is to learn a pooothesee-to-transeiiie (H2T) mapping Mizr that predicts the transcription Y based on N-best hypotheses list V
Y = TMM (V/v,,
(I)
Given the ground-truth transcription Y*, we can finetune the LLM to learn MazT in an ruto-segressive manner, where the cross- cros-entropp loss PueT is formulated as:
T
CHET
log Po 1o ". UN),
(2)
1
where of is the 1-th token of Y and denotes the learnable parameters in L.LM (i.e., adapter).
3

---

Published as a conference paper at ICLR 2024
Response
Frazen
Concatenate
Utterance- levell Noise Emb.
enjoy listening to music
Finetune
Denoise
Language-space Noise Emb.
enjoys listening to music
enjoy listen music
LLM
Adapter
Jjoin listening to musie I enjoy listened mystic
Noisy Speech
Tuner
join listening music
Token-level Noise Emb.
Instruction
+
N-best Hypotheses
Please predict the transcription from N- best hypotheses:
enjoys listening music
1 enjoy listen music
Search
Beam
1 0
enjoy listen
enjoys listening
music
music
1 join listening to music enjoy listened mystic am join listening music
ASR
join listening
Ø
enjoy listened 0
mystie
1 am
join
listening Ø
music
Figure 2: Left: The RobustGER framework that leverages efficient LLM finetuning learn mapping from ASR N-best hypotheses to ground-truth transcription, where we propose a language-space noise embedding with audio distillation denoise GER process. Right: The extraction of language-space noise embedding from N-best hypotheses by measuring its diversity, where we calculate the utterance- and token-level embedding differences between each pair of hypotheses in the N-best list. The details of embedding extraction are illustrated in $4.2 and Eq. (
3.2 ROBUST HYPAAIISE DATASET
Correspondingly, we develop Robust HyPoradise dataset by collecting hypotheses-transcrippio (HT) pairs from common noisy ASR corpus. including CHiME-4 (Vincent et al. 2016). VoiceBank- DEMAND /Valentini-Botinnn
et al., 2016). NOIZEUS (Hu & Loizou 2006). LibriSpeech- FreeSound (Prasad et al. 2021) and RATS (Graff et al. 2014), with details provided in $A We employ Whisper Large-V2 V2 (Radford et al.) 2023). the -fate-of-the-art ASR foundation model to transcribe the noisy speech into N-best hypotheses (N is set to 5). As result, we collect 113K HT pairs in total from various noise domains, and the dataset statistics are presented Table
4 METHOD
In this section, we present our noise-aware generative eтror correction (RobustGER) approach. We first describe the overall framework ($4.1). and then we introduce the extraction of language-space noise embedding from N-best hypotheses (4.2). followed by audio noise distillation ( (.3) at last.
4.1 OVERALL FRAMEWORK
The left part of Fig. 2 presents the overall framework of RobustGER. First, the noisy speech X,, is sent into a pre-trained ASR model to generate N-best hypotheses V/v (Y,, Y,,... YN), where N = 5. Following that, we propose to extract a language-space noise embedding FLx from the N-best list VN to represent the noise conditions of source speech X,, As depicted in the right part of Fig. 2 such noise embedding measures the diversity of N-best hypotheses on both utterance and token levels, which perceives the noise information input speech.
Furthermore, to enhance its noise representation ability, we design KD approach to distill the real noise information in source speech X,, to the extracted language-space noise embedding ELN- Specifically, we employ the audio embedding EASR (X,, from ASR encoder for distillation.
Finally, we add an instruction onto the N-best hypotheses and sent them into LLM to predict the true transcription (i.e., GER), with the language embedding incorporated for denoising. Specifically, add a minus sign before the noise embedding FLN to indicate "denoise". Such minus embedding is then to teach LLM do language-space denoising. Therefore, Eq. should be re-written as:
Y MHET( -ELn),
4

---

Published as a conference paper at ICLR 2024
The Mbzr denotes H2T mapping by efficient LLM finetuning, where we follow the adapter tuning from previous works (Zhang et al. [203bb Yang et al. 2023b). We also borrow their idea of input- level prompting to incorporate our language noise embedding into LLM tuning, and the details are presented in BB.1 Similar to Eq. (2). we follow the original GER benchmark for optimization.
4.2 LANGUAGE-SPACE NOISE EMBEDDING
As directly incorporating audio-space noise embedding into LLM finetuning could harm its stability and performance (Zhang et al. 2023b] Gao et al. 2023). we propose an alternative to extract language- space noise embedding from N-best hypotheses to represent the noise conditions of source speech. The key idea is to perceive the audio noise from the diversity of N-best hypotheses, i.e., the worse noisy conditions (more challenging noise type or lower SNR), the higher uncertainty of ASR beam search decoding, and thus results in more diverse N-best hypotheses (see Table 13 and Fig 6
As illustrated the right part of Fig. 2 we extract the noise embedding on both utterance and token levels to capture rich diversity information: 1) Utterance-level: examine the diversity inside N-best list in terms of the entire utterance's 's semantic meaning, which indicates the affect of audio noise on the global semantics of hypotheses; 2) Token-level: examine the distribution of N-best hypothesis in terms of all the tokens inside, which similar to edit distance and thus directly corresponds to the WER metric. These two embeddings are finally combined to form the resulted noise embedding. i.c., ELN = [ET LiN J. Specifically, we employ sentence- BERT (SBERT) [Reimers & Gurevych 2019) to obtain the embeddings from raw text, which contains rich language-space semantic information.
4.2.1 UTTERANCE-LEVER NOISE EMBEDDING
Given N-best hypotheses VN = (Y1, Y2. YN), we first obtain their sentence embeddings by SBERT encoder Exeert and then calculate their diversity as:
Ex = Concat (Esbent()) Sbenn (Y "1 E R N-(N-11 Deee
where Dabent denotes the embedding size of SBERT extractor. In short, it concatenates all the sentence embedding differences Eshen (Y;) - Esher (Y,) where i > j. resulting in an utterance-level noise embedding ETN E hypotheses list, which thus presents lower confidence and worse transcription quality, i.c., more language noise. Therefore, (q. serves as a measurement of the audio noise in language space. The worse noisy speech would lead larger ASR decoding uncertainty and thus more diverse N-best hypotheses, so that Eq. 4 can capture larger diversity embedding.
The key idea is, Y, ranks lower than Y, in the N-best
4.2.2 TOKEN-LEVEL NOISE EMBEDDING
Apart from utterance-level embedding, we also propose to extract token-level noise embedding that directly corresponds to the WER metric of ASR task. As shown in the bottom- -right part of Fig. 2 similar to the calculation edit distance, we first forced-align the N-best hypotheses to the same length with zero padding (i.e., "Ø"). The aligned N-hest hypotheses yoo' (Yaali Yols ..... Y&*) clearly illustrates the token difference between different candidatures, where each utterance contains T tokens that comes from ASR vocabulary V plus padding Ø:
Y = [3 als yor y E VUØ
(5)
Inspired by edit distance, we design an "edit embedding" to capture the token-level difference between two hypotheses, which directly corresponds their gap in final WER performance. Then, similar to Eq. we calculate the token-level noise embedding by summing up the edit embedding between different pairs of hypotheses in the N-best list:
Etsh = Concat Eedil
Eesu(Y (Y,ali, y ) }=,==1i>ー}} E R
M(N-1) x D
(6)
Eodn (Y,al, y (ESbe
[Esben (v Exbort ( ali)]
Note that we employ SBERT again to extract the token embedding, as it can produce informative embeddings for both utterances and tokens (Reimers & Gurevych] 2019).
4.3 AUDIO NOISE DISTILLATION
5

---

Published as a conference paper at ICLR 2024
Algorithm 1 Audio noisc distillation via mutual information neural estimation (MINE).
Require: LLM MHzT with adapter G.. MINE statisties network v of parameters 0, language embedding tuner of parameters N-hest hypotheses Vv. Parallel noisy speech Xr, and clean speech data Ne. Batch size and the total number iterations M. Hyper-parameter weight X.
1: for m = 1 to M do
2: Draw B N-best hypotheses samples from RobustHP dataset: (>x .. ,..,)
Draw corresponding noisy and clean speech samples: ((x,), X!"). (x. xx!)) ) 1.... (x. x)
4: Extract language space noise embedding from N-best list using Eq. 6 {EiN E EN ((B))
5:
Calculate Eq. 8 I Σ the (F (Xx - log( #/8 > -bb eaa (E Easu(X? 1)
6:
Calculate go V (I) and update 0 by gradient ascent: 0 + + go:
7:
Calculate GER cost function Luzr using Eq. 1
(2. with (E incorporated for denoising;
8: Re-calculate the first term of Eq. Σ1
B 2am1 Thw
(T (bit)) EAS (x(")));
)):
9:
Calculate f0,w = V Vo,w (CHzT XI, and update w, w by gradient descent: e <- v go. w +- w gw:
10:
end for
After extracting the language-space noise em- bedding from N-best hypotheses, we further pro- pose audio noise distillation approach via mu- tual information estimation to enhance its noise representation ability. Mutual information (MI) a measure of dependence between random variables based the Shannon entropy, which is equivalent to the Kullback-Leeilee (KI.-) di- vergence between the joint distribution and the product of the marginal distribution of random variables. Given
0
MI
MINE
Tuner
Clean Speech
Noise Emb.
Noisy Speech
Figure 3: Audio noise distillation by mutual infor- mation neural estimation (MINE). The trainable tuner is designed to maximize the MI between extracted noise embedding and the noisy speech.
random variables X and Z, their MI can be calculated by:
I(X: (Px || PxP2).
(7)
where D (IP Q) denotes KL-divergence. However, it intractable to directly calculate MI based on Eq. 7 we leverage an estimation method called mutual information neural estimation (MINE) from previous work (Belghazi et al. 2018). MINE employs a statisties network vio X R
parameterized by 0 E θ to estimate a neural information measure:
(X; Z) sup Exxz log(Er Px 11 AE
8
In practice, we employ the extracted language-space space noise embedding FLN and noisy audio embed- ding EASR(X (X,)) as the joint distribution, while using FL and clean audio embedding EASR (X.) as the marginal distribution, as the noise information only exists in noisy speech.
Algorithm describes how MINE is utilized for audio noise distillation, which includes two stages. First, the statistics network the trained to learn accurate MI estimation using both the positive and negative sample pairs introduced above. Second, a learnable tuner Tw, is introduced to modulate the language embedding ÉLN capture more real noise information, by maximizing the MI between and the noisy audio embeddings. More details about the MINE-based audio noise distillation are in B.2 In addition, the LLM adapter is also updated in second stage learn H2T mapping for GER.
EXPERIMENTS
5.1 SETUP
We conduct experiments on the proposed RobustHP dataset, which is detailed in A To verify the general effectiveness of our approach, we utilize various latest LLMs for evaluation, including LLaMA-2-7b//l (Touvron al. 2023b), LLaMA-7b (Touvron al.) 2023a) and Falcon-7b (Penedo) et al. 2023). We follow the LLM Lddpter in previous work (Zhang et al. (023b) for both LLM finetuning and noise embedding incorporation. Details of model and experiment setups in C

---

Published as a conference paper at ICLR 2024
Table 1: WER (%) results of RobustGER with LLaMA-2-7b finetuning. "I.M denotes LM
rescoring. "+ Audio Denoising" denotes introducing audio embedding to denoise GER. Onb and Ocp respectively denote the N-best oracle and compositional oracle that are defined in $5.1 percentage denotes relative WER reduction over ASR baseline, i.e., GER improvement.
The subscript
RobustGER (ours)
Oracle
Test Set
Baseline
LMnk
GER
Audio Denoising
Ocp
test-real
12.6
12.2
6.5 4%
6.4_.o 19.2%
5.6-tn.6%
5.6
10.5
3.0
lest sima
15.4
14.5
9.2
9.0-41.69 -1.6%
9.0
8.2
12.9
5.0
CHIME-4
dev-real
10.6
10.3
62.8%
63.8%
4.1 01.3%
9.1
12.4
6.6 300-00.888
5.8 50-2%
3.3
avg.
12.8
12.2
6.9 46.1%
6.7 47.7%
5.9
50.0%
10.8
3.1
baby-cry
8.0
7.8
7.0
7.0-12.0%6
6.9 63.-13..9
6.0 26.0%
4.5
3.0
VB-DEMAND
helicopter
8.4
8.1
7.4_11.99
7.3_ 13.1%
6.9 6.9-
17.9%
4.8
3.2
crowd-pprry
22.6
22.3
21.4 5.39
21.0 21.0.7.11
19.2 16.0%
16.5
11.5
avg.
13.0
12.7
11.9 8..55
11.7 10.00%
10.7 11.77%
8.6
5.9
babble
16.5
16.7
16.5-0005
16,1 16,1-22455
14.5 12.11
9.5
5.8
car
17.4
16.8
15.3-12.1% 12.1%
15.3
15.2-12.455
14.9-14.00
9.9
7.9
12.0
10.3
14.2%
10.3-14.2%
9.5 20.8%
train
15.3
15.2
14.9
15.0_2.0R
14.9 14.9 2..0%
10.3
7.9
NOIZEUS
street
17.4
17.2
17.4
17.1
16.1 7.5%
12.4
9.9
airport
11.2
11.0
10.7 4.5%
10.5 6,39%
9.5 16.2%
7.9
4.5
exhibition
13.2
13.2
12.8 3.0%6
12.4-6.15%
9.5 28.0%
5.8
restaurant
13.2
13.0
12.4_6.15
12.5_5.955
12.0
8.7
6.2
avg.
14.5
14.3
13.5 4.8%
13.6
12.6 13.1%%
9.2
6.6
metro
9.9
9.8
9.5 4.0%
9.4 6.199
8.9 10.1%
7.9
4.9
4.0
3.7 7.3%
12.89
3.1 22.6%
1.8
traffic
8.3
8.2
8.0 3.00
7.8-6.0%%
7.5-9.6%
6.8
4.5
LS-PrreeSound
cafe
9.8
9.5
8.1 87.3%
8.1_17.3%
7-5_
7.1
4.6
babble
32.0
31.8
31.3-2.25
31.6.138
31.1-2.899
28.7
19.3
ac/vacuso
12.4
12.5
12.3_0.8%
12.1-2.45
11.4
10.2
6.2
12.7
12.6
12.2.
12.1 4.7%
11.6
10.6
6.9
RAIS
teat
45.7
45.6
45.2
41.8 2,0%
43.2
38.8
23.6
Table 2: WER (%) results RobustGER on different SNR-level testing conditions. The test sets are from LS-FreeSound dataset, with five SNR levels noise types. More results are in Table
RobustGER (ours)
Oracle
Noise Type
SNR (dB)
Bascline
LM
GER
Audio Denoising
O,b
Oep
0
9.9
9.8
9.5 9.5%
9.4
8.9
10.1%
7.9
4.9
5
7.2
7.0
6.7 6.7_6.9%
6.4 11.1%
5.5 28.6%
5.5
3.2
10
4.8
4.6
4.2-12.5%
4.3 10.4%
4.0 16.7%
3.9
2.3
Metro
15
3.9
3.5
3.2_17.9%
3.2.17.9%
3.0 3.0-28.1%
3.1
1.7
20
3.3
3.1
2.7-18.29
2.6
2.3 30.3%
2.6
1.3
avg.
5.8
5.6
5.3
5.2 10.3%
4.7 19.0%
4.6
2.7
0
12.4
12.5
12.3 0.88%
12.1 12.15
11.4
88.4%
10.2
6.2
5
7.4
7.0
6.5 12.21
6.3
5.8
5.5
3.1
6.6
6.2
5.5 16.7%
5.6_15.2%
5.5 5.5-16.7%
4.5
2.6
AC/Vacuum
4.4
3.7 10.99
3.7 15.99%
3.6 16.2%
3.3
1.8
3.3 13.2%
3.2 15.8%
2.9 23.7%
2.8
1.4
avg.
8.7%
6.2_1 10.1%
5.8 15.9%
5.3
3.0
Clean
0C
3.0
2.8
16.71
2.4. 20.055
2.1
30.0%
2.5
1.4
We report experimental results in terms of word error rate (WER) and relative GER improvement. We also report two oracle WERs for reference: 1) N-best oracle ons: WER the "best candidate" in N-best list, and 2) compositional oracle Ocpi best achievable WER using all the tokens in N-best hypotheses. They indicate the upper-bounds rerank and GER (using occurred tokens), respectively.
5.2 PERFORMANCE OF ROBUSTGER
Table presents the experiment results on LLaMA-2-76 2b,, and more LLMs are evaluated in $D.I First, observe minor gains of performance brought typical LM rescoring the Whisper ASR baseline. Compared to LM rescoring, GER achieves promising progress by leveraging LLMs to generate transcription, while its performance gains in most noisy conditions except CHiME-4 are still

---

Published as a conference paper at ICLR 2024
Table 3: Ablation study of the language-space noise embedding in terms of utterance and token levels. More studies are presented in Table 13 and Table [4
Test Set
Baseline
GER
+ Audio Denoising
Language Denoising Tok. -lvvel
-leell
Both
test-real
12.6
6.5 48.4%
6.4-49
6.4 49.2%
6.1 51.8%
5.9 53.2%
test-simae
15.4
9.2 9.2-10.35 10.3%
9.0 9.0_11..60
9.1 9.1-5
8.9-12229
8.6 8.6_41.2%
CHIME-4
dev-real
10.6
5.0 62.85
4.9 65.85%
55.7%
4.4 58.5%
4.4 58.8%
dev-sima
12.4
6.8 46.2%
6.6 40.8%
6.4 48.4%
6.3 40.2%
6.1 50.88%
avg.
12.8
16.16
6.7-17.7%
6.7
6.4_5000
6.3 50.8%
baby-ery
8.0
7.0 12.0%
6.9 10.8%
6.7 16.3%
6.6 17.30
6.4 20.0%
VB-DEMAND
helicopter
8.4
7.4_111.9
1..111..1
7.3 13.1%
7.1-15.3%
15.5%
crowd party
22.6
21.4 _1.4_
21.0_2.1%
20.8 20.8...05
20.3 -10.5%
19.9-11.00
avg.
13.0
11.9 8.0%
11.7 10.0%
11.6 10.8%
11.3 10.1%
11.1 -1.0%%
03
T972
20
-12
cils
ritra
(a) Language Noise Emb.
(b) Language Noise Emb. KD
(c) Audio Noise Emb.
Figure 4: t-SNE visualizations of (a) language-space noise embedding, (b) language embedding with audio distillation, (c) audio noise embeddings. Cluster distances are in Table 17 Details are in C222
limited. Introducing audio denoising further improves the result but suffers from the cross-modality gap. In comparison, with the proposed language- space denoising approach, our RobustGER achieves significant gains of performance in various noise conditions, with up to 53.9% GER improvement in terms of WER metric, where some results even surpass the reranking upper-hound.
rreootss the performance of RobustGER under different SNRs, where we can observe consistent WER improvements on various noise levels. In addition, RobustGER also shows great effectiveness on clean test data with 30.0% relative WER reduction, which verifies its excellent generality.
5.3 ABLATION STUDY
Table 3 illustrates the ablation study on the extraction of language-space noise embedding, which includes both utterance- and token-level information introduced in 44.2 We can observe that utterance-level embedding only yields minor improvements over vanilla GER, indicating that the global semantics diversity of N-best hypotheses is not fine-grained enough for error correction. On the other hand, token-level information plays a significant role language- denoising for GER, as it directly corresponds to the word error rate metric. Combining both performs the best by leveraging richer information to measure N-best list diversity.
In addition, also conduct ablation studies the language embedding extractor (i.e., SBERT vs. FastText (Grave et al.) 2018), LLaMA embedding.) in $D.3 as well as the audio noise distillation techniques (i.e., MINE vs. contrastive learning, teacher-student learning) in $D.4 All of them verify the effectiveness of our specific designs in RobustGER system.
5.4 ANALYSIS
Visualizations Noise Embeddings. Fig. visualizes the language-space noise embedding to show its representativeeness of audio noise. First, we can observe from Fig. (a) that our extracted language embedding from the N-best list can well represent some noise types (i.e., "ac", "babble", "cafe"), while the others intertwined with clean embeddings. indicating less optimal noise representations. For reference, the audio noise embeddings in Fig. (c) distinguish well between different conditions. Therefore, design KD approach to distill the real noise information in audio embedding to
8

---

Published as a conference paper at ICLR 2024
Table 4: Data efficiency of RobustGER on CHiME- test sets. The "Ik", "2k", etc., denote the number of HT pairs in training data, and "Training Hours" denote its duration of source speech data.
RobustGER 5k
Test Set
Baseline
GER
1k
2k
8k
9.6k (full)
Training Hours
17.5
1.7
3.5
9.2
14.5
17.5
test-real
12.6
6.5 48.4%
9.3 9.3..26.20 2%
7.0
-4444..%%%
5.9 -3.2%
5.7 -5.8%
5.6 55.6%%
test-simu
15.4
9.2 20.35%
11.4
-26.0%
9.5 38 3%
8.8 42.95%
8.4 4.11446.6%
8.2
46.
dev-real
10.6
5.0 62.8%
7.2-32.10 32.1%
5.2
50.9%
4.4 1.4_..80 68.3%
4.1 -G
4.1 -1.8%
dev-simu
12.4
6.8 45.22%
8.9 22 2%
7.1
42.42..77 7%
6.2 00.055
5.9 52.4%
5.8 68.2%
avg.
12.8
6.9 46.1%
9.2.228.19 28.1%
7.2
8%
6.3 60.895
6.0 53.1%
5.9 58.9%
Table 5: Case study of RobustGER. We also implement an in-context learning baseline by ChatGPT
for comparison (details are in $C.22
The test sample is selected from the CHiME-4 dev-real set.
Method
Utterance
WER (%)
the four other utility company owners will also have to take write ups the four other utility company owners will also have to take write ups the four other utility company owners will also have to take write ups the four other utility company owners will also have to take ride outs the four other utility company owners will also have to take ride outs
7.7
7.7
N-best Candidates
7.7
15.4
15.4
In-context Learning
the four other utility company owners will also have to take write-ups
15.4
GER
the four other utility company owners will also have to take write ups
7.7
RobustGER
the four other utility company owners will also have to take write offss
0.0
Ground Truth
the four other utility company owners will also have to take write offs
language embedding. Fig. (b) shows it disentangles the embeddings from different noise conditions and improves their noise representativeness, which leads better WER results as shown in Table [4]
Data Efficiency. As shown in Table 4 we further discuss the data efficiency of RobustGER using the CHiME-4 dataset, whose training set contains 9.6k HT pairs decoded from 17.5-hour speech data. As we gradually reduce the training data, we find that using around half-size data (i.e., 5k pairs) can still maintain the WER performance, i.e. 6.3% vs. 5.9%. When decreases to 2k pairs, RobustGER is still comparable to GER, i.e., 7.2% vs. 6.9%. This experimental evidence verifies the data efficiency of RobustGER, which may originate from the attribute of parameter-efficient LLM finetuning.
Case Study. Table 5 illustrates case study to demonstrate the effectiveness of RobusIGER. There are two errors in N-best hypotheses, i.e., "write ups" (in 1-best) and "ride outs". where the ground truth is "write offs"" Both ChatGPT-based in-context learning and LLaMA-based GER fail to correct this error, because the words "write ups" and "write offs" sound quite similar under noisy scenarios. In comparison, our RobustGER can correct this error by language-ppace denoising, where our proposed noise-representative embedding teaches L.MMS to remove the language noise in N-best hypotheses that is caused by audio noise. More importantly, the semantic meanings of "write ups" and "write off's are opposite, which highlights the significance of successful error correction by our RobustGER.
6 CONCLUSION
In this paper, we first extend the latest ASR generative error correction benchmark to the most com- mon noisy scenarios in real world, with a proposed RobustHP dataset containing 113K hypotheses- transcription pairs decoded from various noisy ASR corpus. Based on that, we propose RobustGER, a noise-aware generative error correction approach based on L.IMS to predict the ground-truth tran- scription based on N-best hypotheses, where an extracted language- distillation leveraged teach L.MS to perform denoising in language space. Extensive experiments on various latest LLMs show that our approach achieves new breakthrough on RobustHP dataset with up to 53.9% error correction improvement in terms of WER while with limited training data. Further analysis verifies the effectiveness of our proposed fanguage-space embedding represent audio noise, under which off-the-sheli LLMs show strong ability of language space denoising.
noise embedding with audio

---

Published as a conference paper at ICLR 2024
REFERENCES
Ebru Arisoy, Abhinav Sethy, Bhuvana Ramabhadran, and Stanley Chen. Bidirectional recurrent neural network language models for automatic speech recognition. In 2015 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 5421-5425. IEEE, 2015.
Mohamed Ishmael Belghazi, Aristide Baratin, Sai Rajeshwar, Sherjil Ozair, Yoshua Bengio, Aaron Courville, and Devon Hjelm. Mutual information neural estimation, In International conference on machine learning, pp. 531-540. PMLR, 2018.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems, 33:1 33:1877-1901, 2020.
Chen Chen, Yuchen Hu, Chao-Han Huck Yang, Hexin Liu, Sabato Marco Siniscalchi, and Eng Siong Chng. Generative error correction for coit-swit speech recognition using large language models. arXiv preprint arXiv:2310.33013, 2023a.
Chen Chen, Yuchen Hu, Chao-Har Huck Yang, Sabato Marco Siniscalchi, Pin- Yu Chen, and Ensiong Chng. Hyporadise: An open baseline for generative speech recognition with large language models. In Thirty-sevnnth Conference on Neural Information Processing Systems Datasets and Benchmarks Track, 2023b.
Feilong Chen. Minglun Han, Haozhi Zhao, Qingyang Zhang, Jing Shi, Shuang Xu, and Bo Xu. X-llm: Bootstrapping advanced large language models by treating multi-modalities as foreign languages. arXiv preprint arXiv:2305.nm (tV.:2305004460 (60 2023c.
Prafulla Dhariwal and Alexander Nichol. Diffusion models beat gans image synthesis. Advances in neural information processing systems, 34:8780-8788 2021.
Qingxiu Dong, Lei Li, Damai Dai, Ce Zheng, Zhiyong Wu, Baobao Chang, Xu Sun, Jingjing Xu, and Zhifang Sui. A survey for in-context learning, arXiv preprint urXiv:2301.00253 2022.
Yassir Fathullah, Chunyang Wu, Egor Lakomkin, Junteng Jia, Yuan Shangguan, Ke Li, Jinxi Guo, Wenhan Xiong, Jay Mahadeokar, Ozlem Kalinli, et al. Prompting large language models with speech recognition abilities. arXiv preprint arXiv:2307.11773
2023.
Philip Feldman, James R Foulds, and Shimei Pan. Trapping Ilm hallucinations using tagged context prompts. arXiv preprint a3X6 06085, 2023.
Frederic Font, Gerard Roma, and Xavier Serra. Freesound technical demo. Proceedings of the 21st ACM international conference on Multimedia, pp. 41 1-412, 2013.
Szu-Wei Fu, Chien-Feng Liao, Yu Tsao, and Shou-De Lin. Metricgan: Generative adversarial networks based black-box metric scores optimization for speech enhancement. In International Conference on Machine Learning, pp. 2031-2041. PMLR, 2019.
Peng Gao, Jiaming Han, Renrui Zhang, Ziyi Lin, Shijie Geng, Aojun Zhou, Wei Zhang, Pan Lu, Conghui He, Xiangyu Yue, et al. adapter v2: Parfmeter-efficient visual instruction model. arXiv preprint w.xiiv2304.10111 2023.
Yuan Gong, Sameer Khurana, Leonid Karlinsky, and James Glass. Whisper-at: Noise-robust automatic speech recognizers are also strong general audio event taggers. In Proc. Interspeech, 2023a.
Yuan Gong, Alexander Liu, Hongyin Luo, Leonid Karlinsky, and James Glass. Joint audio and speech understanding. In IEEE Proc. ASRU, 2023b.
David Graff, Kevin Walker, Stephanie M Strassel, Xiaoyi Ma, Karen Jones, and Ann Sawyer. The rats collection: Supporting hit research with degraded audio data. In IRECC pp. 1970-1977. Citeseer, 2014.
Edouard Grave, Piotr Bojanowski, Prakhar Gupta, Armand Joulin, and Tomas Mikolov. Learning word vectors for 157 languages. In Proceedings of the International Conference on Language Resources and Evaluation (LREC 2018), 2018.
10

---

Published as a conference paper at ICLR 2024
Jinxi Guo, Tara N Sainath, and Ron J Weiss. A spelling correction model for end-to-end speech recognition. In Proc. ICASSP, pp. 5651-5651 IEEE, 2019.
Hans-Günte Hirsch and David Pearce. The aurora experimental framework for the performance evaluation of speech recognition systems under noisy conditions. In ASR2000-Automaate speech recognition: challenges for the new Millenium ISCA tutorial and research workshop (ITRW), 2000.
Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu Yuanzhi Li, Shean Wang, Lu Wang. and Weizhu Chen. Lora: Low-rank adaptation of large language models. arXiv preprini arXiv: 106.06856 2021.
Ke Hu, Tara N Sainath, Ruoming Pang, and Rohit Prabhavalkar. Deliberation model based two-pass end-to-end speech recognition. In ICASSP 2020-2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp- 7799-7803. IEFE, 2020.
Ke Hu, Tara N Sainath, Yanzhang He, Rohit Prabhavalkar, Trevor Strohman, Sepand Mavandadi, and Weiran Wang. Improving deliheration by text-only and semi-supervised training. arXiv preprint arXiv:2206. 14716, 2022a.
Ke Hu, Bo Li, and Tara N Sainath. Scaling up deliberation for multilingual asr. 2022 IEEE Spoken Language Technology Workshop (SLT), pp. 771-776. IEEE, 2023a.
Yi Hu and Philipos C Loizou. Subjective comparison of speech enhancement algorithms. In 2006 IEEE International Conference on Acoustics Speech and Signal Processing Proceedings, volume 1, pp. I-I. IEEE, 2006.
Yuchen Hu, Nana Hou, Chen Chen, and Eng Siong Chng. Interactive feature fusion for end-to- end noise-robust speech recognition. In ICASSP 2022-2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp.
IEEE, 2022b.
Yuchen Hu, Chen Chen, Ruizhe Li, Qiushi Zhu, and Eng Siong Chng. Noise-aware speech enhance- ment using diffusion probabilistic model. arXiv preprint arXiv:2307.08802, 2023b.
Yuchen Hu, Chen Chen, Qiushi Zhu, and Eng Siong Chng. Wav2code: Restore clean speech representations via codebook lookup for noise-robust asr. IEEE/ACM Transactions on Audio, Speech, and Language Processing, 2023c.
Yuchen Hu, Nana Hou, Chen Chen, and Eng Siong Chng. Dual-Path Style Learning for End-to-End Noise-Robus. Speech Recognition. In Proc. INTERSPEECH 2023, pp. 2918 2922,, 2023d.
Yuchen Hu, Ruizhe Li, Chen Chen, Chengwei Qin, Qiu-Shi Zhu, and Eng Siong Chng. Hearing lips noise: Universal diseme-phoneme mapping and transfer for robust audio-visual speech recognition. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume I: Long Papers), pp. 15213-15232, 2023e.
Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv: 14226694,, 2014.
Gautam Krishna, Co Tran, Jianguo Yu, and Ahmed H Tewfik. Speech recognition with no speech or with noisy speech. In ICASSP 2019-2019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 1090-1094, IEEE, 2019.
Yichong Leng, Xu Tan, Rui Wang, Linchen Zhu, Jin Xu, Wenjie Liu, Linquan Liu, Tao Qin, Xiang- Yang Li. Edward Lin, al. Fastcorrect 2: Fast error correction on multiple candidates for automatic speech recognition. arXiv preprint arXiv:2109. 1et::2109.144200 14420, 2021.
Jinyu Li, Li Deng, Yifan Gong, and Reinhold Haeb- .ubbch. An overview of noise-robust automatic speech recognition. IEEE/ACM Transactions on Audio, Speech, and Language Processing, 22(4): 745-777, 2014.
Jinyu Li, Li Deng, Reinhold Haeb-Umbach and Yifan Gong. Robust automatic speech recognition: bridge practical applications, chapter 1, pp. 1-20. Academic Press, 2015.
11

---

Published as a conference paper at ICLR 2024
Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. Blip-2: Bootstrapping ergguage-imgg pre- training with frozen image encoders and large language models. arXiv preprint arXiv:2301. 125977 2023a.
Yanxi Li, Xinghao Chen, Minjing Dong, Yehui Tang, Yunhe Wang, and Chang Xu. Spatial-channel token distillation for vision mlps. In International Conference on Machine Learning, pp. 12685- 12695. PMLR, 2022.
Yuang Li, Yu Wu, Jinyu Li, and Shujie Liu. Prompting large language models for zero-shot domain adaptation in speech recognition. arXiv preprini arXiv:2306. 20XX::2306./6007, 2023b.
Hsin-X Yi Lin, Huan-Hsin Tseng, Xugang Lu, and Tsao. Unsupervised noise adaptive speech enhancement by discriminnator-constaaiii optimal transport. Advances in Neural Information Processing Systems, 34: 1199995-19944 2021.
Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. In International Confer- ence on Learning Representations, 2018.
Rao Ma, Mark JF Gales, Kate Knill, and Mengjie Qian. N-best 15: Robust ast error correction using multiple input hypotheses and constrained decoding space. arXiv preprint arXiv:2303.0044 00456, 2023.
Tomas Mikolov, Martin Karafíát, Lukas Burget, Jan Cernocký, and Sanjeev Khudanpur. Recurrent neural network based language model. In Interspeech, volume 2, pp. 1045-1048. Makuhari, 2010.
OpenAl. Introducing chatgpt. OpenAl Blog, 2022.
OpenAl. Gpt-4 technical report. arXiv preprint arXiv: 230.::330000774.. 2023.
Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to follow instructions with human feedback. Advances in Neural Information Processing Systems, 35: 27730-27744, 2022.
Vassil Panayotov, Guoguo Chen, Daniel Povey, and Sanjeev Khudanpur. Librispeech: an corpus based on public domain audio books. 2015 IEEE international conference on acoustics, speech and signal processing (ICASSP), pp. 5206-5210. IERE, 2015.
Ashutosh Pandey, Chunxi Liu, Yun Wang. and Yatharth Saraf. Dual application of speech enhance- ment for automatic speech recognition. In 2021 IEEE Spoken Language Technology Workshop (SLT), pp. 223-228. IEEE, 2021.
Tae Jin Park, Kunal Dhawan, Nithin Koluguri, and Jagadeesh Balam. Enhancing speaker diarization
with large language models: A contextual beam search approach. arXiv preprini MXXY:23009.0528.. 2023.
Guilherme Penedo, Quentin Malartic, Daniel Hesslow, Ruxandra Cojocaru, Alessandro Cappelli, Hamza Alobeidli, Baptiste Pannier, Ebtesam Almazrouei, and Julien Launay. The refinedweb dataset for falcon Ilm: outperforming curated corpora with web data, and web data only. arXiv preprint arXiv:2306.01 16, 2023.
Archiki Prasad, Preethi Jyothi, and Rajbabu Velmurugan. An investigation of end-to-end models for robust speech recognition. In ICASSP 2021 -022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 6893-6897. IEEF, 2021.
Alec Radford, Jong Wook Kim, Tao Xu, Greg Brockman, Christine McLeavey, and Ilya Sutskever. Robust speech recognition via large-scale weak supervision. In International Conference on Machine Learning, pp. 28492-2888 28518. PMLR, 2023.
Srijith Radhakrishnan, Chao-Han Yang, Sumeer Khan, Rohit Kumar, Narsis Kiani, David Gomez- Cabrero, and Jesper Tegnér. Whispering llama: A cross-modal generative error correction frame- work for speech recognition. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pp. 10007-10016, 2023.
12

---

Published as a conference paper at ICLR 2024
Nils Reimers and Iryna Gurevych. Bentence-bert: Sentence embeddings using siamese bert-networks. arXiv preprint arXiv: 10084, 2019.
Joonbo Shin, Yoonhyung Lee, and Kyomin Jung. Effective sentence scoring method using bert for speech recognition. In Asian Conference Machine Learning, pp. 1081-1099 PMLR, 2019.
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302. 13711, 2023a.
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaci, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint wrxiv:2307.09288, 2023b.
Cassia Valentini-Botinhao, Xin Wang. Shinji Takaki, and Junichi Yamagishi. Investigating mn-based speech enhancement methods for noise-robust text-to-speech. In SSW, pp. 146-152, 2016.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention all you need. Advances in neural information processing systems, 30, 2017.
Christophe Veaux, Junichi Yamagishi, and Simon King, The voice bank corpus: Design, collection and data analysis of a large regional accent speech database. In 2013 O-COCOSDA/CASLRE, pp- 1-4, 2013.
Emmanuel Vincent, Shinji Watanabe, Jon Barker, and Ricard Marxer. The 4th chime speech separation and recognition challenge. URL: http://spandh. des. shef. ac. uk/chime challenge [Last Accessed on August, 2018), 2016.
Quan Wang. Yiling Huang, Guanlong Zhao, Evan Clark, Wei Xia, and Hank Liao. Diarizationlm: Speaker diarization post-processing with large language models. arXiv preprint arXiv:2401.03500 2024.
Siyin Wang, Chao-Han Huck Yang. Wu, and Chao Zhang. Can whisper perform speech- in-context learning. arXiv preprint arXiv:2309.0777 2023.
Shinji Watanabe, Takaaki Hori, Shigeki Karita, Tomoki Hayashi, Jiro Nishitoba, Yuya Unno, Nelson Enrique Yalta Soplin, Jahn Heymann, Matthew Wiesner, Nanxin Chen, et al. Espnet: End-to-end speech processing toolkit. arXiv preprini arXiv:1804.00015. 2018.
Jason Wei, Yi Tay, Rishi Bommasani, Colin Raffel, Barret Zoph. Sebastian Borgeaud, Dani Yogatama, Maarten Bosma, Denny Zhou, Donald Metzler, et al. Emergent abilities large language models. 2022.
arXiv preprint
Jian Wu, Yashesh Gaur, Zhuo Chen, Long Zhou, Yimeng Zhu, Tianrui Wang, Jinyu Li, Shujie Liu, Bo Ren, Linquan Liu, et al. On decoder-only architecture for speech-to-text and large language model integration. arXiv preprint arXiv:2307.03917. 2023a.
Shih-Lun Wu, Xuankai Chang, Gordon Wichern, Jee-weon Jung, François Germain, Jonathan Le Roux, and Shinji Watanabe. Improving audio captioning models with fine-grained audio features, text embedding supervision, and I1m mix-up augmentation. arXiv preprint vrXiv:2309.13355 2023b.
Chao-Han Huck Yang, Linda Liu, Ankur Gandhe, Yile Gu, Anirudh Raju, Denis Filimonov, and Ivan Bulyko. Multi-task language modeling for improving speech recognition of rare words. In Proc. IEEE ASRU, pp. 1087-1093. IEEE, 2021.
Chao-Han Huck Yang, Yile Gu, Yi-Chieh Liu, Shalini Ghosh, Ivan Bulyko, and Andreas Stolcke. Generative speech recognition error correction with large language models and ausk-activating prompting. In Proc. IEEE ASRU, 2023a.
Chao-Han Huck Yang, Bo Li, Yu Zhang, Nanxin Chen, Rohit Prabhavalkar, Tara N Sainath, and Trevor Strohman. From english to more languages: eframeter-effiiiet model reprogramming for cross-lingual speech recognition. In Proc. ICASSP, pp. 1-5. IEEF, 2023b.
13

---

Published as a conference paper at ICLR 2024
Yu Yu, Chao-Han Huck Yang. Jari Kolehmainen, Prashanth G Shivakumar, Yile Gu, Sungho Ryu, Roger Ren, Qi Luo, Aditya Gouray, I-Fan Chen, et al. Low-rank adaptation of large language model rescoring for parameter-efficient speech recognition. In IEEE Proc. ASRU, 2023.
Hang Zhang, Xin Li, and Lidong Bing. Video-Ilama: An instruction-tuued audio-visual language model for video understanding. arXiv preprint arXiv:2306. 02858, 2023a.
Renrui Zhang, Jiaming Han, Aojun Zhou, Xiangfei Hu, Shilin Yan, Pan Lu, Hongsheng Li, Peng Gao, and Yu Qiao. Llama-adapter: Efficient fine-tuning of language models with zero-init attention. arXiv preprint arXiv:2303. 1610999933.119, 2023b.
Long Zhao, Yuxiao Wang. Jiaping Zhao, Liangzhe Yuan, Jennifer J Sun, Florian Schroff, Hartwig Adam, Xi Peng, Dimitris Metaxas, and Ting Liu. Learning view-disentangled human pose representation by contrastive cross-view mutual information maximization. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp- 12793- 12802, 2021.
Hao Zhu, Huaibo Huang, Li, Aihua Zheng, and Ran He. Arbitrary talking face generation via attentional audio- visual coherence learning. In Proceedings of the Twenty-Ninie International Conference on International Joint Conferences on Artificial Intelligence, pp. 2362-2368, 2021.
APPENDIX
A ROBUST HYPORADISE DATASET DETAILS
Table 6: Robust HyPoradise dataset statistics in terms of number of hypotheses-rransccriionn pairs and average utterance length in various noise domains.
Domain
Training Set
Pairs
Length
Test Set
# Pairs
Length
Source
Category
test-real
1,320
16.4
test-simu
1,320
16.4
CHiME-4
Real-world noise
tr05-reat
9,600
17.0
dev-real
1,640
16.8
dev-simu
1,640
16.8
baby-cry
VB VE-DEMAND
Unseen noise
train
23,075
7.5
helicopter
824
7.7
crowd-party
babble car station train street airport exhibition
NOIZEUS
Real-world noise
train
23,807
7.1
30
8.1
restaurant
metro car traffic cafe babble
LS-FrreSound
Real-world noise
28,539
35.0
118
17.4
ac/vacuum
RATS
Radio noise
train
28,504
14.2
test 1,000
10.2
Total
train
113,525
16.8
test
10,340
13.7
A.1
ASR SYSTEM
For ASR beam search decoding, we employ Whisper Large- V2 (Radford et al. 2023), one large-scale pre-trained model developed by OpenAl to generate N-best hypotheses, which has been reported with
14

---

Published as a conference paper at ICLR 2024
several competitive and state-of of-the-art performance. Whisper model follows the encoder-decoder Transformer (Vaswani et al.) 2017) architecture with 1,550 million parameters, which is trained on 680K hours of multilingual and multitask supervised data collected from the web. As result, it shows universal and excellent noise-robustnee in various conditions though lacks of domain specificity (i.e., still lags behind the specifically trained model on certain dataset).
With such pre-trained ASR model, we employ the beam search algorithm for decoding and generate N-best hypotheses list for each speech sample, where the beam size is set to 50. After removing repetitive utterances, we select top-5 hypotheses in terms of posterior probabilities as N-best list. To develop the RobustHP dataset, we carry out this decoding strategy on multiple noisy ASR corpus (see SA.2) and generate data pairs of 5-best hypotheses and ground-truth transcription.
A.2 SPEECH CORPUS SELECTION
For speech corpus selection, our goal is to cover common noisy ASR scenarios in real world. Consequently, collect and simulate the following corpus with evident domain characteristics to compose the Robust HyPoradise dataset:
CHiME-4 (Vincent et al. 2016]: CHiME-4 is a popular dataset for far-field noisy speech recognition. includes real and simulated noisy recordings in four noisy environments, i.e. bus, cafe, pedes- trian area, and street junction. We use its tr05-real split (9,600 utterances) to generate RobustHP training data, as well as the rest-real (1,320 utterances), test-simu (1,320 utterances), dev-real (1,640 utterances) and dev-simu(1 1,640 utterances) splits generate the test data.
is a popular dataset for noisy speech recognition and speech enhancement. We use its training data build RobustHI, which contains 23,075 noisy utterances from 56 speakers in VoiceBank corpus (Veaux et al. 2013) that are recorded at sampling rate of 16 kHz and mixed with 10 different noise types (babble, cafeteria, car, kitchen, meeting, metro, restaurant, speech-shaper noise, station, traffic) at SNR levels of O, 5, 10, and 15 dB. For test set, to simulate the challenging unseen noise conditions in practical, we mix the VoiceBank clean test data with three new types of noise (Lin et al. 2021 i.e., baby-cry, helicopter, and crowd-party, at SNR level of OdB. The test set contains 824 utterances from 2 speakers.
VoiceBank-DEMANN Valentini-Botinhao et al. 2016): VoiceBank-
NOIZEUS (Hu & Loizou 2006): NOIZEUS is noisy speech corpus developed to evaluate noise- robust speech recognition and speech enhancement algorithms. It only contains a test set of 30 IEEE sentences (produced by 3 male and 3 female speakers) corrupted by 8 different real-world noises at SNR levels of 0, 5, 10, and 15 dB, where we select 5 dB for main experiments. The noise was taken from the AURORA-2 2 database (Hirsch & Pearce, 2000) that includes suburban train noise, babble, car, exhibition hall, restaurant, street, airport and train-station noise. To match the short length of NOIZEUS test utterances 18.1 tokens in average), we select the clean speech from LibriS peech train-clean- 100 and VoiceBank corpus that with no more than 12 tokens in transcription, and mix them with AURORA-2 noises at SNR levels of 0, 5, 10, 15, and 20 dB to form training set.
is a simulated noisy speech corpus for noise-robust speech recognition, which mixes the clean speech data from LibriS train-clean-100 split (Panayotov et al. 2015) and noise data from FreeSound corpus (Font et al. 2013) at SNRs of 5, 10. 15, 20, and 25 dB form the training set. For test set, they select 118 clean speech samples from LibriSpeech rest-clean split and mix them with FreeSound noise at SNRs of 0, 5, 10, 15, and 20 dB, where we select dB for main experiments. Six noise types in FreeSound are employed, including metro, car, traffic, cafe, babble and ac/vacuum.
LübriSpeech-FrreSounn (Prasad et al. 2021 I): LibrrSpeechFFreeSooe
RATS (Graff et al.) 2014): Robust Automatic Transcription of Speech (RATS) dataset contains radio- communication speech in ultra high frequency data category that is extremely noisy and challenging for ASR task. Its training data contains 43, 12 noisy speech utterances, where we filter out the low-quality samples (i.c., WER by Whisper is larger than 0.9) to form the training set. Its test set contains 7,591 utterances, where we randomly select 1,000 samples for higher evaluation efficiency.
A.3 STATISTICS
After performing beam search decoding the selected speech corpus introduced above, we collect 1 13K pairs of N-best hypotheses and ground-truth transcription to form the RobustHP dataset. The statistics are presented in Table which illustrates the number of hypotheses pairs and
15

---

Published as a conference paper at ICLR 2024
Adaptation Prompt
Transformer Layers x L
Adapter
Attention
*
Noise Emb.
Finetune
Transformer Layers H--L
Frozen
*
Attention
Concatenate
Subtract
Tokens
Figure 5: LLaMA-Adapter tuning (Zhang al. 2023b) with language-space denoising (ours).
the average utterance length in various domains and splits. We would release the RobustHP dataset to public upon publication and open the development venue for more data.
B
METHOD DETAILS
B.1 DENOISED LLM FINETUNING
B.1.1 EFFICIENT LLM FINETUNING: LLAMA-ADAPTER
As presented Fig. we employ LLaMA-Adapter (Zhang et al. 20235) for efficient LLM finetuning. Given pre-trained LLM with 1/-layer Transformer, it inserts set of learnable adaptation prompts into the top-L layers that learn high-level semantics. Denote the prompt for I-th Transformer layer as 9i RR U where denotes the prompt length and D denotes the LLM embedding size.
Assume we have M tokens containing instruction and already generated response, i.c., Ti E RMXD, where is the layer index, now we to predict the (M + 1)-th token as part of response. In order to finetune the entire system, the learnable adaptation prompt is concatenated with Ti as prefix, i.e., [T E (UU+MM) In this case, the instruction knowledge learned by Gi can guide the Ti to generate the subsequent response under teacher-forcing supervision.
Furthermore, considering the prompt Gi is randomly initialized and thus may disturb the LLM tuning at early training stages, vero-initialized attention mechanism is designed mitigate such disturbance. Suppose the LLM is going to generate the (M + 1)-th token based on the prompt G and history tokens Tj at I-th layer, and we denote the current M-th token T((M) E R In attention mechanism, there are firstly three projection layers to generate query. key and value, respectively:
Q Liinar, (T(()) Ki Linear ([G:: Ti)), Vi Lincar, ((G;;
Thereafter, the attention score between key and value can be formulated as A₁ = Qi Ki/VD
[lX(U+M) which captures the correlation between current token (M(M) and all M existed tokens Ti well the prompt Gi predict next token. Therefore, Ai could be split into parts:
A = :
(10)
where Al
[I denotes the attention score of U adaptation prompts and AT E Mx1 denotes that of M history tokens. Since the adaptation prompts are randomly initialized, their attention scores may cast disturbance next-token prediction in early training stages. To this end, learnable gating factor with zero initialization is introduced to adaptively control the importance of prompt in attention, by directly multiplied with its softmax weights from Eq Eq.(10):
Ap [gt softmax (A$); softmax(A7))
(11)
Finally, the attention output of I-th Transformer layer can be calculated with a linear projection:
MM Linear, (A) Vi) E IXI
(12)
16

---

Published as a conference paper at ICLR 2024
Table 7: Comparison between main configurations of different popular LLMs.
LLM
LLaMA-2-7b
LLaMA-7b 7b
Falcon-7b
LLaMA-2-13b
Number of Transformer Layers
32
32
32
40
Number of Attention Heads Nhead
32
32
71
40
Embedding Size D
4,096
4,096
4,544
5,120
Block Size B
4,096
2,048
2,048
4,096
Vocabulary Size V
32,000
32,000
65,024
32,000
then utilized to predict the next token 1(M++) as part of output response. The proposed zero- initialization mechanism achieves an effective trade-off between the pre- preitri knowledge of LLM and the learned instructional knowledge through adaptation prompt.
B.1.2 DENOISED ADAPTER TUNING
Apart from text instructions, I.LMA--dapper is also capable of generating response based on other modality inputs (Zhang al.) 2023b). However, the cross-modal gap between text and other modalities may affect the finetuning stability and performance (Li et al., (023b). Therefore, we propose to extract language-space noise embedding in $4.2 to replace audio embedding for representing the noise conditions of source speech, i.e., EIN [ E E D according to Eq. 12, where N denotes N-best list size and Daben denotes SBERT embedding size. Then, we incorporate into LLaMA-Adapte for denoising via element-wisg subtraction:
(ELN) E UUD, ghth 91 9i dn
set U == (N 1),
(13)
where Tw Dx denotes the linear projection tuner introduced in $4. for audio noise distillation, the subtraction operation denotes "denoise". The 9fth is a gating factor to control denoising process. Therefore, the resulted gon indicates the adaption prompt with language-space denoising, which will replace the Gj in Eq. 12 for adapter tuning.
B.2 AUDIO NOISE DISTILLATION
As illustrated $4.3
the key idea audio noise distillation to transfer the real noise information in audio embeddings
extracted language-space noise embedding, in order to enhance its representation ability of audio noise. The approach we propose is based on mutual information neural estimation (MINE) (Beighazi al. 2018), which can be split into two stages in Algorithm First, we update the MINE to learn MI estimation, by maximizing the MI between language-space noise embedding and noisy audio embeddings and minimizing the MI between language embedding and clean audio embeddings, i.c., audio noise information exists in noisy speech instead of clean speech. Second, we introduce learnable tuner modulate the language-spaes space embedding include more real noise information by maximizing the MI between it and noisy audio embeddings, which is also jointly optimized with L.M finetuning (i.e., the GER cost function LuZT as formulated Eq.
The rationale we leverage MINE for distillation instead of other techniques like contrastive learning due to its strong distinguishing ability, which has been verified by recent applications (Zhu et al. 2021 Zhao et al. 2021 Li et al. 2022: Hu al. 2023e). On the other hand, directly employing techniques like contrastive learning may not work as the language embedding could be far away from the audio-space noisy and clean embeddings, which means the distance between positive and negative samples (i.e.. within audio space) is much smaller than the distance between them and the anchor (i.e., between audio and language spaces). Our ablation study in Table 14 also verifies this limitation.
17

---

Published as a conference paper at ICLR 2024
C EXPERIMENTAL SETUP DETAILS
C.I MODEL SETUPS
LLMss We select three latest and popular LLMs for evaluation. including L2-74 (Touvron al. 2023b), LLaMA-7t (Touvron et al. 2023a), Falcon-7t (Penedo et al. 2023 In addition, to explore the influence of LLM model size to our approach, we also report some results on LLaMA-2- 13b mode Touvron et al. 2023b). Table 7 compares their main configurations.
Adapter: We follow the default setting of ALLMM--Adaper [Zhang et al. 2023b R9 with some modifications. The number of tunable Transformer layers L is set to 1 1, which means all layers except the first one are tunable with inserted prompts. The prompt length U is set to 20 to match
the length of FLN that equals to N (N 1), where N is the N-best list size set 5. To extract the language-space noise embedding from N-best hypotheses, we utilize sentence-BERT" 10
Reimers &
Gurevych| 2019) whose embedding size shhrrt is 384.
MINE. MINE introduces a statistic network vhe that contains a multi-layer perceptron (MLP) and a Sigmoid activation function to estimate mutual information value between 0 and receives two inputs including the Whisper-encoded audio embeddings of size 1280 and the language-spate noise embedding of size 384, which are first projected same hidden dimension and added together, and then go through MLP to generate output of size 1. In particular, to incorporate the modulated noise embedding (with same size as LLM embedding, different from the input language embedding size 384) into MINE, we design extra interface to receive it intermediate features on language-space feature branch. The noise embedding tuner contains a linear projection from the SBERT size of 384 to the LLM embedding size described in (B.122
C.2 TRAINING AND EVALUATION SETUPS
LLM Finetuning. The learning rate is set to 10-2 for CHiME-4 that is relatively small, and set to x 10-3 for relatively large datasets including VB-DEMAND, NOIZEUS, LS-FFreSound and RATS. The batch size is set to 4, with accumulation iterations set to 8 (e.g., effective batch size is 32). We train 2 epochs with Adam W optimizer (Loshchilov & Hutter 2018), with weight decay set to 0.02 and warmup steps set 20% of one epoch's steps. In addition, MINE is updated using extra Adam optimizer with learning rate that 10% of LLM tuning. where all other configurations keep the same. The hyper-parameter for model training, which takes 1.5 hours for CHiME-4, 2.0 hours for VB-DEMAND, 1.6 hours for NOIZEUS, 4.5 hours for LouFre and 3.8 hours for RATS, respectively.
Algorithm D iis set 0.5. We use 1 NVIDIA A40 GPU
Instruction-followee Finetuning, As presented in Fig. 2 we leverage pnstruction-following finetun- ing strategy for GER, where we design an instruction template:
"Below is the best- hesthhypthheses transcribed from speech recognition system. Please try to revise it using the words which are only included into other-hypothesis, and write the response for the true tran- seription.### Best-hypothesis: (iiiss:((bbssbs I-best hypothesis) (aaessssssttt ### Other-hypothesis: ( (2~NNbest hypotheses) #HH Response:"
We find that different instruction templates would have slight impact on the final GER performance, which is an open question for further discussion. particular, we design some constraints (e.g., only use the words inside N-best hypotheses list for error correction) to control the quality of response and avoid potential LLM hallucinations (Feldman et al.) 2023).
Response Generation. In the generation stage, we adopt temperature of 0.2 and top-1 sampling, ie.. greedy search. We observe the over-confidence phenomenon in our experiments (i.e., output
py
18

---

Published as a conference paper at ICLR 2024
Table 8: WER (%) results of RobustGER with LLaMA-7b finetuning. "Lnk denotes LM
rescoring. "+ Audio Denoising" denotes introducing audio embedding to denoise GER. Onb and Ocp respectively denote the N-best oracle and compositional oracle that are defined in $5.1 The subscript percentage denotes relative WER reduction over ASR baseline, i.e., GER improvement.
RobustGER (ours)
Oracle
Test Sct
Baseline
LM
GER
Audio Denoising
Onb
Ocp
test-real
12.6
12.2
6.8 6..00
6.6.67.0%
5.7-54.0 557.5%.4%
10.5
3.0
test simu
15.4
14.5
10.1 -3.4%
9.7 9.7-37.0%
8.5
12.9
5.0
CHIME-4
dev-real
10.6
10.3
4.9 63.8%
4.7 55.7%
4.0 62.3%
9.1
2.1
dev-simu
11.9
6.9-
65..4
49.2%%
10.6
3.3
avg-
12.8
12.2
7.2 43.9%
7.0 45.35
6.1 62.39%
10.8
3.4
baby-cry
8.0
7.8
7.1-11.35
7.2 10.056
6.5
-8.-18.8%
4.5
3.0
helicopter
8.4
8.1
7-3 15.115.155
7.2 14.35
6.8 -9.0%9
4.8
3.2
VB-DEMAND
crowd-party
22.3
21.5 4.99
21.1-.6.65 6..6%
1..
16.5
11.5
avg.
13.0
12.7
12.0 7.77%
11.8 9..2%
11.1
1.4.6%
8.6
5.9
babble
16.5
16.7
15.3
15.0
13.6
-1..9%
9.5
5.8
car
17.4
16.8
14.9-14.4%
14.8 14.9919999
14.9
-4..9144.4%
14..4%
9.9
7.9
station
10.7 10.8%
10.7 10.7-10.8%
14.2%
train
15.3
15.2
14.5 2..5%
14.2-
12.8 10.3%
10.3
7.9
NOIZEUS
street
17.4
17.2
16.9
16.7 4.00%
16. 1
12.4
9.9
airport
11.0
10.3 8.008
10.1 9.8%%
9.5 16.2%
7.9
4.5
exhibition
13.2
13.2
13.2 00.0%
13.0-11.00
12.8 33.0%
8.3
5.8
restaurant
13.2
13.0
13.6
13.2
12.0 99.1%
8.7
6.2
avg.
14.5
14.3
13.7 6.33%
13.5 6..9%%
12.8
11..5%
9.2
6.6
metro
9.9
9.8
9.4 5.1%
9.2 7.1%
8.2
17.2%
7.9
4.9
3.5 12.0%
3.6 10.0%
17.6%
1.8
traffic
8.3
8.2
8.3
8.3
8.2 1.2%
6.8
4.5
L.S-FreSSonnd
cafe
9.8
9.5
9.3 9.3_
9.1 9.1_7.1%
8.5 15.3%
7.1
4.6
babble
32.0
31.8
31.7-0.99
31.4..99
30.9 55.4%
28.7
19.3
ac/vacuum
12.4
12.5
11.8-1.8%
11.6_6.5%%
11.2 9.72_
10.2
6.2
avg.
12.7
12.6
12.3
12.2
11.7
10.6
6.9
RATS
test
45.7
45.6
45.5
45.2
43.6
38.8
23.6
probability distribution for decision close to one-hot), which results similar performance with different & for top-k sampling. Therefore, we select top-1 sampling for higher decoding efficiency.
LM Rescoring Baseline. For LM baseline in Table D we use a Transformer- LM for typical rescoring, which is trained the text transcriptions of each RobustHP subset using ESPnet toolkit [1] (Watanabe et al. (0088). The LM contains 16 Transformer layers with heads and 512 attention units, and it is trained for 25 epochs with Adam optimizer (Kingma & Ba 2014). The learning rate set to 5 x 10 with 25,000 warm-up up steps.
In-context Learning Baseline. We implement in-context learning baseline for case study in Table 5 which is effective making full use LLM's powerful reasoning ability and linguistic knowledge (Dong et al.) 2022). In partieular, utilize ChatGPT to conduct GER task using task- activated prompting (TAP) (Yang et al. 2023al: we first prompt ChatGPT to summarize what is ASR and typical LM rescoring, and then inform the definition of ASR generative error correction. followed by several examples to teach it how to do such kind of error correction. With above background knowledge, we finally ask perform GER for our sample in case study.
Details of i-SNE Visualization. Fig. 4 and 6 present the t-SNE visualization of the language and audio noise embeddings. The language embeddings are the outputs of distillation tuner, which are selected from the LS- ---ereeSodnd test samples. The audio embeddings are encoder outputs of Whisper ASR model, where the speech samples also come from LS-FFreSound test samples. In particular, for better visualization (i.e., around 5.78 in total from LS-FreeSound test data), as the distance between different phonemes much larger than that between different noise conditions.
employ Stable- " to extract the speech segments of same word "for"
I httees//tttti.ecessss////e//eeer httees//gitthbb..cm///aaafftttfa
19

---

Published as a conference paper at ICLR 2024
Table 9: WER (%) results of RobustGER with Falcon-7b finetuning. "I.M denotes LM rescoring. "+ Audio Denoising" denotes introducing audio embedding to denoise GER. Onb and Ocp respectively denote the N-best oracle and compositional oracle that defined in S.1
RobustGER (ours)
Oracle
Test Set
Baseline
L.Maakk
GER
Audio Denoising
Onb
Orp
lest-reul
12.6
12.2
7.4_ 41.3%
7.2 42.0%
6.2 50.8%%
10.5
3.0
14.5
10.2
33.5%
10.0 a5.17s
42.25
12.9
5.0
CHIME-4
dev-real
10.6
10.3
5.8 45.0%
5.5 4.514
4.8 64.1%%
9.1
2.1
dev-simu
1244
11.9
7.7-37.0%
7.4-41.7%
6.5 6.5-47.0%
10.6
3.3
avg.
5.8
7-5_
6.8_4884%%
baby-cry
10.0%
12.99
10.8%
4.5
3.0
VB-DEMAND
helicopter
8.4
7-8 7.1%
s.31
7.2 7.2- 14.89
4.8
3.2
crowd-party
22.6
22.3
21.7 21.7-...0%
21.4....5%
20.5_..3%
16.5
11.5
avg.
12.7
12.2 -.20%
7.77%
11.5%%
5.9
babble
16.7
46.94.2.4%
16.5 0..0%
15.3 7..5%
9.5
5.8
17.4
16.8
15.7 -9.7-9.8%
15.4_11.55
14.9
14.9-14.4%
9.9
7.9
station
12.0
11.6
11.6
11.2 6.7%
9.1
6.6
5.0
train
15.3
15.2
16.5
14.6
12.8 16.3%
10.3
7.9
NOIZEUS
17.2
16.1 7.00
16.0 8.0%
16.1 7.5%
12.4
9.9
airport
11.2
11.0
10.7
10.6
10.3 88.0%
7.9
4.5
exhibition
13.2
13.2
12.8-3.09
12.5
12.0 -9.1%
5.3
5.8
restaurant
13.8
12.6_..5%%
12.0 99.1%
arg.
14.5
14.3
14.1
13.7
12.8 1.1.9%
9.2
6.6
netro
9.9
9.8
10.3 4.00%
9.9 0.0%
8.9 10.1%
7.9
4.9
car
4.0
4.0
3.7 -..5
3.7 3.7-7.556
3.5
3.0
1.8
traffic
8.2
8.2-1.2%
8.0_5..6%
7.5 9..6%
4.5
LS-FreeSound
cafe
8.1-1
8.0_188.95
7.9 19.4%
7.1
4.6
babble
32.0
31.8
31.1 -1.1_-2.8%
30.9 30.9_3.4%
30.5
28.7
19.3
ac/vacuum
12.5
12.6 110005
12.6 1.00%
12.2 1.6%
10.2
6.2
avg.
12.7
12.6
12.3-3.1%
12.2 -1.99
11.8 7.15%
10.6
RATS
45.7
45.6
45.3_..98
41.9
43.3-s.3%
38.8
23.6
Table WER (%) results of RobustGER with LLaMA-2-13M finetuning. "I.Mrant" denotes LM rescoring. "+ Audio Denoising" denotes introducing audio embedding to denoise GER. and Ocp respectively denote the N-best oracle and compositional oracle that are defined in $5.1
RobustGER (ours)
Orucle
Test
Baseline
LM
GER
Audio Denoising
Ogb
Oup
test-real
12.6
12.2
5.5 50.0%
5.3
4.9
61.1%
10.5
3.0
8.1 47.4%
8.2 8.2..6.5%%
48.7%
12.9
5.0
CHiME-
dev-real
10.6
10.3
4.1 61.3%
3.8 64.2%
3.3
08.0%
9.1
2.1
dev-sima
12.4
11.9
6.1 65..80
5.9 2.....2.49
5.1
10.6
3.3
avg.
12.8
6.0 53.1%
5.8 54.7%
5.3
68.0%
10.8
3.4
baby-cry
6.7 19.0%
6.6.17.00%
21-20..99 0%
4.5
3.0
VB-DEMAND
helicopter
8.4
8.1
142_14.3%%
7.0_166.99
6.5 6.5_
22.88
4.8
3.2
crowd--arry
22.6
22.3
21.0
20.6 20.6....8%
19.6
16.5
11.5
avg.
13.0
11.6 10.8%
11.4 12.3%
10.7 17..7%
8.6
babble
16.5
15.3-7.39
15.2 7.99%
15.3 7.30%
9.5
5.8
17.4
16.8
14.9-14.4%%
14.7 15..9%
14.0
19.8%
9.9
7.9
station
12.0
11.6
9.5
9.4
9.1
6.6
5.0
train
15.3
15.3 -0.05
14.7 3..15
12.8 36.096
10.3
7.9
NOIZEUS
17.4
16.9 2..0
16.9 2.99
16.9 2.9%%
12.4
9.9
airport
11.2
11.0
10.7-1.5%%
10.3 10.3-s.0%
8.7
7.9
4.5
exhibition
13.2
13.2
12.0 12.0-9.199
11.6_
10.7 16.9%%
8.3
5.8
restaurant
12.4 6.29%
12. 9.3%
22.22%
8.7
6.2
avg.
14.5
14.3
13.4-.7.68 7.09%
13.1
12.2 15.99
9.2
6.6
metro
9.9
9.8
9.7
9.4_s
8.6 13.1%
7.9
4.9
car
4.0
4.0
3.7 77.5%
3.8 00
3.5
1..51
12.8%
3.0
1.8
traffic
8.3
8.2
7.6 88.4%
LS LS-FreeSound
cufe
9.8
9.5
8.7 8.7-11.29 11.2%
8.5 10.3%%
7.5 29.6%
7.1
4.6
babble
32.0
31.8
31.8 0.07%
31.66...15
30.8 30.8-3.a9g 3.88%
28.7
19.3
acfvacuum
11.5_7.3%
11.4-s.1%
11.0 11.3%
10.2
avg.
12.7
12.6
12.3 12.3__a.9%
12.2 12.2
11.5 9..4%
10.6
6.9
RATS
test
45.7
45.6
44.4 2.8%
44.0....15 8.7%%
43.0 6.00
38.8
23.6
20

---

Published as a conference paper at ICLR 2024
Table 11: WER (%) results of RobustGER different SNR-level testing conditions. The test sets are from LS- LL--rreSound dataset, with five SNR levels (i.e., {0, 5, 10, 15, 20)dB) on six noise types (i.e., "Metro", "Car", "Traffic", "Cafe", "Babble", and "AC/Vacuum").
RobustGER (ours)
Oracle
Noise Type
SNR (dB)
Baseline
LMrank
GER
Audio Denoising
Oyb
0
9.9
9.8
9.5 9.0%%.0%
9.4 5.1%
8.9
-0.1%
7.9
4.9
5
7.2
7.0
6.7 68.9%
6.4
5.5 23.6%
5.5
3.2
10
4.8
4.6
4.2 12.5%
4.3
4.0
3.9
2.3
Metro
15
3.9
3.5
3.2 17.9%
3.2-17.9%
17.9%
3.0 25.1%
1.7
20
3.3
3.1
2.7 2.7 18.2%
2.6-21.2%
2.3 30.3%
2.6
avg.
5.8
5.6
5.3 8.6%%
5.2 10.3%
4.7 19.0%
0
4.0
4.0
3.7 7..5%
3.5 12.5%
3.1 22.8%
3.0
5
3.8
3.5
3.1 18.49
3.1-18.4% 18.45
2.8
26.3%
Car
10
3.2
3.3
3.2 -0.0%
3.0 -6.3%
2.2
81.3%
2.4
15
2.8
2.7
2.5 10.7%
2.5 10.75%
2.3
-1.9%%
2.4
1.4
20
3.1
2.8
2.5 19.4%
2.4 22.66%
2.1 -3.3%
2.4
1.4
3.4
3.3
3.0
2.9
2.5
26.5%
2.6
1.5
0)
8.3
8.2
8.0 8.6%
7.8-6.05
7.5 99.6%
6.8
4.5
5
6.3
6.1
5.6
5.5
4.9 222.2%
4.9
3.2
10
3.8
3.6
3.3 13.236
3.3
3.2
3.2
1.9
Traftic
15
3.4
3.1
2.9 14.7%
2.8. 155-117.656
2,4
2.8
1.7
20
3.7
3.5
3.4. 8.1%
3.3 10.88%
3.0 11.9%
2.9
avg.
5.1
4.9
9.8%%
4.5 11.8%
4.2 17.6%
0
9.8
9.5
8.
17.3%
8.1 117.5%
7.5. 29.8%
7.1
4.6
5
5.7
5.7
5.4-..3% 5.3%
5.4
5.6 5.886
5.3 7.0%
4.5
10
5.0
4.7
4.5 10.0%
4.2 16.0%
4.0 20.0%
3.8
Cafe
15
3.6
3.5
3.3 8.3%
3.2 11.11%
3.0 16.7%
2.7
20
3.5
3.2
2.7 22.9%
2.9 17.116
2.9 17.1%
2.6
1.5
5.5
5.3
4.8
4.8
4.5 18.2%
4.1
2.5
0
32.0
31.8
31.3
31.6
31.1 2288%
28.7
19.3
5
17.0
16.8
17.0 0.00%
16.6 2..4%%
16.3 4.1%%
13.9
9.2
10
8.8
9.0
8.6 2.33%
8.4.4.55%
8.1 88.0%
6.5
3.9
Babble
15
6.5
6.1
5.8
5.7 -1..12
5.4-16.90 16.9%
4.7
3.0
20
10.5
10.1
7.6
27.6%
7.6_227.05
7.6 27.8%
9.6
2.0
avg.
15.0
14.8
14.1 6.0%
6.79%
13.7 8..7%
12.7
0
12.4
12.5
12.3
12.1-2.4%
11.4 11.4-8.15
10.2
6.2
5
7.4
7.0
6.3 14.9%
5.8 21.6%
5.5
AC/Vacuum
10
6.6
6.2
5.5
16.7%
5.6 15.255
5.5 16.7%
15
4.4
4.2
3.7
15.9%
3.7 15.99
3.6 18.2%
20
3.8
3.7
3.3
18.2%
3.2 15.88
2.9
28.7%
2.8
1.4
avg.
6.9
6.7
6.3 8.7%
6.2 10.11%
5.8 15.99
5.3
3.0
Clean
3.0
2.8
2.5
16.7%
2.4
2.4-20.05%
2.1-30.0%
50.00
2.5
1.4
Table 12: WER (%) results of RobustGER on clean test data from VB-DEMAND and LS- S--FreSSonn.
RobustGER (ours)
Oracle
Test set
Bascline
LMrank
GER
Audio Denoising
Onb
Ocp
VB-DEMAND LS-FreeSound
1.5
1.3 0..0%
1.2 7.7%
0.7 46.2%
0.3
3.0
2.8
2.5 16.7%
2.4-20.05
2.1 30.0%
2.5
1.4
D SUPPLEMENTARY EXPERIMENTS
D.1 RESULTS ON DIFFERENT LLMS
Apart from LLaMA-2-7b, also evaluate proposed RobustGER approach popular LLaMA- 7b and Falcon-7b models as illustrated in Table and To further investigate the effect of LLM size on RobustGER, conduct extra experiments LLMMA-2-13b in Table 10
Similar to the results of LLaMA-2-76 in Table our proposed RobustGER achieves consistent gains of performance various LLMs and testing conditions, which verifies its general effectiveness. On the other hand, there exists some performance difference between different LLMs. In particular, LLaMA-2-153 13b outperforms all the 7b L.MSs due its larger model capacity and stronger language
21

---

Published as a conference paper at ICLR 2024
Table 13: Ablation study of the language-space noise embedding in terms of text embedding extractor.
"LLaMA Emb." denotes the input embedding layer of LLaMA-2-76 model.
Language Denoising Fastleat
Test Set
Bascline
+ Audio Denoising
LLaMA Emb.
SBERT
test real
12.6
6.5
6.4
6.6 47.0%
6.2
5.9
test-simas
9.2-60.3%
9.0
12.5%
8.7-15.55
-1..2%
CHiME
dev-real
10.6
5.0_82.8%
4.9-55.8%
4.9 69.8%
4.5-67.88
4.4 688.8%
dev-simu
12.4
6.8_15.2%
0.6-16.65
0.7-16.055
6.4-18.4%
6.1
avg.
6.9 -6..1%
6.7 44..47
00.09t
40.2%
-0.88%
baby-cry
8.0
7.0_1225%
6.9
6.8 6.8 15.06
6.5-18.8%
6.4 6.4_20.0%
TE-DEMAND
crowd-porty
helicopier
22.6
8.4
21.4_..35
7.4
7.3--33.1%
2..0
20.9 -7.5%
7.5 10.75
20.3-10.244
7.4
15.0% 7..-15
10..-11.000 11..9%
13.0
11.9_8.8%
11.7 11.7-10.055
11.7 10.008
11.4_12.85
11.1 14.0%
Table 14: Comparison of different techniques for audio noise distillation. "T-S Learning" denotes teacher-studen, learning with KL regularization, " "Contra. Learning" denotes contrastive learning.
Audio Noise Distilation Contra. learning
Test Set
Baseline
GER
Lang. Demoising
T-S learning
MINE
test-reaf
12.6
6.5
48.499
5.9 50.2%
5.9 68.2%
5.8 54.0%
5.6 66.0%
test simu
15.4
9.2 40.3%
8.6
8.7 8.7
45.5%
8.4 46.8%
8.2 8.2_%6..%
CHIME-4
dev-rea!
10.6
5.0-
4.4_ss
4.5-s7
4.2-c0.15
4.1 -61.3%
dev-simm
12.4
6.8
6.
6.0
6.1 -0.0%%
5.8
avg.
46.1%
6.3 $0.8%
6.3 50.855
$2.8%
68.0%
baby-cry
8.0
7.0 12.0%%
6.4 20.0%
6.4 20.005
6.2 22.0%
6.0 25.0%
VB DEMAND
helicopter
8.4
7.4-
7.1-11.58 15-
7.2-14.3%
67.91
6.9_17.9%
croud party
22.6
21.4
19.9 11.998
20.1 11.1%
19.5 13.75
19.2. 16.0%
ovg.
13.0
11.9
11.1 14.855
11.2 10.8%
10.8
10.7 11.7%
generation ability. Among 7b models, LLaMA-2-76 outperforms LLaMA-7b and Falcon-70 7 thanks larger-scale training data and longer context length.
D.2 RESULTS ON DIFFERENT SNRS
Table 11 reports more results different-SNR testing conditions. Similar to Table we can observe consistent performance gains RobustGER over vanilla GER and audio denosing baselines under different noise levels, i.e., ranging from 0 dB (quite noisy) to 20 dB (relatively clean). In addition, RobustGER also surpasses the reranking upper-bound ond under some testing scenarios, indicating the effectiveness of RobustGER over conventional LM rescoring methods.
Furthermore, we also report error correction results on clean test data from VB-DEMAND and LS- LS-FFreSSund datasets, where significant GER improvement with 46.2% and 30.0% relative WER reductions are achieved by RobustGER approach. This experimental evidence demonstrates the excellent generality of RobustGER for various ASR scenarios.
D.3 ABLATION STUDY OF LANGUAGE EMBEDDING EXTRACTOR
Table 13 illustrates the ablation study of proposed language-space noise embedding with different text embedding extractors. First, we try the input lord-to-embedding layer L.LaMA-2-7b to extract both utterance- and token-level embeddings in $4.2) which leads to minor gains over audio denosing baseline, indicating that the LLaMA embedding is less discriminative for audio noise modeling. The supervised text classifier FastText (Grave al. 2018 provides better solution to extract text embeddings for modeling the N-best list diversity. Benefiting from the powerful global context modeling ability of Transformer (Vaswani et al. 2017). SBERT (Reimers & Gurevych 2019) presents the best performance for language-space noise embedding extraction, which well represents both utterance- and token-level embeddings as shown in Table 3
22

---

Published as a conference paper at ICLR 2024
N-best hypotheses from speech sample under different noise conditions. We two noise types (i.e., Babble and AC/Vacuum) and two SNR levels (i.c., and 10 dB) from LibriSpeech- FreeSound test set, where the original sample id is "237-134500-000" the decoding score from Whisper Large-V V2 model, which is calculated by negative entropy. Red font highlights the wrong tokens compared to ground-truth transcription.
Table 15:
The "Acoustic Score" denotes
Noise Type
Acoustic Score
SNR (dB)
N-best Hypotheses
WER (%)
pray them but that is the same pray pray for them but that is not the same pray science pray for them but that is not the same if prayed for pray for sons pray for them but that is not the same if pray for sam
-0466
33.3
0.485
33.3
0.516
26.7
pray for them but that is not the same
0.517
33.3
0.521
33.3
Babble
pray for you but that is not the same as if you prayed yourself
0.328
0.0
pray for you but that is not the same as if you prayed yourself
0.328
0.0
pray for you but that is not the same as if you pray yourself
0..40
6.7
pray for you but that not the same as if you pray for yourself
0,426
13.3
pray you but that not the same as you prayed for yourself
0.449
pray for you but that not the same as if you prayed yourself pray for you but that not the same as if you pray yourself pray for you but that not same as if you pray for yourself
0.329
0.0
0.369
0.388
would pray
you but that not the same if you prayed yourself
0.428
6.7
pray for you but that not the same if you prayed for yourself
0.429
AC
pray for but that not the same as if you prayed yourself
0.305
0.0
pray for you but that is not the same as if you prayed yourself
0.305
0.0
10
prayed for you but that is not the same as if you prayed yourself
0.343
6.7
prayed for you but that is not the same as if you prayed yourself
0.343
6.7
prayed for you but that is not the same as you prayed yourself
0.343
6.7
pray for you but that is not the same as if you prayed yourself
-0.280
0.0
pray for you but that is not the sume as if you prayed yourself
0.280
0.0
Clean
OU
pray for you but that is not the same as if you prayed yourself
-0.800
0.0
pray for you but that not the same as if you prayed yourself
-0.280
pray for you but that not the same as you prayed yourself
0,280
Ground Truth
pray for you but that is not the same as if you prayed yourself
babble OdB
10:B
ac, 0dB
ac
clean
0
-0
-40
-60
80
60
40
2.0
20
40
Figure 6: The t-SNE visualizations of language-spue noise embeddings from source speech under different noise types and SNR levels. The average distances between embeddings of clean and various noisy conditions are: 58.6 (babble OdB), 24.5 (babble 10dB), 22.6 (ac OdB) and 14.3 (ac 10dB).
D.4 ABLATION STUDY AUDIO NOISE DISTILLATION
Table 14 explores different KD approaches for audio noise distillation. The first one is teacher- student learning, which implements distillation by performing KL-divergence regularization between trainable student and frozen teacher, but it shows minor gains of performance. In comparison, contrastive learning technique achieves better results by introducing positive vs. negative samples to learn distinctiveness. However, it is still sub-optimal due the large distance between language
23

---

Published as a conference paper at ICLR 2024
and audio spaces, i.e., the anchor (language noise embeddinn) is far away from the positive (noisy audio embedding) and negative (clean audio embedding) samples that are relatively closer to each other. To this end, utilized MINE introduces a neural network to estimate and maximize mutual information, which more direct and effective in manipulating representations in different spaces for knowledge distillation. As a result, MINE achieves the best performance of audio noise distillation.
D.5 RELATIONSHIP BETWEEN NOISY SPEECH AND N-BEST LIST DIVERSITY
As introduced in D our insight of proposing language-space noise embedding to represent audio noise is the relationship between the noise conditions of source speech and the diversity of decoded N-best list from ASR model, i... the worse noisy conditions (more challenging noise type or lower SNR), the higher uncertainty of ASR beam search decoding, and thus results more diverse N-best hypotheses. To verify the reliability of this insight, sample under different noise conditions in Table [5] For Babble noise, we can observe that 0 dB yields higher decoding uncertainty (i.e., lower acoustic scores) than 10 dB, which results in more diverse N-best hypotheses and worse 1-best WER, i.e., more language noise. Similar phenomenon can be observed in AC noise condition. On the other hand, we notice from Table [T that Babble noise under same SNR level yields worse ASR results than AC noise, which means Babble is more challenging noise type. As a result, Babble OdB produces more diverse N-best list than AC OdB, which is same for Babble_ 10dB and AC_10dB. In particular, the highly intelligible clean speech yields no N-best diversity. Fig. 6 visualize the language noise that originates from different audio noise, where the distances between clusters well represent the noise levels of source speech.
present the N-best hypotheses from speech
In summary, the relationship between the audio noise in source speech and the language noise in decoded N-best list inspires us to propose language-space denoising. Fortunately, the powerful generation ability of LLMs promotes the success of this research idea.
E LIMITATIONS
Though effective in improving noisy ASR performance, there still exist some limitations in the proposed RobustGER.
Table 16 presents a failure case CHiME-4 dev-real set. There is one error in N-best hypotheses, i.e., the word "Miss" that should be "Ms" in ground truth. The GER baseline successfully corrects this error while RobustGER fails. The reason could be, the words "Ms" (/miz/) and "Miss" (/mus/) sound similar especially under noisy scenarios, GER cannot distinguish them rich linguistic knowledge and powerful reasoning ability, LLMs enable GER to generate the correct word "Ms" that is more appropriate than "Miss" in this context. On the other hand, with the proposed language-space denoising, RobustGER successfully perceives the trivial difference between their pronunciations but find the word is more likely to be "Miss" (e.g., maybe the speaker's 's pronunciation is not standard), Such information misleads LLMs to generate the wrong word. Therefore, this is a problem of trade-off between contextual information and denoising for LLMs to generate correct transcription: 1) when both homophones suit the context, LLMS should carefully in denoising to find the correct word (see Table 5). 2) when one of homophones is obviously more suitable to the context than another one, LLMs may not need denoising as it could provide misleading information. We believe this could be a promising research direction for future work on GER.
depends on L.LMs to decide based on context. Thanks the
We observe from main results in Table 1 that both GER and our RobustGER achieves significantly more improvements on CHiME-4 dataset than other datasets. This phenomenon has been also observed and analyzed in the original GER benchmark (Chen et al. 2023b), as there are many financial terminologies in the transcriptions of CHiME-4 that are relatively easier for LLMs to correct. Therefore, in future may need analysis of error types for CHiME-4 to understand how RobustGER works there.
After our initial draft was released on OpenReview in October 2023, we also learned that there have been recent developments in post-recognition text modeling, as well as LLM based efforts in audio understanding (Gong et al.) 2023a Wu et al.) (023b) and speaker diarization (Park et al. 2023 Wang al. 2024).
We hope to align the efforts of different
24

---

Published as a conference paper at ICLR 2024
Table 16: Failure case of RobustGER, The test sample from CHiME-4 dev-real dataset with ID as "M03.052CO1ORBBUS"
Method
Utterance
WER (%)
miss amsterdam declined to comment
20.0
miss amsterdam declined to comment
20.0
N-best Candidates
ms amsterdam declined to comment
0.0
miss amsterdam declined comment
20.0
miss amsterdam decline to comment
40.0
GER
ms amsterdam declined to comment
0.0
RobustGER
miss amsterdam declined to comment
20.0
Ground Truth
ms amsterdam declined to comment
Table 17: Distances between the language noise embeddings from clean and different noisy conditions. The corresponding t-SNE visualizations are presented in Fig.
Clean vs.
ac
babble
cafe
car
metro
traffic
avg.
Language Noise Emb.
59.7
54.9
32.4
12.7
19.1
17.4
32.7
Audio Distillation
57.6
87.5
53.2
37.5
32.1
51.8
53.3
research groups to enable more robust and resilient text modeling evaluations for various speech and audio processing tasks the future, as part a collaborative community effort.
25


Reviews:
[{"review_id": "txipfBRm01", "reviewer": "Reviewer_BbAN", "summary": "This paper addresses the noise-robust ASR. The method is based on the LLM-based generative error correction (GER), but contrary to the existing approaches, it extract the noise information from the N-best hypotheses of transcription languages, not directly from the audio. The proposed method significantly outperformed the existing baseline and advanced the area of noise-robust ASR.", "questions": "- The notation seems to be mixed up. $\\mathcal{P}$ may be used in the sense of probability density function in equation (2), but this is also used to mean \"prompt\". In the definition of KL, the probability distribution is denoted as $\\mathbb{P}$. \n- I don't feel the need to use too much fancy notation like tensor product $\\mathbb{P}_X \\otimes \\mathbb{P}_Z$ and Radon-Nikodym derivative $\\log \\frac{d\\mathbb{P}}{d\\mathbb{Q}}$.\n- It may be just because I am conservative but $1e^{-2}$ looks like $1/\\exp(2)$.\n- $\\mathbb{E}_p p \\log p$ looks strange in Table 15.\n- $\\textit{i.e.}$ should not be italicized in standard writing convention.", "limitations": "", "rating": 8, "confidence": 3, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "Originality:\nWhile it would normally be better to reduce the noise before ASR or to estimate the noise directly from the acoustic data, the idea of doing noise estimation from N-best transcription hypotheses is interesting. It is interesting that this method avoids the difficulty of cross-modal fine-tuning by doing so. It is also compelling because it seems that humans actually perform similar processing in noisy environments.\n\nQuality:\nThe paper is rich in evaluation, and the improvement interval is significant compared with the existing baselines. Also, it provides theoretical hypotheses why the proposed method works better that sound convincing.\n\nClarity:\nThe description appears complete. I have not read all the details, but I get the impression that this paper is very well organized.\n\nSignificance:\nThe task addressed is clearly significant because it has many practical applications. The novel approach presented in this paper is also be interesting, and I think it can potentially be applied in other modalities.", "weaknesses": "- It was very difficult to find anything to explicitly criticize about the technical content of the paper. There may be flaws and room for improvement, but that is no longer something to do at the peer review stage of this paper.\n- If I had to say something, I was concerned that the notation seemed sometimes inconsistent.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "T1qsgmrJWC", "reviewer": "Reviewer_ZWfm", "summary": "This work addresses noisy conditions by proposing a language-space noise embedding derived from ASR hypotheses to aid denoising with large language models (LLMs)-based generative error correction (GER). A knowledge distillation strategy further enhances noise representation. Tests on various LLMs show up to 53.9% improvement in word error rate with limited training data, demonstrating the effectiveness of the proposed noise embedding and denoising ability of LLMs.", "questions": "see above", "limitations": "", "rating": 10, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 4, "strengths": "From a generally purposed GER benchmark to a more focused noise-robust problem, it is a suitable extension in depth and kind of milestone using LLM for robust ASR.", "weaknesses": "The illustrations of (b) GER with audio-space denoising (Zhang et al., 2023b; Fathullah et al., 2023) and (c) GER with language-space denoising (ours) are a little challenging to follow. Is denoised audio directly fed to the LLM adapter, or is there something else you want to express?\n\nWe noticed that the HP database only comes from the n-best of a few models. Is it possible to introduce more diverse system outputs from various models?\n\nAccording to the method in the article, Section 4.3 should be the most important part, relatively speaking. Unfortunately, the space allocated to it in the article is too cramped—too many things to fit into this small section, which is not very reader-friendly.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "QzpBM6QEwz", "reviewer": "Reviewer_QKKW", "summary": "This work extends an established benchmark of generative error correction with a new \"HyPoradise\" dataset, in order to enable LLMs to perform error correction. The study presents application on noisy-robust speech recognition and claims that it reaches up to 53.9% improvement on word error rate.\n\nThe study itself follows the latest trend of research, where LLM is used to", "questions": "1. I wonder the motivation of using Robust Hyporadise dataset for noisy ASR condition. What kind of noise it exactly contains? Is it replacible with other noisy datasets that are more commonly known to the ASR community, such as Switchboard and VoxCeleb (just two examples, they may not be good fit)?\n2. Do you think your model will be sensitive to sampling frequency? I mentioned Switchboard in the last question, which is an 8KHz dataset.\n3. In section 4.2.2, why you think MINE can enhance the noise representation ability? It looks like MINE is not part of novelty here, so any work backing it up?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "1. The work itself holds certain level of novelty, with good review on earlier literatures on both error correction in ASR and LLM.\n2. The methodology is clearly presented, along with the novelty of the paper.\n3. With some ambiguities in the middle, the paper itself clarifies the idea with experiments subtlely.", "weaknesses": "1. I think the topic of error correction might be a poor fit to the conference. But perhaps I am wrong on this so correct me if so.\n2. There lacks the practical discussion on additional workload, especially on resources.\n3. The description of building the embedding space is somehow confusing in particular terms. For example, in Section 4.2.1 - what is \"diversity similar to variance\"?\n\nMinor issues:\n1. Section 5.4 - What is Table 14?\n2. I suggest to put the definition of embedding a bit earlier from the beginning of Section 4. Otherwise, Figure 2 looks a bit confusing.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "vgYXmOA4qN", "reviewer": "Reviewer_nSVq", "summary": "In a previous study, generative error correction (GER) is achieved by learning the mapping from ASR N-best hypotheses to ground-truth transcription through efficient LLM finetuning. This paper extends this idea and focuses on noisy conditions. To avoid the cross-modality gap, the authors propose a novel idea to extract a language-space noise embedding from the N-best list to represent the noise conditions of source speech. Furthermore, in order to enhance its representation ability of audio noise, a knowledge distillation (KD) approach via mutual information estimation (MINE) is employed. The experiments show that the proposed method can significantly outperform the conventional LM rescoring baseline. Several additional experiments are also included in the Appendix which provide more insight into the proposed RobustGER. Overall, the paper is very clear and well-written. It describes the problem and explains the solution well. The experiments done are reflective of the proposed model's performance.", "questions": "1)\tAs pointed out on page 5, the noise embedding is calculated by their diversity “similar to variance”, however in eq (4) and (6), the sentence embedding differences are simply summed, so I guess an abs or square operation is needed?\n2)\tFollowing the previous question, in the appendix page 16, you mentioned that the dimension of language-space noise embedding E_LN is N(N-1)xD_sbert. Could you explain where N(N-1) comes from? I cannot see this dimension from eq (4) and (6).\n3)\tIn figure 2, and eq (3), why the language-space noise embedding is ‘subtracted’ from the prompt? I found another related equation in eq (13) of the Appendix and the authors only mention “the subtraction operation denotes “denoise””, more explanation is needed.\n4)\tIn eq(8), IΘ(X; Y ) should be IΘ(X; Z)", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 4, "strengths": "1)\tNovel idea to apply LLM for noise-robust ASR.\n2)\tExtract language-space noise embedding with knowledge distillation based on mutual information.\n3)\tGood performance improvement.\n4)\tPlenty of experiments and ablation studies.\n5)\tInsightful discussions, such as t-SNE visualization, and the relationship between noisy speech and n-best list diversity.", "weaknesses": "The way to extract the audio noise embedding is from the ASR encoder (i.e., Whisper Large-V2). This may only make sense for the Whisper ASR, as a recent paper [r1] pointed out that the noise-robustness of Whisper does not come from noise-invariant, but recognizes speech conditioned on the noise type. In summary, the Whisper encoder is a suitable model to extract noise information. On the other hand, other ASR models may not have such ability and they achieve noise-robustness through the noise-invariant encoder. If this is the case, those ASR encoders may not be suitable to extract audio noise embedding. A discussion and simple experiment about this would be great.\n\n\n[r1] Gong, Y., Khurana, S., Karlinsky, L., & Glass, J. (2023). Whisper-at: Noise-robust automatic speech recognizers are also strong general audio event taggers. arXiv preprint arXiv:2307.03183.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (spotlight)