HyPoradise: An Open Baseline for Generative Speech Recognition with Large Language Models
Authors: CHEN CHEN
Yuchen Hu
Chao-Han Huck Yang
Sabato Marco Siniscalchi
Pin-Yu Chen
EngSiong Chng
Venue: NeurIPS 2023 Datasets and Benchmarks Poster (2023)
Keywords: Automatic speech recognition
language model rescoring
N-best Hypothesis list.
TL;DR: We introduce the first open-source benchmark leveraging large language models (LLMs) for generative speech recognition error correction, presenting a new dataset, "HyPoradise."

Abstract: Advancements in deep neural networks have allowed automatic speech recognition (ASR) systems to attain human parity on several publicly available clean speech datasets. However, even state-of-the-art ASR systems experience performance degradation when confronted with adverse conditions, as a well-trained acoustic model is sensitive to variations in the speech domain, e.g., background noise. Intuitively, humans address this issue by relying on their linguistic knowledge: the meaning of ambiguous spoken terms is usually inferred from contextual cues thereby reducing the dependency on the auditory system. Inspired by this observation, we introduce the first open-source benchmark to utilize external large language models (LLMs) for ASR error correction, where N-best decoding hypotheses provide informative elements for true transcription prediction. This approach is a paradigm shift from the traditional language model rescoring strategy that can only select one candidate hypothesis as output transcription. The proposed benchmark contains a novel dataset, "HyPoradise" (HP), encompassing more than 316,000 pairs of N-best hypotheses and corresponding accurate transcriptions across prevalent speech domains. Given this dataset, we examine three types of error correction techniques based on LLMs with varying amounts of labeled hypotheses-transcription pairs, which gains significant word error rate (WER) reduction. Experimental evidence demonstrates the proposed technique achieves a breakthrough by surpassing the upper bound of traditional re-ranking based methods. More surprisingly, LLM with reasonable prompt design can even correct those tokens that are missing in N-best list. We make our results publicly accessible for reproducible pipelines with released pre-trained models, thus providing a new paradigm for ASR error correction with LLMs.

Hy PoPoreiie:: An Open Baseline for Generative Speech Recognition with Large Language Models
Chen Chen Yuchen Hult
Chao-Han Huck Yang2
Sabato Marco Siniscalchi3.4 Pin-Yu Yu Chen Eng Siong Chng²
Nanyang Technological University Georgia Institute of Technology SNVIDIA Research Norwegian University of Science and Technology 'IBM Research Al
{ {chen1436, yuchen005) @e. ntu. edu. hucklyangogatech.co edu
Abstract
Advancements in deep neural networks have allowed automatic speech recognition (ASR) systems attain human parity on several publicly available clean speech datasets. However, even state-of-the-art ASR systems experience performance degradation when confronted with adverse conditions, as a well-trained acous- tic model is sensitive to variations in the speech domain, e.g., background noise. Intuitively, humans address this issue by relying on their linguistic knowledge: the meaning of ambiguous spoken terms is usually inferred from contextual cues thereby reducing the dependency on the auditory system. Inspired by this observa- tion, we introduce the first open-source benchmark to utilize external large language models (LLMs) for ASR error correction, where N-best decoding hypotheses pro- vide informative elements for true transcription prediction. This approach is paradigm shift from the traditional language model rescoring strategy that can only select one candidate hypothesis as the output transcription. The proposed benchmark contains a novel dataset, "HyPoradise" (HP), encompassing more than 334,000 pairs of N-best hypotheses and corresponding accurate transcriptions across prevalent speech domains. Given this dataset, we examine three types eгror correction techniques based on LLMs with varying amounts of labeled hypotheses-transsciitoon pairs, which gains a significant word error rate (WER) reduction. Experimental evidence demonstrates the proposed technique achieves a breakthrough by surpassing the upper bound of traditional re- re-ranking based meth- ods. More surprisingly, LLM with reasonable prompt and its generative capability can even correct those tokens that are missing in N-best list. We make our results publicly accessible for reproducible pipelines with released pre-trained models, thus providing new evaluation paradigm for ASR error correction with L.LMs.
Introduction
Automatic speech recognition (ASR) has become increasingly important in modern society, it
enables efficient and accurate transcription of spoken languages. This capability facilitates access to
information and enhances communication across various domains, including education [7]. health-
care [50], and business [36]. Driven by the recent advances in deep learning, remarkable success has
been achieved several ASR tasks through end-to-end training techniques [28] 27 9 22
30 100 15).
However, a major challenge of applying ASR in practical conditions lies in effectively handling
variations in speech caused by different factors such as background noise [T speaker accent [85
and speaking styles [82] 2 These adverse factors are common and inevitable in speech signal, significantly affecting the accuracy of the recognition results (55).
Work done & open source while the author was Georgia Tech; Corresponding author. tEqual contribution.
37th Conference on Neural Information Processing Systems (NeurIPS 2023) Track on Datasets and Benchmarks.

---

Case (i): n better than 1"
Case (ii): : T2 n
Audio Input
Transcription Output
0.14
0.34
Exp:
EXP:
LM
T1
22
T3
T4
Ts
1
T1
22
33 13
74
75 45
st Utterance
th
T1
22
Pg
n
T1
44
2dd Uttannee
Prob.
Prob.
ASR
N Utterance
N-best Hypothesis
nd
20
Jsterances
2nd 20 Utterances
Figure I: The left part shows the pipeline generate the N-best hypotheses using vanilla ASR engine with beam search decoding. The right part counts the probabilities of case (i) and case (ii) on the test of LibriSpeech dataset. It indicates the discarded information in 2nd ~ 20 utterances. Green and red T3 in "Exp" respectively denote correct and wrong tokens compared with ground-truth.
Humans demonstrate remarkable robustness when faced with the above variations in acoustic environ- ment, the human recognition system does not only rely acoustic cues usually speculate the ambiguous or distorted spoken terms based on speech context and our inherent linguistic knowledge. Similarly, current ASR system typically employs an independent language model (LM) for rescoring during the decoding process 83 by an ASR engine with beam search decoding, trained language model (LM) can be used to re-score each utterance and select the one with the highest likelihood (referred to as the 1s utterance) as the
25 As shown in Fig. | given N-best hypotheses generated
output of the ASR; whereas, the other sentences (the 2nd Noh utterances) discarded. However, is widely believed [68] that the N-best list contains useful information [87 [66. as each hypothesis an independent textual representation of the input speech. Consequently, discarded sentences might also carry correct tokens for accurately predicting the true transcription. To validate this belief, we have conducted experiments on the LibriSpeech dataset (66). counting the probabilities of two scenarios observed during LM rescoring: (i) the discarded utterances contain a better candidate with lower word error rate (WER), and (ii) the other discarded hypotheses can provide the right answer for the wrong tokens in 1s utterance. The statistical results of 2nd ~ 20th utterances are shown in the left part of Fig. 1 Taking 2nd discarded utterance as example, it has 14% probability of having a lower WER than the 1s utterance. Furthermore, given a wrong token in 1s utterance, there is 34% probability of finding the correct token the 2nd utterance.
To better mine the information in N-best hypotheses, we propose the first attempt on publicly available ASR generative error correction benchmark that directly predicts true transcription, rather than selecting candidate from the N-best list. To put forth this benchmark, we introduce a novel dataset named HyPoradise (HP), which comprises various open source N-best hypotheses provided by tftt-oof-the-rr ASR systems and their paired true transcriptions. Considering real-life applications, HP dataset covers various challenging speech domains, including scenarios with background noise, specific contexts, and speaker accents. Furthermore, in terms of resources availability, we define three settings mimic the deployment of ASR systems in real-world scenarios: (i) Zero-shot Learning. In this setting, only test set hypotheses are available for inference. This corresponds applying well-trained ASR model to new scenarios without any training data. (ii) Few-shot Learning. few in-domain hypotheses with true transcription are available for training. This setting aims to address Nomain-speciie ASR tasks with a few manual annotations. (iii Fine-tuning. A sufficient training set is available to learn the mapping between hypotheses and transcription.
To exploit the three aforementioned scenarios, we present multiple error correction techniques using large language models (LLMs), which have shown the outperforming ability of language generation and reasoning in recent studies [5] 107 48 844. For seot and few-shot settings, we design an in-context learning method without any parameter tuning, which directly performs error correction based on task prompt and in-domain demonstration. In the fine-tuning scenario, we develop two training solutions, H2T-ft and H2T-LoRA, which adapt pre-trained L.I.ss to specific transcription domains. Experimental results show that all learning strategies can be beneficial to reduce the WER in different resource settings, providing potential solutions for alleviating the

---

negative impact of speech variation. Additionally, with reasonable prompt design, LLMs can correct those specious tokens that are exclusive from N-best list. We will release the HP datasets, reproducible pipelines, and pre-trained models on Github under MIT licence.
Our contribution can be summarized follows:
We propose the first open and reproducible benchmark to evaluate how LLMs can be utilized to enhance ASR results with N-best hypotheses, where a new dataset HyPoradise more than 334K hypotheses- bypothesee-transeripion pairs are collected from the various ASR corpus in most common speech domains.
! with
We develop three ASR correction techniques based on LLMs in different resource settings to directly predict the true transcription from the N-best hypotheses. Experimental results in the fine-tuning setting show that our new approach can surpass a performance upper-bound (e.g., oracle WER from n-best list) of traditional re-ranking based methods.
We introduce an evaluation paradigm of generative error correction for ASR. The acoustic model generates word elements in the hypotheses list; subsequently, LLMs predict accurate transcription utilizing linguistic knowledge and contextual information.
2 Related Work
2.1 ASR Rescoring and Error Correction
In order to improve the linguistic acceptability of ASR results, LM rescoring has been widely employed and achieved stable performance gain for ASR systems 77 62 Typically, an external
LM is trained separately and utilized to re-score the N-best list of hypotheses generated by ASR
decoding with beam search. Various approaches for LM integration have been proposed. such as
shallow fusion [17 104 46 83). deliberation [98] 32
41
40 91 39,, component fusion [76]. and
cold fusion 81. Some authors have used pre-trained LM models to replace trainable I.Ms [86] 74], and the log- ligeliieldoodd of each hypothesis is computed using unidirectional models, e.g.. GPT-2, or pseudo-log souud--lo-iikellhoodd likelihood using bidirectional models like BERT 21 and RoBERTa [59]. In ASR, LMs
are also widely used for the error correction task in different languages 996 29,. leveraging only the
23 [09 77.. Furthermore, more recent works [60 52 51] utilize candidates list after decoding for error correction. Though Grammatical Error Correction (GEC) has been actively explored (20, 93 100), ASR error correction is distinct
1-best hypothesis generated by the ASR model [53
6 106
with GER due to the arbitrariness of the spoken language 2 which requires the efforts from both speech and NLP communities [18].
2.2 Large Language Models
More recently, there has been a surge of interest in Transformer- LLMs 84 70 75 107 in both academia and industry. By learning from massive amounts of text data, LLMS can capture linguistic patterns and semantic relationships, which have led to impressive performance for a wide range of
natural language processing (NLP) tasks 55 65 (55.
In-context Learning. Given specific task descriptions or pair-wise contextual information, LLMs show outstanding adaptability on downstream NLP tasks without any parameter tuning 63 64
[00]
Such capability of task-specific inference is also known as in-context learning (ICL) 99 which utilize LLMs to generate text that more coherent and relevant the specific domain or task
44 16
49, 73, 8 [08]. Recently, task- ausk-activating Prompting (TAP) [10) is one of the most relevant works, employing the injection of input-output pairs of trientrie contexts (e.g., initiating the question prompt from a broad domain to refine preceding contexts as shown in Figure 2) with the aim of enhancing the zero-shot and few-shot capabilities of frozen-pretrained LLMSs for second-pas pass ASR. We further evaluate the TAP-based zero-shot and few-shot approaches with examples.
Low-rank Approximation based Neural Adapter. Tuning all LLM parameters for given down- stream task is usually not feasible due to memory constraints. Many researchers sought to mitigate that problem by either adapting only few parameters or leveraging external trainable modules for
ihttpa://githul com homm/Hyyotheses-e Sttpppoheeee-Paaadis//ttttpooop Paradisee/ypooorran
"Denoted Hypotheses Paradise, inspired by "Icha Icha Paradise" from Naruto.
3

---

new task [58 33 A pioneer work [ showed that the learned over parametrriedd models in fact reside on low intrinsic dimension, consequently, a low-rank adaptation (LoRA) approach [38] was proposed to indirectly tune some dense layers by optimizing rank decomposition matrices of the dense layers. Due to its computational efficiency, LoRA adaptation has been rapidly adopted as
new paradigm for LLMs tuning, which was useful in various downstream tasks [105] 24 92
3 Hypothesis Generation and Dataset Creation
We introduce the generation process of the HyPoradise dataset in this section. The employed ASR system for N-best hypotheses generation illustrated in 3.1 and then we introduce the selected speech domain in 3.2 Finally, we provide statistic information and generated HP in 3.2
3.1 ASR System
We employ two state-of-the-art ASR models, namely WavLM [14] and Whisper 66 for N-best hypotheses generation. Besides their remarkable performance and popularity, those models are representative in the deployment of an ASR because: (1) WavLM is a well-trained ASR model on LibriSpeech [66] but suffering from domain mismatch, and (2) Whisper is a universal ASR model but lacking domain specificity. More details about those two ASR models are described below:
WavLM: We utilize the ESPnet toolkit (94] along with the pre-trained model from HuggingFace to deploy our WavLMM-asse ASR system. The WavLM architecture consists of two blocks: the front-end, and the ASR model (433 million parameters in total). The front-end consists of 24
Transformer-based 88 encoder layers and is pre-trained using combination of LibriLighht [45]
(60k hours of data), Gigaspeech [12] (10k hours of data), and VoxPopuli [90] (24k hours of data). Front-end features are fed into the ASR back-end for fine-tuning. The back-end consists of 12 Conformer- based (30] encoder layers. and 6 Transformer-basee decoder layers. The fine-tuning process performed on 960-hour LibriSpeech data. Additionally, the WavLM decoding recipe incorporates an external LM rescoring option, where the external LM adopts Transformer architecture with 16 encoder layers and is trained using the text of LibriSpeech 960 hours data and extra LM training data from the web.
Whisper: We employ the Whisper-Large model developed by OpenAl to generate hypotheses, without in-domain language model rescoring. The used configuration consists of an encoder-decoder Transformer architecture with 1,550 million parameters, which is trained on 680,000 hours of mulilingual-weakly labeled speech data collected from the web.
Leveraging these two pre-trained ASR models, we have employed the beam search algorithm during decoding and generated Nebest lists of sentence hypotheses for each input waveform. For both WavLM and Whisper, the default beam size was set 60. After removing repeatable utterances, select t utterances with highest probabilities as N-best list, as they have carried sufficient elements to accurately predict transcription. Subsequent experiments confirm this belief by calculating the accurately upper-bound WER using 5-hest hypotheses list. To build the HP dataset, we carry out this decoding strategy multiple popular ASR datasets (please Section 3.2p and generate paired data consisting of an 5-best hypotheses list and I ground-truth transcription. The pre-processing and generation code are also released for integrating new ASR corpus into HP. All the links of relevant resources are presented in Appendix.
3.2 Selected Speech Corpora
For corpora selection, our goal to cover common scenarios of ASR task, e.g., noisy background and speaker accent. Consequently, we collect and modify the following corpora with evident domain characteristics to compose the HP dataset.
LibriSpeech is public corpus of read speech from audiobooks, including 1,000 hours of speech data with diverse speakers, genders, and accents. For generating HP training data, exclude some simple cases from its train-960 split that show WER result of O, resulting in 88,200 training utterances. We the entire lest-clean and lest-other splits for HP test data generation.
LibriSpeech 666:
CHiME-4 [89] CHiME-4 a dataset for far-field speech recognition. It includes real and simulated noisy recordings in four noisy environments, i.e., bus, cafe, pedestrian area, and street junction. We

---

Table 1: HP dataset statistics in terms of the number of transtissestraanscriiiiooo pairs and average utterance length in various domains.
Domain
Training Set
Pairs
Length
Test Set
# Pairs
Length
Source
Category
test-clean test-other
2,620
20.1
I LibriSpeech
Audiobooks
train-960
88,200
33.7
2,939
17.8
CHiMP4
Noise
train
8,738
17.0
test-real
1,320
16.4
dev93 eval92
503
16.7
WSJ
Business news
train-si284
37,514
17.5
333
17.3
SwitchBoard
Telephone
Irain
36,539
11.8
eval2000
2,000
11.8
Common Voice
Accented English
train-accent
49,758
10.5
test-accent
2,000
10.5
Tedlium-3
TED talk
train 47,500
12.6
test 2,500
12.6
LRS2
BBC audio
main
42,940
7.6
test
2,259
7.6
ATIS
Airline info.
train 3,964
12.4
test
809
11.3
CORAAL
Interview
train
1,728
24.2
test
100
24.0
Total
train
316,881
18.1
Test
17,383
its train (with 8,738 utterances) and lest-real (with 1,320 utterances) splits generate HP training and test data. The four different noises in test-real split are also evaluated separately in Table 3
WSJ [67] The Wall Street Journal (WSJ) is widely-used benchmark for speech recognition. It includes read speech from speakers in a controlled environment, with focus on business news and financial data. We use its train-si284 split (with 37,514 utterances) generate HP training set. The dev93 (with 503 utterances) and eval92 (with 333 utterances) are applied to build test sets.
The SwitchBoard corpus is telephone speech dataset collected from conver- sations between pairs of speakers. It focuses on North American English and involves over 2.4k conversations from approximately 200 speakers. We randomly select 36,539 samples from its train split to generate HP training set, as well as 2,000 utterances from the eval2000 split for HP test set.
SwitchBoard 26]
Common Voice 3 Common Voice 5.1 a freely-available dataset for speech recognition. It contains speech recordings from diverse speakers in over 60 languages. To generate HP dataset, we randomly select 51,758 samples from its train-en split with accent labels, i.e. African, Australian, Indian, and Singaporean, where training contains 49,758 samples and test set contains 2,000 samples.
Tedlium-3 35 Tedlium-3 is a dataset of speech recorded from TED Talks multiple languages. contains a diverse range of background noise, speaker accents, speech topics, etc. Considering its large size, we randomly select 50,000 samples from its train split for HP dataset generation, where training set contains 47,500 samples and test contains 2,500 samples.
Lip Reading Sentences (LRS2) a large-scale publicly available labeled audio-visual dataset, consisting of 224 hours of video clips from BBC programs. We randomly select 42,940 samples from its train split as training set, and the remaining 2,259 samples are used for test set.
LRS2 19
ATIS 34 Airline Travel Information System (ATIS) is a dataset comprising spoken queries for air travel information, such as flight times, prices, and availability. contains around 5,000 to 5,400 utterances. which are recorded from around 500 to 550 speakers.
The Corpus of Regional African American Language (CORAAL) is the first public corpus of AAL data. includes audio recordings along with the time-aligned orthographic transcription from over 150 sociolinguistie interviews. To generate HP dataset, select 1,728 samples as training set and 100 samples as test set.
CORAAL [7
3.3 HyPoradise (HP) Dataset Statistics
After performing beam search decoding on the selected speech datasets introduced in Section 3.2 we collected more than 334K pairs of hypotheses list and transcription form the HP dataset, including training and test sets. The statistics for the HP dataset given in Table I which shows the number
5

---

you automatic speech recognition:
Self-activated In-context Learning
R: Yes, ASR is [...]
you know language model rescoring AS8?
R: Suze, IM rescoring [...]
provide [CLS]
Q: Can you genezate example with 5-best list? 5-best hypothesis list: [...], and answer [...]
The N-best hypothesis list is: [...l,
the true transcription 1s: [...]
Please do
thing following N-best list:
demonstration
[8-best hypothesis]
course.
transcription abould be: [...]
Figure 2: A scalable evaluation of Tassk-Activating Prompting 100 (TAP) based in-context learning.
The demonstration in blue box is drawn from the training set, which is optional for LLMs input.
Hidden
Predicted transcription
K
B=0
Pre-trained Weights
(a)
Correction Model
(b)
A= N
K
Prompt N-best hypotheses list
Input
Figure 3: (a) Structure of H2T-ft. (b) Reparametrizatio in H2T-LoRA. Solid box denotes the module fixed during tuning while dashed box stands for trainable. Blue color denotes the weights has been pre-trained on another dataset.
of pairs and average length in various domains and splits. We would release our generated datasets and kindly call for more pppotheses-transeripppe pairs toward sustainable community efforts.
ASR Error Correction from Hypotheses to Transcription
We hereby introduce pppotheses-to-transcrppptt (H2T) training scheme utilizing the collected HP dataset to enhance ASR performance with LLM integration. With limited labeled data, in-context learning [100] is employed to form specific prompts and in-domain demonstrations: Linguistic knowledge in LLM is exploited without parameter tuning. Furthermore, present two trainable methods fine-tuning (ft) and H2T-LoRA to learn the hypotheses- torraascriiptiiii mapping when a sufficient amount of labeled data is available.
4.1 Hypotheses-tt-TTrassrippp (H2T) Training
In addition to in-context learning, we introduce two parameter-tunabl methods to learn hypotheses- to-transcription mapping
manner: H2T-ft and H2T-/oRA.
H2T-fft denotes fine-tuning all parameters of a neural model with labeled data of each HP domain. Specifically, we introduce a similar method with N-best T5, which utilizes other hypotheses to improve the 1-best hypothesis as shown in Fig. 3 To constrain the decoding space, we add an
new item criterion Lft Σn a;; log (zx Jar, 0), where (x()) is the i-th hypothesis in N-best list. This item aims to encourage the correction model to preferentially consider tokens into the N-best hypotheses list, preventing arbitrary modification in huge decoding space. a; is for i-th hypothesis that decreases with the order ranked by the acoustic model.
H2T-LoRA avoids tuning the whole set of parameters of a pre-trained model by inserting neural module with a small number of extra trainable parameters to approximate the full parameter updates, allowing for efficient learning of the H2T mapping without affecting the pre-trained parameters of the LLM. H2T-LoRA introduces trainable low-rank decomposition matrices into LLMs' existing layers, enabling the model to adapt to new data while keeping the original LLMMs fixed to retain the previous knowledge. Specifically, LoRA performs a reparameterization of each model layer expressed as matrix multiplication by injecting low-rank decomposition matrices (Fig 3 (b)). As result, the

---

Table 2: WER (%) results of H2T-ft and H2T-LoRA in finee-tuning setting. "Onb" and "Oep" respectively denote n-best oracle and compositional oracle that are defined in 5.2
H2T-ft
H2T-LORA
Oracle
Test Set
Baseline
LM
T5
LLaMA
T5
LLaMA
Ocp
WSJ
4.5
4.3
4.0
3.8
2.7 40.0%
2,2 -51.1%
4.1
1.2
ATIS
8.3
6.9
2.7
3.4
1.7 79.5%
1.9 -77.1%
5.2
1.1
CHiME-4
11.1
11.0
7.9
8.2
7.0 36.9%
6.6 40.5%
9.1
2.8
Tedlium-3
8.5
8.0
6.6
5.2
7.4-12.9%
4.6 15.9%
3.0
0.7
CV-accent
14.8
16.0
12.9
15.5
11.0-25.7%
11.0 25.7%
11.4
7.9
SwitchBoard
15.7
15.4
15.9
18.4
14.9 5.1%
14.1 10.2%
12.6
4.2
LRS2
10.1
9.6
9.5
10.2
6.6 34.7%
8.8 12.9%
6.9
2.6
CORAAL
21.4
21.4
23.1
22.9
20.9-2.3% 2.3%
19.2 10.3%
21.8
10.7
representations generated by the LLM are distorted due to task-specific tuning, while the adapter module acquires the capability to predict the true transcription from the N-best hypotheses.
Benefiting from efficient training, method, which is expected to understand the task deseription and capture correlation in the N-best list. Meanwhile, instead of adding extra training objective in H2T-ft, constrain the decoding space of H2T-LoRA by adding requirement in task description.
employ large-scale language model in the H2T-LoRA
Experimental Results
5.1 Language Models Configurations
TS (0.75B~3B): T5 family [72 is of encoder-decoder models pre-trained multi-task mixture unsupervised and supervised tasks and for which each task is converted into a text-to-text format. T5 works well variety of tasks out-of-the-box by prepending different prefix to the input corresponding to each task, c.g., for machine translation or text summarization. In this paper, we select TS-large (0.75B) as the correction model in H2T-fi method.
is a collection of foundation language models ranging from 7B, 13B, 30B, and 65B parameters. It trained on publicly available datasets exclusively, and shows remarkable efficiency on NLP benchmarks. We select LLaMA-13B for LoRA adaptation in H2T-LoRA method one best setup under ablations.
LLaMA (7B~65B): Proposed by Meta AI, LLaMA 84
GPT-3.5 (175B): Proposed by OpenAT, GPT-3.5-turoo one the most advanced large language models, which powers the popular ChatGPT. It has been optimized from the GPT-3 (5) for chat purposes but works well for traditional completions tasks as well. We utilize GPT-3.5-tun in task-activated in-context learning [100), which conduct zero-shot and few-shot learning experiments with designed task prompt.
5.2 Training and Evaluation
For few-shot settings, the specific task prompts, along with the LLM's responses from task-activated ICL prompting [100, are provided in the Appendix (page 20). For fine-tunin setting, the detailed configuration of H2T-ft and H2T-LoRA are also explained in Appendix. Furthermore, release some of the pre-trate preitreined correction models to allow interested readers to reproduce our results.
We report WER results the evaluation metric for all methods. Additionally, we report the two oracle WER for comparison, which are 1) the n-best oracle oыg: WER of the "best candidate" in N-best hypotheses list, and 2) the compositional oracle method Orp: achievable WER using "all tokens" in N-best hypotheses list. The on can be viewed as upper bound performance the re-rank based method, while Ocp denotes the upper bound of correction using occurred elements in the list.

---

Table 3: Cross-domain WER results by task-activated ICL [0O] in zero-shot and few-shot settings. "Onb" and "Oep" respectively denote n-best oracle and compositional oracle that are defindd in 5.2
Domain Shift
nhshot In Dn-Conexxt Learning (ICL)
Oracle
Test Set
Baseline
n
n ==
10
Onb
Oup
Specific Scenario
WSI-dev93
9.0
8.5 5.66
7.8 13.3%
7.7 14.4%
7.1 21.1%
6.5
5.3
WSI-eval92
7.6
7.3 -3.9%
6.6
13.2%
6.6-13.28 13.2%
6.3-17.1%
5.5
4.7
ATIS
5.8
5.5 5.5-5.2%
5.1- 12.1%
5.0 13.8%
4.7 4.7-19.0%
3.5
2.4
CHiME4-bus
18.8
17.6 17.6-6.4% 6.4%
16.7 11.2%
16.2
13.8%
15.9
16.8
10.7
Common
CHiME4-cof
16.1
14.7 1..7%
14.3-11.155 11.1%%
13.7
1177-11.000 14.9%
13.2 18.0% 0%
13.3
9.1
Noise
CHiME4-ped
11.5
10.9
14.4%
9.7- 15.7%
9.4 155-18.99
8.5
5.5
CHiMH4-str
11.4
10.9 4..4%
10.0
12.3%
9.7 14.0%
9.2 19.39
9.0
6.0
CV-af
25.3
2.69%
24.2 24.2-4.3%%
23.6 6.7%
22.6-10.7%
23.6
21.7
Speaker Accent
CV-au
25.8
25. 1 12.7%
24.1 88.0%
24.0
24.0__7.0%
23.3 99.7%
24.9
21.8
CV-in
28.6
27.6 27.6-
25.6 10.3%
25.0 12.66%
21.4 14.7%
27.1
22.6
CV-sg
26.4
26.540.495
25.0 5.3%
25.1 4.0%
23. 10.2%
25.5
22.2
Table 4: Case study of ICL. The utterance is drawn from WSJ-dev93 dataset.
Type
Utterance
WER
1st Hypo.
Bankers in Hong Kong expect xinnepec to return for more loans as it develops China's petro chemical industry.
16.7
by AM
2nd Hypo. by AM
Bankers in Hong Kong expect xinepec return for more loans as it develops China's petrochemical industry.
8.3
Correction by LLM
Bankers in Hong Kong expect Sinopec to return for more loans as it develops China's petrochemical industry.
Ground-truth Transcription
Bankers in Hong Kong expect Sinopec to return for more loans as it develops China's petrochemical industry.
5.3 Results of H2T-ft and H2T-LoRA
We first report the WER results for H2T-ft and H2T-LoRA in the fine-tuning setting, where the training set of HP is available to learn H2T mapping. Whisper is employed acoustic model for hypotheses generation, and vanilla language model LM is trained using in-domain transcription of the training set, and then it re-ranks the hypotheses according to perplexity. From Table we observe that 1) correction techniques achieve significant performance gain in specific scenarios, where H2T-LoRA respectively reduces 77.1% and 55.15 % relative WER on ATIS and WSJ. 2) WER performances on CHiME-4 and CV-accent demonstrate proposed correction methods improves the robustness of on background noise and speaker accent. Additionally, H2T-LoRA on these two datasets both surpass the upper-bound of re-ranking based method referring to ond- 3) In general, H2T-LoRe usually generate better WER results than H2T-ft, as the low-rank adapter allows LLMs to keep pre- pre-trained knowledge and avoid over-fitting problem.
Limitation and Failure Studies. We notice that an over over-fitting phenomenon existing in our correction techniques, especially in H2T-ft where all parameters are tunable. Furthermore. the mean and variance of the utterance length can potentially influence the WER result, since H2T-ft results on CORAAL (long-form speech) and SwitchBoard (large variance in length) both fail enhance ASR performance. On LibriSpeech, when the WER is low (1.8% by WavL.M, there is less room to correct recognition errors with proposed framework. The experimental results and list the representative failure cases can be found in Appendix Bld Table Given the evidence of ample room for further performance improvement, our proposal thus serves as an appropriate benchmark to assess the contribution of current and future LLMs to ASR.
5.4 In-context Learning Results
We conduct in-context learning experiments in the practical scenario when a well-trained ASR system encounters domain mismatch. To this end, the WavLM is selected as the in in-domain acoustic

---

model, and GPT-3.5 serves as the LLM for correction. We mainly consider common domain shifts of application: specific scenario, common background noise, and speaker accent, where 5-best hypotheses are selected as context input. From Table 3 we can observe that: (1) Without any in-domain data, LLM can benefit from ASR results based on the hypotheses list. This performance gain mainly relies on the linguistic knowledge of LLM and task-activating [100] descriptions (e.g., chains of task hints) in pipeline.
(2) A few in-domain pairs effectively enhance the performance gain in terms of WER. From the final output of the reasoning process, we find that LLM attempts to summarize the regulation from the demonstration and then apply it to the given test example. (3) Leveraging the vast knowledge base, LLM can even correct missing tokens that are exclusive from hypotheses list in terms of context information.
To illustrate the third observation, we conduct the case study on WSI-dev93 in Table 4 According to the ground-truth transcription, two errors (shown as red) are included in 1st hypothesis, where "petro chemical" is wrongly recognized as two tokens perhaps due to the speaking style of the speaker. LLM correct this error since "petrochemical" can be found in 2nd hypothesis. However, "Sinopee" is unseen during ASR training, leading it to be recognized as weird tokens ("xinnepec" or "xinepec") in hypotheses. In this case, LLM shows human-like correction - it successfully infers the correct token based on the pronunciation of "xinnepec", as well as the context of "China's petrochemical", In fact, Sinopec is a petrochemical--elatee Chinese company.
5.5
Additional Discussion
Effect on Spoken Language Intent Detection. We examine the effect of error correction on downstream task of spoken intent detection [0) (SID). To this end, we reproduce an BERT-based SID model [13] and respectively feed the 1-best utterance and corrected utterance by H2T-LoRA for comparison. The ablation results on ATIS dataset are reported in Appendix, which shows that our correction technique can also benefit to SID task in terms of detection accuracy. (3) LLM correction based on N-best hypotheses can effectively enhance the downstream SIT result, which achieves comparable accuracy with using ground-truie transcription (97.4% v.s. 97.9%).
Zero-shot Prompting Results. We finally report an initial prompting evaluation on CHiME-4 in zero-shot setting. Considering the task difficulty, T5 and LLaMA are employed for hypothesis correction. For comparison, we also provide the correction results using a far smaller GPT-2 (1.5B) with a 5- 3-gaam LM baseline trained by in-domain transcription. We used LLaMA 13B to perform these zero-shot error correction tasks. Using the test set extracted from Whisper, we observed that the zero-shot method did not yield improved results on CHiME-4 (11.5 + 0.5%) and CV-accent (14.9% + 1.5%). This zero-shot pipeline performed less stably on the other test set discussed in Table 2, which we consider failure case with standard deviation exceeding absolute value of 10% in terms of WER. For T5- based eттor correction, we noticed that the method also failed to perform zero-shot error correction by using 0.75B.
Future work. We find that LLMs potentially perceive acoustic information during pre-training, as they tend to perform error correction using tokens with similar pronunciation. Therefore, our first future work is including more acoustic information in HP dataset, such as token-level confidence provided by ASR engine. Furthermore, considering different data amount of each domain, more parameter-efficient training methods besides low-rank adaptation should be discussed for LLMs tuning 54), e.g., model reprogramming [102 31, prompting [10] and cross-modal adaptation 197 101
Conclusion
To explore the benefits in speech- language co-learning, this work introduces new ASR benchmark that utilizes LLMS for transcription prediction from N-best hypotheses. Our benchmark contains a new HP dataset consisting of more than 334K bypotheses-transcrrpiio pairs that are collected from 9 different public ASR corpora. In few-shot settings, we demonstrate that LLMSs with in-context learning can serve as a plug-and-play back end effectively alleviate domain shift of ASR. In the fine-tuning setting, our proposed еггоr correction technique based on LLMs achieves better WER performance than the upper-bound of re-ranking based method, which provides a new paradigm for applying ASR in some challenging conditions, such as background noise and speaker accent. We believe our benchmark and findings provide new and unique insights into LLM-enhanced ASR.
speech-language

---

References
[1] Armen Aghajanyan, Luke Zer,,emoyer, and Sonal Gupta. Intrinsic dimensionality explains the effectiveness of language model fine-tuning. arXiv preprint arXiv:20/2. /3255, 2020.
[2] Alena Aksenova, Daan van Esch, James Flynn, and Pavel Golik. How might we create better benchmarks for speech recognition? In Proceedings of the Ist Workshop Benchmarking: Past, Present and Future, pages 22-44,2 2021.
[3] Rosana Ardila, Megan Branson, Kelly Davis, Michael Henretty, Michael Kohler, Josh Meyer, Reuben Morais, Lindsay Saunders, Francis M Tyers, and Gregor Weber. Common voice: A muasively-multilinge speech corpus. arXiv preprint arXiv: arXiv: 1912.06670, 2019.
[4] Ebru Arisoy, Abhinav Sethy, Bhuvana Ramabhadran, and Stanley Chen. Bidirectional recurrent neural network language models for automatic speech recognition. In 2015 IEEE International Conference Acoustics, Speech and Signal Processing (ICASSP), pages 5421-5425. IEEE, 2015.
[5] Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhari- wal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems, 33:1877- 1901. 2020.
[6] Hui Bu, Jiayu Du, Xingyu Na, Bengu Wu, and Hao Zheng. Aishell-1: An open-source mandarin speech corpus and a speech recognition baseline. In 2017 20th conference of the oriental chapter of the international coordinating committee on speech databases and speech 1/O systems and assessment (O- (O-COCOSDA), pages 1-5. IEEE, 2017.
[7] Daniela Caballero, Roberto Araya, Hanna Kronholm, Jouni Viiri, André Mansikkaniemi, Sami Lehesvuori, Tuomas Virtanen, and Mikko Kurimo. Ast in classroom today: Automatic visualization of conceptual network in science classrooms. In Data Driven Approaches in Digital Education: 12th European Conference on Technology Enhanced Learning, EC-TEL 2017, Tallinn, Estonia, September 12-15, 2017, Proceedings 12, pages 541-544. Springer, 2017.
[8] Stephanie Chan, Adam Santoro, Andrew Lampinen, Jane Wang, Aaditya Singh, Pierre Richemond, James McClelland, and Felix Hill. Data distributional properties drive emergent in-context learning in transformers. Advances in Neural Information Processing Systems, 18:18888-1889, 2022.
[9] William Chan, Navdeep Jaitly, Quoc Le, and Oriol Vinyals. Listen, attend and spell: neural network for large vocabulary conversational speech recognition. In 2016 IEEE international conference on acoustics, speech and signal processing (ICASSP), pages 4960-4964. IEEE, 2016.
[10] Kai- Wei Chang, Yu-Kai Wang, Hua Shen, lu-thing Kang. Wei-Cheng Tseng, Shang- Wen Li, and Hung-yi Lee. Speechprompt v2: Prompt tuning for speech classification tasks. arXiv preprint arXiv:2303.00733, 2023.
[11] Chen Chen, Nana Hou, Yuchen Hu, Shashank Shirol, and Eng Siong Chng. Noise-robust speech recognition with 10 minutes unparalleled in-domain data. In ICASSP 2022-2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 4298-4302. IEEE, 2022.
[12] Guoguo Chen, Shuzhou Chai, Guanbo Wang, Jiayu Du, Wei-Qiang Zhang, Chao Weng, Dan Su, Daniel Povey, Jan Trmal, Junbo Zhang, et al. Gigaspeech: An evolving, multi-domain asr corpus with 10,000 hours of transcribed audio. arXiv preprint 106.v:21166099000 2021.
[13] Qian Chen, Zhu Zhuo, and Wen Wang. Bert for joint intent classification and slot filling. arXiv preprint 19xiv:1902. 10909, 2019.
[14] Sanyuan Chen, Chengyi Wang, Zhengyang Chen, Yu Wu, Shujie Liu, Zhuo Chen, Jinyu Li, Naoyuki Kanda, Takuya Yoshioka, Xiong Xiao, et al. Wavim: Large-scale self-supervised pre-training for full stack speech processing. IEEE Journal of Selected Topics Signal Processing, 1606))1505-1511 2022.
10

---

[15] Zih-Ching Chen, Chao-Har Huck Yang, Bo Li, Yu Zhang, Nanxin Chen, Shou- Yiin Chang, Rohit Prabhavalkar, Hung-yi yi Lee, and Tara N Sainath. How to estimate model transferability of pre-trained speech models? Proc. Interspecch, 2023.
[16] Hyunsoo Cho, Hyuhng Joon Kim, Junyeob Kim. Sang- Woo Lee, Sang-goo Lee, Kang Min Yoo, and Taeuk Kim. Prompt-augmented línear probing: Scaling beyond the limit of few-shot in-context learners. arXiv preprint arXiv:2212. 10873, 2022.
Sang-Woo
[17] Jan Chorowski and Navdeep Jaitly. Towards better decoding and language model integration sequence to sequence models. arXiv preprint arXiv: 16t:: 1612.0265, 2016.
[18] Grzegorz Chrupata. Putting natural in natural arXiv:2305.0453 2023.
language processing. arXiv preprint
[19] Joon Son Chung, Andrew Senior, Oriol Vinyals, and Andrew Zisserman. Lip reading sentences in the wild. In 2017 IEEE conference on computer vision and pattern recognition (CVPR). pages 3444-3453, IEEE, 2017.
[20] Daniel Dahlmeier and Hwee Tou Ng. Better evaluation for grammatical error correction. In Proceedings of the 2012 Conference of the North American Chapter of the Association for Computational Linguisties: Human Language Technologies, pages 568-572 2012.
[21] Jacob Devlin, Ming- Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint 2018.
[22] Linhao Dong, Shuang Xu, and Bo Xu. sreech-tranfformer. a no-recurrence sequence-to- sequence model for speech recognition. In 2018 IEEE international conference on acoustics, speech and signal processing (ocrgg/(CAA (ICASSP), pages 5884-5885 IEEE, 2018
[23] Samrat Dutta, Shreyansh Jain, Ayush Maheshwari, Souvik Pal, Ganesh Ramakrishnan, and Preethi Jyothi. Error correction in asr using sequence-to-sequeee to-seuuence models. arXiv preprint trXiv:2202.01 157, 2022.
[24] Zhe Gan, Linjie Li, Chunyuan Li, Lijuan Wang, Zicheng Liu, Jianfeng Gao, al. Vision- language pre-training: Basics, recent advances, and future trends. Foundations and Trends@ Computer Graphics and Vision, 14(3 43:163-3:3333 2022.
[25] Ankur Gandhe and Ariya Rastrow. Audio-attention discriminative language model for asr rescoring. ICASSP 2020-2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 7944-7948. IEEE, 2020.
[26] John Godfrey, Edward C Holliman, and Jane McDaniel. Switchboard: Telephone speech corpus for research and development. In Acoustics, Speech, and Signal Processing. IEEE International Conference on, volume 1, pages 517-520. IEEE Computer Society, 1992.
[27] Alex Graves. Sequence transduction with recurrent neural networks. arXiv preprint arXiv:/211.3711, 2012.
[28] Alex Graves, Santiago Fernández, Faustino Gomez, and Jürgen Schmidhuber. Connectionist temporal classification: labelling unsegmented sequence data with recurrent neural networks. In Proceedings of the 23rd international conference on Machine learning, pages 369-376,
2006.
[29] Terri A Greenslade and J César Bhiix-Braddeer. Error correction and learner perceptions in 12 spanish writing. In Selected Proceedings of the 7th Conference on the Acquisition of Spanish and Portuguese as First and Second Languages, pages 185-194. Somerville, MA: Cascadilla Proceedings Project, 2006.
[30] Anmol Gulati, James Qin, Chung-Cheng Chiu, Niki Parmar, Yu Zhang, Jiahui Yu, Wei Han, Shibo Wang, Zhengdong Zhang, Yonghui Wu, et al. Conformer: Convolution-augmentte transformer for speech recognition. Proc. Interspeech 2020, pages 5036-5040, 2020.
11

---

[31] Karen Hambardzumyan, Hrant Khachatrian, and Jonathan May. Warp: Word-level adversarial reprogramming. arXiv preprint arXiv:2101.00121, 2021.
[32] Hany Hassan, Anthony Aue, Chang Chen, Vishal Chowdhary, Jonathan Clark, Christian Feder- mann, Xuedong Huang. Marcin man parity on automatic chinese to english news translation. arXiv preprint arXiv: 035666 2018.
Downunt, William Lewis, Mu Li, et al. Achieving hu-
[33] Ruidan He, Linlin Liu, Hai Ye, Qingyu Tan, Bosheng Ding, Liying Cheng, Jia- Wei Low, Lidong Bing, and Luo Si. On the effectiveness of adapter-based tuning for pretrained language model adaptation. arXiv preprint arXiv:2106.03 2021.
[34] Charles T Hemphill, John Godfrey, and George R Doddington. The atis spoken language systems pilot corpus. In Speech and Natural Language: Proceedings of Workshop Held at Hidden Valley, Pennsylvania, June 24-27, 1990, 1990.
[35] François Hernandez, Vincent Nguyen, Sahar Ghannay, Natalia Tomashenko, and Yannick Esteve. Ted-lium 3: Twice much data and corpus repartition for experiments on speaker adaptation. In Speech and Computer: 20th International Conference, SPECOM 2018, Leipzig, Germany, September 18 18, 2018, Proceedings 20, pages 198-208. Springer, 2018.
[36] Pavel Hlubík, Martin Spaněl, Marek Boháč, and Lenka Weingartová. Inserting punctuation to ast output in a real-time production environment. In Text, Speech, and Dialogue: 23rd Inter- national Conference, TSD 2020, Brno, Czech Republic, September 8-11, 2020, Proceedings, pages 418-425. Springer, 2020.
[37] Due Tam Hoang, Shamil Chollampatt, and Hwee Tou Ng. Exploiting n-best hypotheses to improve an smt approach to grammatical error correction. arXiv preprint arXiv: 1606.00210, 2016.
[38] Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models. arXiv preprint
2021.
[39] Ke Hu, Bo Li, and Tara N Sainath. Scaling up deliberation for multilingual asr. In 2022 IEEE Spoken Language Technology Workshop (SLT), pages 771-776. IEEE, 2023.
[40] Ke Hu, Ruoming Pang, Tara N Sainath, and Trevor Strohman. Transformer based deliberation for two-pass speech recognition. 2021 IEEE Spoken Language Technology Workshop (SLT), pages 68-74. IEEE, 2021.
[41] Ke Hu, Tara N Sainath, Ruoming Pang, and Rohit Prabhavalkar. Deliberation model based two-pass end-to-end speech recognition. In ICASSP 2020-2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 7799-7803. IEEE, 2020.
[42] Zhiqiang Hu, Yihuai Lan, Lei Wang. Wanyu Xu, Ec-Peng Lim, Roy Ka-Wei Lee, Lidong Bing. and Soujanya Poria. LIm-adapters: An adapter family for parameter-efficient fine-tuning of large language models. arXiv preprint arXiv:2304.0/933 2023.
[43] Hongzhao Huang and Fuchun Peng. An empirical study of efficient asr rescoring with 2019.
transformers. arXiv preprint
[44] Yukun Huang, Yanda Chen, Zhou Yu, and Kathleen McKeown. In-context learning distilla- tion: Transferring few-shot learning ability of pre-trained language models. arXiv preprint 0rXiv:2212.106777 2022.
[45] Jacob Kahn, Morgane Riviere, Weiyi Zheng, Evgeny Kharitonov, Qiantong Xu, Pierre- Emmanuel Mazaré, Julien Karadayi, Vitaliy Liptchinsky, Ronan Collobent, Christian Fuegen, et al. Libri-light: A benchmark for asr with limited or no supervision. In ICASSP 2020-2020 IEFE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 7669-7673, IEEE, 2020.
12

---

[46] Anjuli Kannan, Yonghui Wu, Patrick Nguyen, Tara N Sainath, Zhijeng Chen, and Rohit Prabhavalkar. An analysis of incorporating an external language model into sequence-to- sequence model. In 2018 IEFF International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 1-5828. IEFF, 2018.
[47] Tyler Kendall and Charlie Farrington. The corpus of regional african american language. version 2021.07. eugene, or: The online resources for african american language project, 2021.
[48] Anis Koubaa. Gpt-4 vs. gpt-3.5: A concise showdown. arXiv preprint, 2023.
[49] Andrew K Lampinen, Ishita Dasgupta, Stephanie CY Chan, Kory Matthewson, Michael Henry Tessler, Antonia Creswell, James L McClelland, Jane X Wang, and Felix Hill. Can language models learn from explanations in context? arXiv preprint arXiv:2204. 02329, 2022.
[50] Siddique Latif, Junaid Qadir, Adnan Qayyum, Muhammad Usama, and Shahzad Younis. Speech technology for healthcare: Opportunities, challenges, and state of the art. IEEE Reviews in Biomedical Engineering, 14:342-356, 2020.
[51] Yichong Leng, Xu Tan, Wenjie Liu, Kaitao Song, Rui Wang, Xiang- Yang Li, Tao Qin, Ed Lin, and Tie-Yan Liu. Softcorrect: Error correction with soft detection for automatic speech recognition. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 37, pages 13034-13042, 2023.
Xiang-Yang
[52] Yichong Leng, Xu Tan, Rui Wang, Linchen Zhu, Jin Xu, Wenjie Liu, Linquan Liu, Tao Qin, Xiang-Yane Li, Edward Lin, et al. Fastcorrect 2: Fast enror correction on multiple candidates for automatic speech recognition. arXiv preprint arXiv:2109. /4420, 2021.
[53] Yichong Leng, Xu Tan, Linchen Zhu, Jin Xu, Renqian Luo, Linquan Liu, Tao Qin, Xiangyang Li, Edward Lin, and Yan Liu. Fastcorrect: Fast error correction with edit alignment for automatic speech recognition. Advances in Neural Information Processing Systems, 34:21708- 21719, 2021.
[54] Brian Lester, Rami AI-Rfou, and Noah Constant. The power of scale for parameter-efficient prompt tuning. arXiv preprint 2xXiv:2104.0866 2021.
[55] Jinyu Li, Li Deng, Yifan Gong, and Reinhold HabbhUUmbchh, An overview of noise-robust automatic speech recognition. /EEE/ACM Transactions on Audio, Speech, and Language Processing, 22(4):745-777, 2014.
[56] Mingda Li, Weitong Ruan, Xinyue Liu, Luca Soldaini, Wael Hamza, and Chengwei Su. Improving spoken language understanding by exploiting asr n-best hypotheses. arXiv preprint arXiv:2001.05284. 2020.
[57] Tatiana Likhomanenko, Qiantong Xu, Vineel Pratap, Paden Tomasello, Jacob Kahn, Gilad Avidov, Ronan Collobert, and Gabriel Synnaeve. Rethinking evaluation in asr: Are our models robust enough? arXiv preprini arXiv:2010. 1745, 2020.
[58] Zhaojiang Lin, Andrea Madotto, and Pascale Fung. Exploring versatile generative language model via parameter-efficient transfer learning, arXiv preprint arXiv:2004.03829 2020.
[59] Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. Roberta: A robustly optimized bert pretraining approach. arXiv preprint arXiv: 1692, 2019.
[60] Rao Ma, Mark JF Gales, Kate Knill, and Mengjie Qian. N-best 15: Robust ast error cor- rection using multiple input hypotheses and constrained decoding space. arXiv preprint arXiv:2303.0045 2023.
[61] Anirudh Mani, Shruti Palaskar, Nimshi Venkat Meripo, Sandeep Konam, and Florian Metze. Asr error correction and domain adaptation using machine translation. In ICASSP 2020-2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 6344-6348. IEEE, 2020.
13

---

[62] Tomas Mikolov, Martin Karafíát, Lukas Burget, Jan Cemockýý, and Sanjeev Khudanpur. Recurrent neural network based language model. In Interspeech, volume 2. pages 1045-1048. Makuhari, 2010.
[63] Sewon Min, Mike Lewis, Luke Zettlemoyer, and Hannaneh Hajishirzi. Metaicl: Learning to learn in context. arXiv preprint arXiv:2 10.110..555 15943, 2021.
[64] Sewon Min, Xinxi Lyu, Ari Holtzman, Mikel Artetxe, Mike Lewis, Hannaneh Hajishirzi, and Luke Zettlemoyer. Rethinking the role of demonstrations: What makes in-context learning work? arXiv preprint arXiv:2202. 12837, 2022.
[65] Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models follow instructions with human feedback. Advances in Neural Information Processing Systems, 35:27700-27744, 2022.
[66] Vassil Panayotov, Guoguo Chen, Daniel Povey, and Sanjeev Khudanpur. Librispeech: an asr corpus based on public domain audio books. 2015 IEEE international conference on acousticx, speech and signal processing (ICASSP), pages 5206-5210. IEEE, 2015.
[67] Douglas B Paul and Janet Baker. The design for the wall street journal-based esr corpus. In Speech and Natural Language: Proceedings of Workshop Held at Harriman, New York, February 23-26, 1992, 1992.
[68] Fuchun Peng, Scott Roy, Ben Shahshahani, and Françoise Beaufays. Search results based n-best hypothesis rescoring with maximum entropy classification. In 2013 IEEE Workshop Automatic Speech Recognition and Understanding, pages 422-427. IEEE, 2013.
[69] Alec Radford, Jong Wook Kim, Tao Xu, Greg Brockman, Christine McLeavey, and Ilya Sutskever. Robust speech recognition via large-scale weak supervision. arXiv preprint arXiv:2212.04356. 2022.
[70] Alec Radford, Jeffrre Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. Language models are unsupervised multitask learners. OpenAl blog, 1(8):9, 2019.
[71] Srijith Radhakrishnan, Chao-Han Huck Yang. Sumeer Ahmad Khan, Rohit Kumar. Narsis A Kiani, David Gomez-Cabrero, and Jesper N Tegner. Whispering llama: cross-modal gener- ative error correction framework for speech recognition. arXiv preprint 2t.i:2310006433. 2023.
[72] Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. Exploring the limits of transfer learning with unified text-to-text transformer. The Journal of Machine Learning Research, 21(1):5485-5511, 2020.
[73] Ohad Rubin, Jonathan Herzig, and Jonathan Berant. Learning to retrieve prompts for in-context learning. arXiv preprint arXiv:2 2,06633, 2021.
[74] Julian Salazar, Davis Liang. Toan Q Nguyen, and Katrin Kirchhoff. Masked language model scoring. arXiv preprint arXiv: /4659, 2019.
[75] Teven Le Scao, Angela Fan, Christopher Akiki, Ellie Pavlick, Suzana Ilié, Daniel Hesslow, Roman Castagné, Alexandra Sasha Luccioni, François Yvon, Matthias Gallé, et al. Bloom: A 176b-parameter open-access multilingual language model. arXiv preprint arXiv:221 11.22110510 1,05100, 2022.
[76] Changhao Shan, Chao Weng, Guangsen Wang, Dan Su, Min Luo, Dong Yu, and Lei Xie. Component fusion: Learning replaceable language model component for end-to-end speech recognition system. In ICASSP 2019-2019 IEEE International Conference Acoustics, Speech and Signal Processing (oCiongggCCCC (ICASSP), pages 5361-5635. IEFF, 2019.
[77] Kai Shen, Yichong Leng, Xu Tan, Siliang Tang. Yuan Zhang, Wenjie Liu, and Edward Lin. Mask the correct tokens: embarrassingly simple approach for error correction. arXiv preprint arXiv:2211.13252. 2022.
14

---

[78] Xian Shi, Qiangze Feng, and Lei Xie. The asru 2019 candarin-englinng code-switching speech recognition challenge: Open datasets, tracks, methods and results. arXiv preprint arXiv:2007.05516. 2020.
[79] Joonbo Shin, Yoonhyung Lee, and Kyomin Jung. Effective sentence scoring method using bert for speech recognition. In Asian Conference on Machine Learning, pages 1081-1093. PMLR, 2019.
[80] Prashanth Gurunath Shivakumar, Mu Yang, and Panayiotis Georgiou. Spoken language intent detection using confusion2vec. Proc. of Interspeech, 2019.
[81] Anuroop Sriram, Heewoo Jun, Sanjeev Satheesh, and Adam Coates. Cold fusion: Training seq2seq models together with language models. arXiv preprint arXiv: 170.0666,, 2017.
[82] Piotr Szymanski, Piotr Zelasko, Mikolaj Morzy, Adrian Szymczak, Marzena HyppeH Joanna Banaszezak, Lukasz Augustyniak, Jan Mizgajski, and Yishay Carmiel. Wer we are and wer we think we are. arXiv preprint arXiv:2010.03432. 2020.
[83] Shubham Toshniwal, Anjuli Kannan, Chung-Cheng Chiu, Yonghui Wu, Tara N Sainath, and Karen Livescu. A comparison of techniques for language model integration in encoder- decoder speech recognition. 2018 IEEE spoken language technology workshop (SLT), pages 369-375. IEEE, 2018.
[84] Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timo- thée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models. arXiv preprint (XKIv:2302.13977/ 2023.
[85] Mehmet Ali Tugtekin Turan, Emmanuel Vincent, and Denis Jouvet. Achieving multi-accent asr via unsupervised acoustic model adaptation. In INTERSPEECH 2020, 2020.
[86] Takuma Udagawa, Masayuki Suzuki, Gakuto Kurata, Nobuyasu Itoh, and George Saon. Effect and analysis of large-scale language model rescoring on competitive asr systems. arXiv preprint ar0iv:2204,00 0020120,0222.... 2022.
[87] Ehsan Variani, Tongzhou Chen, James Apfel, Bhuvana Ramabhadran, Seungji Lee, and Pedro Moreno. Neural oracle search on n-best hypotheses. In ICASSP 2020-2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 7824-7828. IEEE, 2020.
[88] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention all you need. Advances in neural information processing systems, 30, 2017.
[89] Emmanuel Vincent, Shinji Watanabe, Jon Barker, and Ricard Marxer. The 4th chime speech separation and recognition challenge. URL: http://spandh. des. shef. ac. uk/chime u//chime_challenge/eee challenge//last accessed on / August, 2018), 2016.
[90] Changhan Wang, Morgane Riviere, Ann Lee, Anne Wu, Chaitanya Talnikar, Daniel Haziza, Mary Williamson, Juan Pino, and Emmanuel Dupoux. Voxpopuli: A large-scale multilingual speech corpus for representation learning, semi-supervised learning and interpretation. arXiv preprint 20 10000090,,00.0000 2021.
[91] Weiran Wang, Ke Hu, and Tara Sainath. Deliberation of streaming rnn-transducer by non- autoregressive decoding. In ICASSP 2022-2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 7452-7456. IEEE, 2022.
[92] Yidong Wang, Zhuohao Yu, Zhengran Zeng, Linyi Yang, Cunxiang Wang, Hao Chen, Chaoya Jiang, Rui Xie, Jindong Wang, Xing Xie, et al. Pandalm: An automatic evaluation benchmark for Im instruction tuning optimization. arXiv preprint arXiv:2306.0508/ 2023.
[93] Yu Wang, Yuelin Wang, Kai Dang, Jie Liu, and Zhuo Liu. A comprehensive survey of grammatical error correction. ACM Transactions on Intelligent Systems and Technology (TIST), 12(5):1-51, 2021. 2021.
15

---

[94] Shinji Watanabe, Takaaki Hori, Shigeki Karita, Tomoki Hayashi. Jiro Nishitoba, Yuya Unno, Nelson Enrique Yalta Soplin, Jahn Heymann, Matthew Wiesner, Nanxin Chen, et al. Espnet: End-to-end speech processing toolkit. arXiv preprint arXiv::804.00000 www::/00400055 2018.
[95] Jason Wei, Yi Tay, Rishi Bommasani, Colin Raffel, Barret Zoph, Sebastian Borgeaud, Dani Yogatama. Maarten Bosma, Denny Zhou, Donald Metzler, et al. Emergent abilities of large language models. arXiv preprint arXiv:2206.0700 2022.
[96] Johannes Wirth and Rene Peinl. Automatic speech recognition in german: A detailed error analysis. In 2022 IEEE International Conference on Omni-layer Intelligent Systems (COINS), pages 1-8. IEEE, 2022.
[97] Haibin Wu, Kai-Wei Chang, Yuan-Kuei Wu, and Hung-yi Lee. Speechgen: Unlocking the generative power of speech language models with prompts. arXiv preprint arXiv:2306.02207, 2023.
[98] Yingce Xia, Fei Tian, Lijun Wu, Jianxin Lin, Tao Qin, Nenghai Yu, and Tie- Yan Liu. De- liberation networks: Sequence generation beyond one-pass decoding. Advances in neural information processing systems, 30, 2017.
[99] Sang Michael Xie, Aditi Raghunathan, Percy Liang, and Tengyu Ma. An explanation of in-context learning implicit bayesian inference. arXiv preprint arXiv:2/11.02080. 2021.
[100] Chao-Han Huck Yang, Yile Gu, Yi-Chieh Liu, Shalini Ghosh, Ivan Bulyko, and Andreas Stoleke. Generative Speech Recognition Error Correction with I Large Language Models and Task-Activating Prompting. arXiv preprint 1tti::3399.164999 2023.
[101] Chao-Han Huck Yang, Bo Li, Yu Zhang, Nanxin Chen, Rohit Prabhavalkar, Tara N Sainath, and Trevor Strohman. From english to more languages: Parametee--effiieet model reprogramming for cross-lingual speech recognition. Proc. of ICASSP, pages 1-5. IEEE, 2023.
[102] Chao-Han Huck Yang, Yun- Yun Tsai, and Pin-Yu Yu Chen. Voice2series: Reprogramming acoustic models for time series classification. In International Conference on Machine Learning, pages 1180991889,,,1111111111111111119 PMLR, 2021.
[103] Xi Ye and Greg Durrett. Explanation selection using unlabeled data for in-context learning. arXiv preprint 2tti::330000811333 2023.
[104] Albert Zeyer, Kazuki Irie, Ralf Schlüter, and Hermann Ney. Improved training of end-to-end attention models for speech recognition. arXiv preprint arXiv:1805.0322 2018.
[105] Renrui Zhang, Jiaming Han, Aojun Zhou, Xiangfei Hu, Shilin Yan, Pan Lu, Hongsheng Li, Peng Gao, and Yu Qiao. alame--dapter: Efficient fine- fine-tuning of language models with zero-init attention. arXiv preprint arXiv:2303. 16199, 2023.
[106] Shiliang Zhang. Ming Lei, and Zhijie Yan. Automatic spelling correction with transformer for ctc-based end-to-end speech recognition. arXiv preprint arXiv: 1904. 1004. 2019.
[107] Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christopher Dewan, Mona Diab, Xian Li, Xi Victoria Lin, al. Opt: Open pre-trained transformer language models. arXiv preprint arXiv:2205.01068, 2022.
[108] Yuanhan Zhang, Kaiyang Zhou, and Ziwei Liu. What makes good examples for visual in-context learning? arXiv preprint arXiv:2301. 13670, 2023.
[109] Yun Zhao, Xuerui Yang, Jinchao Wang, Yongyu Gao, Chao Yan, and Yuanfu Zhou. Bart based semantic correction for mandarin automatic speech recognition system. arXiv preprint arXiv:2104.05507,,22 2021.
16

---

Checklist
The checklist follows the references. Please read the checklist guidelines carefully for information on how to answer these questions. For each question, change the default [TODO] to [Yes], [No], , or [N/A] - You are strongly encouraged to include justification your answer, either by referencing the appropriate section of your paper or providing a brief inline description. For example:
Did you include the license to the code and datasets? [Yes]
Please do not modify the questions and only use the provided macros for your answers. Note that the Checklist section does not count towards the page limit. In your paper, please delete this instructions block and only keep the Checklist section heading above along with the questions/answers below.
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope? [Yes]
(b) Did you describe the limitations of your work? [Yes] See Section 5.3
(c) Did you discuss any potential negative societal impacts of your work? [N/A]
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [N/A]
(b) Did you include complete proofs of all theoretical results? [N/A]
3. If you experiments (e.g. for benchmarks)..
(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either the supplemental material or URL)? [Yes] See Section 3
(b) Did you specify all the training details (e.g., data splits, byperparameters, how they were chosen)? [Yes] See Section 5.1
(c) Did you report error bars (e.g., with respect the random seed after running experi- ments multiple times)? [N/A]
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)? [Yes]
4. If you using existing assets (e.g., code, data, models) curating/releasing assets...
(a) If your work uses existing assets, did you cite the creators? [Yes]] See Appendix.
(b) Did you mention the license of the assets? [Yes] See Appendix.
(c) Did you include any new assets either in the supplemental material or as a URL? [Yes]
(d) Did you discuss whether and how consent was obtained from people whose data you're using/curating? [N/A]
(e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content? [N/A]
5. If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given participants and screenshots, applicable? [N/A]
(b) Did you deseribe any potential participant risks, with links to Institutional Review Board (IRB) approvals, applicable? [N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation? [N/A]
17

---

Appendix
Have LLMs ASR transcription during pre-training?
We raise a concern about LLM correction: is the remarkable correction result owing to to the fact that the ASR transcriptions exist in the pre-trained corpus of L.M?
To evaluate this concern for GPT-3.5, we manually create some hypotheses to simulate the utterance in case study in Table 4, and we show the correction results in Table
Table 5: Synthetic hypotheses for LLM correction
ID
Utterance
Patrue China and xinnepec own all national oil pipeline infrastructure. Corrected item: "Patrue China" > "PetroChina", " "xinnepec" - "Sinopec"
The Legend of Saida is the most anticipated game of this year from nine ten door. Corrected item: "Saida" "Zelda", "nine ten door" "Nintendo"
The world error rate is the primary matrix to evaluate ASR performance. Corrected item: "world" - "word", "matrix" > "metric"
3
He missed the mid nine flight from Singapore to rose island. Corrected item: "mid nine" "midnight", "rose" + "Rhodes"
We observe that given arbitrarily fabricated utterances in different scenarios, LLM can still give reasonable correction results, especially on specific terms. Furthermore, we attach the LLM reasoning process for two examples, which shows the utilization of context information:
ID I: Please note that the transcription assumes "Patrue China" was an error, and "PetroChina" was intended instead. Additionally, "xinnepec" has been interpreted as "Sinopec," " which is another major Chinese oil and gas company.
ID 2: Please note that there may have been some confusion with the phrase "nine ten door" as it doesn't match any known company or widely recognized term. As a result, have inferred that may have been a misinterpretaatin or typographical error. The transcription assumes "Zelda" as possible replacement for "Saida" " based on its similarity to the well-known game series "The Legend of Zelda" developed by Nintendo.
In H2T-ft, the T5 model was pre-trained C4 dataset that does not include WSJ, but it still achieves sigmificant WER reduction on WSJ. In H2T-LoRA, we do not find the any ASR transcript exists in the pre-trained data of LlaMa according to their papers public information.
In general, we argue that the improved results of LLMS correction primarily stem from its extensive knowledge base and the ability to utilize context information as part of a formal scientific experimental process. However, the methods through which indirect knowledge captured and propagated across large remain open questions for future investigation. We hope that by initiating this study, the results would encourage more further and ongoing works on knowledge grounding in the future.
Resources for Hypotheses Paradise (HP) Generation
The related toolkit and pre-trained models used to generate HP dataset are listed as follows:
For ASR model, we used pre-trained WavLM from HuggingFace and the ESPNet toolki
When using WavL. for ASR decoding, we also incorporate an external LM rescoring option, which trained using extra text data from wel
Apart from WavLM, also select Whisper-Large for ASR decoding.
wav wavlm. wtw..lle.rgr...rrrrbbbbb.p.pppp.
http:///ttr..ooaaarccoc/coooooec 1m-norm. wwtorm.ttttgg................... txt.gz
https://ggthau..oom/geee////////
18

---

Task-Activated Prompting (TAP) for Error Correction Evaluation
We illustrate the given prompt for task-activated prompting [100] (TAP), as well the response from the frozen pretrained LLM for each turn, as proposed by Yang et al. [100] in a recent work.
Q: Are you familiar with speech recognition?
R: Yes, am familiar with speech recognition. Speech recognition, also known as automatic speech recognition (ASR) or speech-to-tent is the process of converting spoken language into text. This technology involves using algorithms and machine learning models to analyze and transcribe the acoustic features of spoken words and phrases. Speech recognition has many applications, including voice-controlled assistants, automated phone systems, and transcription services.
Q: Are you familiar with language model rescoring in ASR?
R: Yes, I am familiar with language model rescoring for speech recognition. Language model rescoring is a technique used to improve the accuracy of speech recognition systems. It involves using separate language model evaluate the likelihood of given hypothese list. This separate model is rypically more complex and powerful than the initial language model used for the transcription, and it is used to re-score the transcription based on the probability of the words occurring in the given context. The rescoring process involves taking the outpul of the initial language model, which usually based statistical methods such as Hidden Markov Models, and then applying a more advanced language model, such as a neural network-hasee language model, to generate a more accurate transcription. This accomplished by re-ranking the possible transcriptions based on the probabilities assigned by the more advanced language model. Language model rescoring has been shown to significantly improve the accuracy of speech recognition systems, particularly in noisy or challenging environments where the initial language model may not perform well.
Q: Can you give a possible example language model rescoring with 5-best hypotheses?
R: Sure, here is an example of language model rescoring for ASR with 5-best hypotheses:
want to go to the store.
2. I want to go to the storm.
3. want to go to the stove.
4. want to go to the star.
5. I want to go to the storage.
After rescoring, I think the ground-truth of this speech should be: I want go the store.
Q: Nice job, will give you real example as a demonstration from [target domain). The best hypothesis is:[hypotheses list from training set), and I expect your output is: [corresponding transcription). Following this example, can you report the true transcription from the following 5-best hypotheses:? [hypotheses list for inference]
is noted that the responses from GPT-3.5 vary from random seeds. In our experiments, we utilize the API key from OpenAI, and fix the above template before the last turn eтror correction.
We observe that many recent works 1031 of ICL focus the selection strategy for in-domain demonstration, which might have an obvious impact on inference results. We leave the demonstration selection as future work, and in our few-shot learning, we manually select those utterances with long lengths according to 64
Hypotheses-to- Transscrpiooan (H2T) Training Configuration
H2T-ft. We employ the T5-v1. 1i-v1. pre-trained model (0.75B) downloaded from HuggingFace Compared with the original T5 model, GELU Sevres as activation function in the feed-forward layer to replace ReLU. Furthermore, T5 Version 1.1 was only pre-trained on C4 excluding any supervised training. Therefore, this model has to be fine-tuned before it is applied on downstream task.
httpp:///unggggggooooooogogg/// large
19

---

We finetune 20 epochs on each domain of HP dataset with a batch size of 16. To select the best model, we first split a validation set with 5% data amount training set. The learning rate varies from 1 x e 1 x e according to data amount of each domain, and AdamW is employed for optimization. The a1 to 0n are set as 0.1, 0.05, 0.05, 0.05 respectively, as the 2nd utterances are usually more informative than others as shown in Fig.1. In practice, we observe the over- fitting phenomenon during training. The WER on training set can be lower than 1%, however, the performance on CORAAL dataset even is even worse than the baseline. In other words, H2T-ft still has room for improvement by adding some techniques for avoiding over-fitting.
H2T-LORA. We select LlaMa- 13B as the frozen pre-trained model in our method, which is down- loaded from HuggingFace 2 The learning rate set as 1e -4, and the batch size 128. For the low-rank adapter, we implement by peft | where the configuration of rank r is set as 8. Similarly, we also use 1--aree as pre-trained model with low-rank adapter for experiments, where the learning rate is set as 3e and the lora_r is set as 16.
We train 10 epochs using AdamW optimizer, and the prompt for LLM designed as follows:
"Below a best-hypotheses that transcribed from an automatic speech recognition system. Write a response to predict the true transcription using the tokens from other-hypotheses.fe ### best- hypothesis:/1* utterance #iilt other-lypothesis:f22" ~ 5thh utterances) ##fResponse:"
The prompt template is not unique, and it leaves slight impact on the final WER result. Additionally. we calculate the WER using Sclite" " toolkit, which keep consistent with evaluation script of ESPNe
L.Maank is an Transformer-basse the training transcription from each HP domain is utilized for typical LM training. The Transformer layer of each model varies from 8 to 16 in terms of data amount. The training epoch is set as 20, and Adam is employed optimizer. The initial learning rate is set as 0.002 with warm up strategy. During decoding, the perplexity of each hypothesis is calculated for re-ranking the N-best list, and the utterance with the lowest perplexity is selected as the final output.
language model that is implemented using ESPNet toolkit
13 where
LibriSpeech Results and Failure Cases Study
We list two representative failure cases from LibriSpeech--est-other Table. 7 For the first case, "ward" is corrected by "warde" as there is an "his" behind it. Additionally, we observe that "warde" also appears in the 2nd hypothesis, so LLM adopts according context information. For the second case, LLM directly adopts the 2nd utterance in the N-best list, as "think" does not often appear at the beginning of a sentence from a grammatical perspective. Therefore, [100] and explained in future work 55.5 we argue that LLM correction should also consider acoustic information provided by the ASR system, which helps to avoid "over-correction" cases and keeps the fidelity to spoken language.
Table WER (%) results on LibriSpeech dataset, "onb" " and "Oep" respectively denote n-best oracle and compositional oracle that are defined in 5.2
Correction with H2T-ft
Oracle Oab
Test Set
Baseline
LM
H2T-LoRA
Opp
LS-clean
1.8
1.8
1.8 1.8-0.0%
1.7 1.7-5.6%
1.0
0.6
L-S-tthr
3.7
3.7
3.945.4%
3.8 2.7%
2.7
1.6
Results on Mandarin and code-switching Dataset
We include AISHELL-1 6 as Mandarin dataset into HP benchmark, consisting of training set with 120098 utterances and a testing set with 7176 utterances. We randomly select 20k examples (16.7%) from training set evaluate the effect of proposed H2T-LoRA. For the foundation model,
httegs//ggghhbbb.o///ggggggggggg
20

---

Table 7: Failure cases corrected by H2T-LoRA, The utterances are drawn from LibriSpecch--eestorhher
Type
Utterance
WER
1st Hypo.
Yet there was gambling again the second night between ward and several others of his profession.
by AM
Correction by H2T
Yet there was gambling again the second night between warde and several others of his profession.
6.25
Ground-truth Transcription
Yet there was gambling again the second night between ward and several others of his profession.
1s Hypo. by AM
Think he really needs it he pursued
Correction by H2T
He really needs it he pursued
14.3
Ground-trut
Think he really needs it he pursued
Transcription
employ Chinese LlaMa2-78 7b from Huggingface and keep other settings consistent with H2T-LoRA
in this paper. Notably. ASR Mandarin dataset usually evaluated by character error rate (CER), as character is equal to word in Chinese. We also evaluate H2T-LoRA on ASRU [78] that a Mandarin- English code-switching dataset. Each utterance in ASRU contains 8.6 Chinese characters and 1.6 English words on average, and the transcripts cover many common fields, including entertainment, travel, daily life, and social interaction. Similarly AISHELL-1, Chinese LlaMa2-7b is selected as foundation model, and 15k Hypotheses- Hypothesse-TTanscripite Transcription pairs are randomly used in H2T training. Additionally, the evaluation metric for code-switching ASR is mixed error
From the results reported in Table S we observe that H2T-LoRA demonstrates generalization on both Mandarin and code-switching dataset, which respectively achieves 20.6% CER and 24.5% MER reductions. Furthermore, H2T-LoRA exhibits remarkable data efficiency since the training pairs is equivalent to less than 20 hours of speech data.
Table 8: Mandarin and code-switching results on AISHELL-1 (CER) and ASRU (MER) dataset, where 20k and 15k Traathssss-raannriiiition pairs are respectively used in H2T training.
Dataset
Language
Baseline
H2T-LoRA
Onb
Ocp
AISHELL-1
Man
6.3
5.0 20.6%
4.1
3.1
ASRU
Man Eng
11.0
24.5%
8.6
6.5
Results on Spoken Language Intent Detection (SID) task
We first train an intent detection model using the transcription of ATIS training set, as the intent label
available for each example. Then, during testing, we respectively feed the 1st ~ 5th utterances in Whisper hypotheses list, utterance after correction, and ground-truth transcription as input text for intent detection. The accuracy results are reported in Table
Table 9: Accuracy (%) results of intent detection with different input on ATIS test set.
n th
utterance in Hypotheses list, n
Textual input
After Correction
Oracle
2
3
5
Acc. (%)
94.9
95.5
94.2
94.3
94.2
97.4 2..5%
97.9
1lma--2--7
21

---

We observe that: (1) When we use corrected text for intent detection, the accuracy is 97.4% which achieves absolute improvement of 2.5% over 1s utterance in the hypothesis list. (2) 2nd utterance is more suitable for intent detection than 1st utterance in terms of accuracy. This phenomenon validates the case (ii) from a perspective other than WER, where the discarded utterances in the N-best hypotheses might be better than the selected utterance.
Preliminary Results on Zero-shot Prompting- Error Correction
To examine the zero-shot ability of LLM, we propose framework that requires targeted LLM perform either (i) ranking-baas error correction or (ii) single-sentence generative correction. We follow the self-activated prompting method mentioned in the previous appendix section to repurpose the language model in the form of zero-shot eπror correction, without providing instructions. Using the same decoding test set from WavLM and Whisper, 5-gram language model (coefficient of 0.1) combined with its acoustic model score showed a 2.95% WER relative improvement. This result slightly worse than the LMyank baseline. The current limitations on the results regarding the zero-shot abilities of LMs could be attributable the model scale. The zero-shot or emergent abilities of these models have been reported [T00] be more significant when the parameter scale of LLM exceeds 100B.
Hypotheses Paradise (HP) Dataset Visualizations
We have open-sourced Colab example for HP dataset visualizations and analysis. First, same as Fig. M we visualize and analyze the information in N-best hypotheses from both utterance- and token-levels. Fig. 4 illustrate more visualizations on CHiME-4 sets, where we can observe valuable information in N-best hypotheses.
Proportion n-best better than 1-best
Distribution of ground-tree -tuth tokens N-best llst
0.200
0.14
0.150
Proportion
0.10
Probability
0.125
0.08
0.100
0.06
0.075
0.04
0.050
0.02
0.025
0.00
0.000
2
5
5
Figure 4: Probabilities of the case (i) and (ii) CHiME-4 test set, similar the right part of Fig.
Furthermore, we also visualize and compare the word frequency in N-best hypotheses and ground- truth transcription in Fig. 5 where can observe some but limited gap between them.
N-best Hypotheses Distribution
Fig. 6 visualizes the distribution of N-best hypotheses generated by different-sized Whisper models, ie.., from 'tiny' to 'large". We can observe very limited diversity in the N- hypotheses generated by Whisper models. Considering such high monotonicity, we only collect the top-5 hypotheses to form our HP dataset.
Limitations
Though the proposed HP benchmark provides a new paradigm of generative error correction for ASR, we analyze and discuss the limitations of this work from the following perspective:
http:://oooaabbeeearreeee
22

---

Word frequency of N-best hypotheses
frequency ground-truth transcription
0.06
0.05
0.05
0.04
Frequency
Frequency
0.03
0.03
0.02
0.02
0.00
0.00
Word
Word
Figure 5: Top-10 word frequencies in N-best hypotheses and ground-truth transcription of CHiME-4 test set.
N-best Hypotheses Distribution
whose feet the feet of hearts and underneath the everlasting ams (WER: 7.7)
who feet are as the feet of hearts and underneath the everlasting
(WER: 23.1)
medium
whose feet are as the feet of harts and underneath the everlasting
GT
10
N-best
Figure N-best hypotheses distribution with different Whisper models. Each color denotes an unique hypothesis, 'GT" denotes the ground-truth transcription. The sample is selected from LibriSpeech test-clean set, ie., 11089/134991/109- 134691-0005..f
Evaluation metric. As an ASR error correction benchmark, HP employs WER as the primary metric to evaluate the system performance. Nevertheless, prior work [2] has pointed out that WER can be too coarse-grained for describing the performance of ASR models. Furthermore, 8 raise community awareness regarding the problems caused by the optimistic bias toward ASR accuracy. In the future, we aim provide more annotations for spoken language, e.g., entity spans and dependency structure. Accordingly, a comprehensive evaluation framework can established to assess the quality and interpretability of output from the LLM-enhanced ASR system.
Robustness in reality. HP benchmark covers mainstream domains where ASR tasks are usually deployed. However, as shown in [57), single validation or datasets is sufficient to measure transfer to real-world audio data. Since all test sets of HP benchmark are drawn from existing ASR corpus, despite enhancing the WER performance, we are unable to ascertain the extent to which it can mitigate the gap between well- rell-trained ASR models and real-world application scenarios. Furthermore, considering the discrepancy between spoken language and written language, more efforts are required from both speech and NL.P communities to build a human-like robust ASR system beyond single modality
from public
18
Broader Impact
With recent advances in using large-scale neural language models to solve problems once believed to be challenging for machines to learn and understand, we believe it is timely move to the next
23

---

providing publicly accessible n-best hypotheses as transcription resources from LLM decoding. This motivation inspires this work, offering a collection of hypotheses paradise, inspired by in-context learning.
milestone:
Who may benefit from this research: Rescarchers working on speech technology and language model based error correction; as well as the users using the related techniques for responsible and reproducible machine learning technology.
Who may be put at disadvantage from this research: When our work revealed that open- source hypotheses
be used to generate malicious recognition, understood the re- sponsibility of properly explaining the results to the public and providing reproducible evaluations. We have discussed terms of for reusing these hypotheses with legal and regulatory experts, addressing potential risks and concerns.
Whether the task/method leverages biases in the data: To alleviate possible bias in the data and model, have made efforts design reproducible metries and evaluate wide variety of reproducible data sources and training configurations. We have also conducted user studies to highlight potential bias in "terms of use" provided in our Github repo.
Maintenance Plan
Who will be supporingghhoxinggniinttn actively maintained by the authors of this paper. We are still actively updating the dataset that focus on specific ASR scenario, which are noise-robust ASR and multi-lingual ASR. In INTERSPEECH 2023, will have tutorial introdunce the related Hypotheses Paradise-V V2 with some excited experimental results. Furthermore, we also open the link to collect more sypothesis-transcriptioo pairs from public.
the dataset? Hypotheses Paradise has been
of the dataset be contacted? To contact the main devel- opers, we encourage users to use our emails: {chen1436, y0444466yc00005058888 0e ntu edu. sg, huckiyang@gatech. edu.
How can the owwer/curator/manne
Is there erralum? Users can use GitHub to report issues/bugs, and we would actively improve the codes accordingly. We also have a HuggingFace Model card under an non-profit ace.
organization in
ceattaasess/Paacffflll http///aaaaaaaaaaeeeaaaa
Will the dataset be updated? Yes, we are actively updating Hypotheses Paradise codes and data sources. Users could get information and the newly updated version through our GitHub repository.
If the datasel relates to people, are there applicable limits on the relention of the data associated with the instances? No for the Dataset.
Will older versions of the dataset continue be suppotee/hstednniniineld Yes, will keep the old version that generated by Whisper. All versions can be found on our GitHub repository
If others want to extend/aangmen//bull on/contribute to the dataset, is there a mechanism for them to do so? We maintain Hypotheses Paradise on GitHub and we encourage all users to share their ideas to extend Hypotheses Paradise to more speech recognition cases. Users can use GitHub to report issues/bugs, and send us emails to discuss solutions.
Acknowledgement
This research is supported by the National Research Foundation, Singapore under its AI Singa- pore Programme (AISG Award No: AISG2-100E-2022-111 article was partially performed resources of the National Supercomputing Centre, Singapore (htts://www.nscc.gg)) The corresponding author is supported by Wallace H. Coulter Fellowship.
The computational work for this
24


Reviews:
[{"review_id": "Tt9ZB1aIVn", "reviewer": "Reviewer_LMrw", "summary": "", "questions": "", "limitations": "It includes English ASR reduction and could be extended to multi-lingual scenarios.", "rating": 6, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "1. Interesting idea. The author proposes an error reduction method for ASR and leverages the LLMs model for dataset creation, which is a new and interesting idea.\n2. Comparison studies. The authors compare different LLMs and ASR models in various datasets and provide H2T-ft and H2T-LoRA for ASR error correction from hypotheses to transcription.\n3. Evaluation. The authors present in-context learning results with interesting findings and demonstrate the results with distinct error reduction.", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": ""}, {"review_id": "9pyrs3MaFK", "reviewer": "Reviewer_Vw1k", "summary": "", "questions": "", "limitations": "The authors discuss technical limitations and they are discussed sufficiently. They however do not discuss the limitations of doing an english only benchmark.", "rating": 8, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "First of all I'd like to state that the community has long awaited for such a paper to emerge. It's main strength is that it uses LLMs for correct ASR transcripts and evaluating if they do a good job at rescoring hypotheses. This benchmark is well needed because a variety of systems can be tested against it, including both the classical kaldi solutions and seeking how lattices can be embedded with LLM embeddings for rescoring scenarios, and also rule based rescoring injections. The proposed experimental setup with both an in-context learning scheme and a trainining scenario is fullfiling the variety of approaches currently used in academia and industry.", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": ""}, {"review_id": "ha1Yn7hZ7r", "reviewer": "Reviewer_y46i", "summary": "", "questions": "", "limitations": "similar as above reasons", "rating": 7, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "This is timely important work for the community of ASR, showing how to integrate the LLMs (and PLMs) with acoustic models. This paper can be an ASR database/benchmark milestone.\n\nThe novel dataset they introduced, \"Hypotheses Paradise,\" is used to train and test this integrated system. The authors also proposed an in-context learning method to finetune with this dataset on LLMs.\n\nThe proposed system significantly reduces the word error rate (WER), indicating better performance than traditional re-ranking-based methods. As the system even corrects tokens missing in the N-best list.", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": ""}, {"review_id": "rMTdFcfyWY", "reviewer": "Reviewer_4B4v", "summary": "", "questions": "", "limitations": "This paper has a very limited discussion of limitations, and the limitaions are _entirely_ focused on the baseline models, and not the HP benchmark. For a paper which is introducing a new benchmark and dataset, I believe that such as discussion should be significantly more robust, and for the benchmark itself, a limitations section could include discussions of:\n- The sampling process from each of the datasets, and the demographic/dataset balancing performed on the data (if any), and how this might impact the performance of the re-writing models/n-best models.\n- The limitations of pulling data from existing data sources, instead of collecting novel utterances\n- The limitations of focusing on english-only data compared to global applications\n- The limitations (and benefits) of not producing a single output number, and how the dataset is balanced around the samples\n- The limitations of fine-tuning the n-best models on Librispeech, instead of on each dataset specifically\n\nFor example, the authors write in the broader impacts section of the appendix: \"When our work revealed that opensource hypotheses can be used to generate malicious recognition, we understood the responsibility of properly explaining the results to the public and providing reproducible evaluations. We have discussed terms of use for reusing these hypotheses with legal and regulatory experts, addressing potential risks and concerns,\" It would be nice to see some of the discussion, and have additional explanations of what was discussed, and what the potential risks and concerns might be. \n\nSimilarly, there is no discussion of the user studies of potential bias, that was mentioned in the broader impacts section.\n\nFor the models themselves, the limitations are also relatively weak, and do not discuss limitations of using LLMs for speech applications including:\n- Introducing language-only hallucinations during the process (for generative models)\n- Cost of LLMs (monetary, environmental, social)\n- Discussion comparing re-writing to re-ranking, and understanding the differences between these models.", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "A clear strength of the paper is the need for a consistent benchmark for ASR re-ranking. ASR hypothesis re-ranking is a common part of many production ASR systems, and thus, improvements in this field could have significant impact on the perceived quality of ASR systems. Most traditional ASR re-ranking/re-writing methods evaluate on a collection of ad-hoc benchmarks, generated by a wide range of root ASR models, which can make it difficult to directly compare the performance of the methods directly. \n\nThe proposed HP benchmark is fairly large, and contains both training and test sets, which means that non zero-shot methods (such as the H2T-ft and H2T-lora models) can be evaluated on the same data. Further, the HP benchmark covers a fairly large number of domains: (LibriSpeech, CHiME-4, WSJ, SwitchBoard, CommonVoice, TED-lium-3, LRS2, ATIS and CORAAL), with attention to the global nature of data (included several accents in CommonVoice, and the inclusion of CORAAL).\n\nThe introduced model baselines are interesting, and while somewhat limited, demonstrate that there is significant space for potential improvement over baseline approaches (even strong baselines such as Whisper).", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": ""}, {"review_id": "n8cDxP0neH", "reviewer": "Reviewer_7FEE", "summary": "", "questions": "", "limitations": "\n1. This paper is more like a regular paper, not a paper on Track Datasets and Benchmarks. Because this paper focuses on utilizing LLM to improve ASR error correction, and constructing the pairs of N-best hypotheses and accurate transcriptions is one of the step of the method.\n\n2. The paper only compares the proposed method with LM reranking method. It would be better to compare to more ASR error correction approaches.", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "\n1. The authors propose three approaches, including self-activated in-context learning and hypotheses-to-transcription training (H2T-ft and H2T-lora).\n\n2. The dataset is constructed on multiple ASR corpora with different domains on 9 datasets.\n\n3. Experimental results show that the proposed method can significantly reduce the WER of ASR.", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": ""}]

Author rebuttal:

Meta-review: The paper proposes an error correction benchmark for ASR (Hypotheses Paradise) and new baselines set with the use of LLMs (T2H-ft and T2H-Lora), demonstrating how well LLMs perform on the error correction task.

In the speech community we test error correction methods or new methods on rescoring hypothesis by varying the later component but selecting some ASR system, which is not standardized in the community, thus hard to compare between different works. Proposed benchmark will allow researchers to focus only on error correction methods, including investigation of how LLMs can perform on this task / can be used and integrated, to have fair comparison in future. However, in future the benchmark should be updated based on the more advanced general ASR development as this will change the type of error we need to correct.

All Reviewers are positive about the dataset and benchmarking of LLMs. They raised several important concerns which authors addressed mostly during rebuttal and included already in the updated version: e.g. extending to more languages (not only English), details on data collection and sampling, comparison with prior models on error correction, the beam size choice, improvement of documentation / github / dataset card / evaluation scripts, explaining issue with the license (related to the use of fine-tuning code, while dataset is under MIT), the choice of ASR systems used to generate dataset.

I agree with Reviewers that it is a "timely important work for the community of ASR, showing how to integrate the LLMs (and PLMs) with acoustic models." and moreover considered baselines (in-context learning scheme and a training scenario) are widely used methods in academia and industry.

I also would recommend authors to include all promised things (most are incorporated already), e.g. limitation section should be extended with discussion on the data collection itself, and possible issues that LLMs were trained on the ASR transcriptions in advance, so that future evaluations are done carefully with this.
Decision: Accept (Poster)