SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Chen Chen $^{1,\dagger}$  Yuchen Hu $^{1,\dagger}$  Chao-Han Huck Yang $^{2^{*},3}$
Sabato Marco Siniscalchi $^{2,4}$  Pin-Yu Chen $^{5}$  Eng Siong Chng $^{1}$
$^{1}$ Nanyang Technological University  $^{2}$ Georgia Institute of Technology  $^{3}$ NVIDIA Research
$^{4}$ Norwegian University of Science and Technology  $^{5}$ IBM Research AI
{chen1436,yuchen005}@e.ntu.edu.sg huckiyang@gatech.edu
Advancements in deep neural networks have allowed automatic speech recognition (ASR) systems to attain human parity on several publicly available clean speech datasets. However, even state-of-the-art ASR systems experience performance degradation when confronted with adverse conditions, as a well-trained acoustic model is sensitive to variations in the speech domain, e.g., background noise. Intuitively, humans address this issue by relying on their linguistic knowledge: the meaning of ambiguous spoken terms is usually inferred from contextual cues thereby reducing the dependency on the auditory system. Inspired by this observation, we introduce the first open-source benchmark to utilize external large language models (LLMs) for ASR error correction, where N-best decoding hypotheses provide informative elements for true transcription prediction. This approach is a paradigm shift from the traditional language model rescoring strategy that can only select one candidate hypothesis as the output transcription. The proposed benchmark contains a novel dataset, "HyPoradise" (HP), encompassing more than 334,000 pairs of N-best hypotheses and corresponding accurate transcriptions across prevalent speech domains. Given this dataset, we examine three types of error correction techniques based on LLMs with varying amounts of labeled hypotheses-transcription pairs, which gains a significant word error rate (WER) reduction. Experimental evidence demonstrates the proposed technique achieves a breakthrough by surpassing the upper bound of traditional re-ranking based methods. More surprisingly, LLM with reasonable prompt and its generative capability can even correct those tokens that are missing in N-best list. We make our results publicly accessible for reproducible pipelines with released pre-trained models, thus providing a new evaluation paradigm for ASR error correction with LLMs.
Automatic speech recognition (ASR) has become increasingly important in modern society, as it enables efficient and accurate transcription of spoken languages. This capability facilitates access to information and enhances communication across various domains, including education [7], healthcare [50], and business [36]. Driven by the recent advances in deep learning, remarkable success has been achieved on several ASR tasks through end-to-end training techniques [28, 27, 9, 22, 30, 100, 15]. However, a major challenge of applying ASR in practical conditions lies in effectively handling variations in speech caused by different factors such as background noise [11], speaker accent [85], and speaking styles [82, 2]. These adverse factors are common and inevitable in speech signal, significantly affecting the accuracy of the recognition results [55].
*Work done & open source while the author was at Georgia Tech; Corresponding author. †Equal contribution.
Figure 1: The left part shows the pipeline to generate the N-best hypotheses using a vanilla ASR engine with beam search decoding. The right part counts the probabilities of case (i) and case (ii) on the test set of LibriSpeech dataset. It indicates the discarded information in  $2^{nd} \sim 20^{th}$  utterances. Green and red  $T_{i}$  in "Exp" respectively denote correct and wrong tokens compared with ground-truth.
Humans demonstrate remarkable robustness when faced with the above variations in acoustic environment, as the human recognition system does not only rely on acoustic cues – we usually speculate the ambiguous or distorted spoken terms based on speech context and our inherent linguistic knowledge. Similarly, current ASR system typically employs an independent language model (LM) for rescoring during the decoding process [83, 46, 43, 25]. As shown in Fig. 1, given N-best hypotheses generated by an ASR engine with beam search decoding, a trained language model (LM) can be used to re-score each utterance and select the one with the highest likelihood (referred to as the  $1^{st}$  utterance) as the output of the ASR; whereas, the other sentences (the  $2^{nd} - N^{th}$  utterances) are discarded. However, it is widely believed [68] that the N-best list contains useful information [87, 37, 56], as each hypothesis is an independent textual representation of the input speech. Consequently, discarded sentences might also carry correct tokens for accurately predicting the true transcription. To validate this belief, we have conducted experiments on the LibriSpeech dataset [66], counting the probabilities of two scenarios observed during LM rescoring: (i) the discarded utterances contain a better candidate with lower word error rate (WER), and (ii) the other discarded hypotheses can provide the right answer for the wrong tokens in  $1^{st}$  utterance. The statistical results of  $2^{nd} \sim 20^{th}$  utterances are shown in the left part of Fig. 1. Taking  $2^{nd}$  discarded utterance as example, it has a  $14\%$  probability of having a lower WER than the  $1^{st}$  utterance. Furthermore, given a wrong token in  $1^{st}$  utterance, there is a  $34\%$  probability of finding the correct token in the  $2^{nd}$  utterance.
To better mine the information in N-best hypotheses, we propose the first attempt on publicly available ASR generative error correction benchmark that directly predicts a true transcription, rather than selecting a candidate from the N-best list. To put forth this benchmark, we introduce a novel dataset named HyPoradise (HP), which comprises various open source N-best hypotheses provided by state-of-the-art ASR systems and their paired true transcriptions. Considering real-life applications, HP dataset covers various challenging speech domains, including scenarios with background noise, specific contexts, and speaker accents. Furthermore, in terms of resources availability, we define three settings to mimic the deployment of ASR systems in real-world scenarios: (i) Zero-shot Learning. In this setting, only test set hypotheses are available for inference. This corresponds to applying a well-trained ASR model to new scenarios without any training data. (ii) Few-shot Learning. A few in-domain hypotheses with true transcription are available for training. This setting aims to address domain-specific ASR tasks with a few manual annotations. (iii) Fine-tuning. A sufficient training set is available to learn the mapping between hypotheses and transcription.
To exploit the three aforementioned scenarios, we present multiple error correction techniques using large language models (LLMs), which have shown the outperforming ability of language generation and reasoning in recent studies [5, 107, 48, 84]. For zero-shot and few-shot settings, we design an in-context learning method without any parameter tuning, which directly performs error correction based on task prompt and in-domain demonstration. In the fine-tuning scenario, we develop two sequence-to-sequence training solutions, H2T-ft and H2T-LoRA, which adapt pre-trained LLMs to specific transcription domains. Experimental results show that all learning strategies can be beneficial to reduce the WER in different resource settings, providing potential solutions for alleviating the
negative impact of speech variation. Additionally, with reasonable prompt design, LLMs can correct those specious tokens that are exclusive from N-best list. We will release the HP datasets, reproducible pipelines, and pre-trained models on Github  ${}^{2}$  under MIT licence.
Our contribution can be summarized as follows:
- We propose the first open and reproducible benchmark to evaluate how LLMs can be utilized to enhance ASR results with N-best hypotheses, where a new dataset HyPoradise  $^{3}$  with more than 334K hypotheses-transcription pairs are collected from the various ASR corpus in most common speech domains.
- We develop three ASR error correction techniques based on LLMs in different resource settings to directly predict the true transcription from the N-best hypotheses. Experimental results in the fine-tuning setting show that our new approach can surpass a performance upper-bound (e.g., oracle WER from n-best list) of traditional re-ranking based methods.
- We introduce an evaluation paradigm of generative error correction for ASR. The acoustic model generates word-piece elements in the hypotheses list; subsequently, LLMs predict accurate transcription utilizing linguistic knowledge and contextual information.
2.1 ASR Rescoring and Error Correction
In order to improve the linguistic acceptability of ASR results, LM rescoring has been widely employed and achieved stable performance gain for ASR systems [79, 62, 4]. Typically, an external LM is trained separately and utilized to re-score the N-best list of hypotheses generated by ASR decoding with beam search. Various approaches for LM integration have been proposed, such as shallow fusion [17, 104, 46, 83], deliberation [98, 32, 41, 40, 91, 39], component fusion [76], and cold fusion [81]. Some authors have used pre-trained LM models to replace trainable LMs [86, 74], and the log-likelihood of each hypothesis is computed using unidirectional models, e.g., GPT-2, or pseudo-log-likelihood using bidirectional models like BERT [21] and RoBERTa [59]. In ASR, LMs are also widely used for the error correction task in different languages [96, 29], leveraging only the 1-best hypothesis generated by the ASR model [53, 61, 106, 23, 109, 77]. Furthermore, more recent works [60, 52, 51] utilize a candidates list after decoding for error correction. Though Grammatical Error Correction (GEC) has been actively explored [20, 93, 100], ASR error correction is distinct with GER due to the arbitrariness of the spoken language [2], which requires the efforts from both speech and NLP communities [18].
2.2 Large Language Models
More recently, there has been a surge of interest in Transformer-based LLMs [84, 70, 75, 107] in both academia and industry. By learning from massive amounts of text data, LLMs can capture linguistic patterns and semantic relationships, which have led to impressive performance for a wide range of natural language processing (NLP) tasks [5, 65, 95].
In-context Learning. Given specific task descriptions or pair-wise contextual information, LLMs show outstanding adaptability on downstream NLP tasks without any parameter tuning [63, 64, 100]. Such a capability of task-specific inference is also known as in-context learning (ICL) [99], which utilize LLMs to generate text that is more coherent and relevant to the specific domain or task [44, 16, 49, 73, 8, 108]. Recently, task-activating Prompting (TAP) [100] is one of the most relevant works, employing the injection of input-output pairs of task-oriented contexts (e.g., initiating the question prompt from a broad domain to refine preceding contexts as shown in Figure 2) with the aim of enhancing the zero-shot and few-shot capabilities of frozen-pretrained LLMs for second-pass ASR. We further evaluate the TAP-based zero-shot and few-shot approaches with examples.
Low-rank Approximation based Neural Adapter. Tuning all LLM parameters for a given downstream task is usually not feasible due to memory constraints. Many researchers sought to mitigate that problem by either adapting only a few parameters or leveraging external trainable modules for
$^{2}$ https://github.com/Hypotheses-Paradise/Hypo2Trans  
 $^{3}$ Denoted as Hypotheses Paradise, inspired by "Icha Icha Paradise" from Naruto.
a new task [58, 33]. A pioneer work [1] showed that the learned over-parametrized models in fact reside on a low intrinsic dimension, consequently, a low-rank adaptation (LoRA) approach [38] was proposed to indirectly tune some dense layers by optimizing rank decomposition matrices of the dense layers. Due to its computational efficiency, LoRA adaptation has been rapidly adopted as a new paradigm for LLMs tuning, which was useful in various downstream tasks [105, 24, 42, 92].
3 Hypothesis Generation and Dataset Creation
We introduce the generation process of the HyPoradise dataset in this section. The employed ASR system for N-best hypotheses generation is illustrated in 3.1, and then we introduce the selected speech domain in 3.2. Finally, we provide statistic information and generated HP in 3.2.
3.1 ASR System
We employ two state-of-the-art ASR models, namely WavLM [14] and Whisper [69] for N-best hypotheses generation. Besides their remarkable performance and popularity, those models are representative in the deployment of an ASR because: (1) WavLM is a well-trained ASR model on LibriSpeech [66] but suffering from domain mismatch, and (2) Whisper is a universal ASR model but lacking domain specificity. More details about those two ASR models are described below:
WavLM: We utilize the ESPnet toolkit [94] along with the pre-trained model from HuggingFace to deploy our WavLM-based ASR system. The WavLM architecture consists of two blocks: the front-end, and the ASR model (433 million parameters in total). The front-end consists of 24 Transformer-based [88] encoder layers and is pre-trained using a combination of LibriLight [45] (60k hours of data), Gigaspeech [12] (10k hours of data), and VoxPopuli [90] (24k hours of data). Front-end features are fed into the ASR back-end for fine-tuning. The back-end consists of 12 Conformer-based [30] encoder layers, and 6 Transformer-based decoder layers. The fine-tuning process is performed on 960-hour LibriSpeech data. Additionally, the WavLM decoding recipe incorporates an external LM rescoring option, where the external LM adopts Transformer architecture with 16 encoder layers and is trained using the text of LibriSpeech 960 hours data and extra LM training data from the web.
Whisper: We employ the Whisper-Large model developed by OpenAI to generate hypotheses, without in-domain language model rescoring. The used configuration consists of an encoder-decoder Transformer architecture with 1,550 million parameters, which is trained on 680,000 hours of multilingual-weakly labeled speech data collected from the web.
Leveraging these two pre-trained ASR models, we have employed the beam search algorithm during decoding and generated N-best lists of sentence hypotheses for each input waveform. For both WavLM and Whisper, the default beam size was set to 60. After removing repeatable utterances, we select top-5 utterances with highest probabilities as N-best list, as they have carried sufficient elements to accurately predict transcription. Subsequent experiments confirm this belief by calculating the accurately upper-bound WER using 5-best hypotheses list. To build the HP dataset, we carry out this decoding strategy on multiple popular ASR datasets (please see Section 3.2) and generate paired data consisting of an 5-best hypotheses list and 1 ground-truth transcription. The pre-processing and generation code are also released for integrating new ASR corpus into HP. All the links of relevant resources are presented in Appendix.
3.2 Selected Speech Corpora
For corpora selection, our goal is to cover common scenarios of ASR task, e.g., noisy background and speaker accent. Consequently, we collect and modify the following corpora with evident domain characteristics to compose the HP dataset.
LibriSpeech [66]: LibriSpeech is a public corpus of read speech from audiobooks, including 1,000 hours of speech data with diverse speakers, genders, and accents. For generating HP training data, we exclude some simple cases from its train-960 split that show WER result of 0, resulting in 88,200 training utterances. We use the entire test-clean and test-other splits for HP test data generation.
CHiME-4 [89]: CHiME-4 is a dataset for far-field speech recognition. It includes real and simulated noisy recordings in four noisy environments, i.e., bus, cafe, pedestrian area, and street junction. We
Table 1: HP dataset statistics in terms of the number of hypotheses-transcription pairs and average utterance length in various domains.
Domain	Training Set	# Pairs	Length	Test Set	# Pairs	Length
Source	Category
LibriSpeech	Audiobooks	train-960	88,200	33.7	test-clean	2,620	20.1
test-other	2,939	17.8
CHiME4	Noise	train	8,738	17.0	test-real	1,320	16.4
WSJ	Business news	train-si284	37,514	17.5	dev93	503	16.7
eval92	333	17.3
SwitchBoard	Telephone	train	36,539	11.8	eval2000	2,000	11.8
CommonVoice	Accented English	train-accent	49,758	10.5	test-accent	2,000	10.5
Tedlium-3	TED talk	train	47,500	12.6	test	2,500	12.6
LRS2	BBC audio	train	42,940	7.6	test	2,259	7.6
ATIS	Airline info.	train	3,964	12.4	test	809	11.3
CORAAL	Interview	train	1,728	24.2	test	100	24.0
Total	train	316,881	18.1	test	17,383	14.1
use its train (with 8,738 utterances) and test-real (with 1,320 utterances) splits to generate HP training and test data. The four different noises in test-real split are also evaluated separately in Table 3.
WSJ [67]: The Wall Street Journal (WSJ) is a widely-used benchmark for speech recognition. It includes read speech from speakers in a controlled environment, with a focus on business news and financial data. We use its train-si284 split (with 37,514 utterances) to generate HP training set. The dev93 (with 503 utterances) and eval92 (with 333 utterances) are applied to build test sets.
SwitchBoard [26]: The SwitchBoard corpus is a telephone speech dataset collected from conversations between pairs of speakers. It focuses on North American English and involves over 2.4k conversations from approximately 200 speakers. We randomly select 36,539 samples from its train split to generate HP training set, as well as 2,000 utterances from the eval2000 split for HP test set.
CommonVoice [3]: CommonVoice 5.1 is a freely-available dataset for speech recognition. It contains speech recordings from diverse speakers in over 60 languages. To generate HP dataset, we randomly select 51,758 samples from its train-en split with accent labels, i.e., African, Australian, Indian, and Singaporean, where training set contains 49,758 samples and test set contains 2,000 samples.
Tedlium-3 [35]: Tedlium-3 is a dataset of speech recorded from TED Talks in multiple languages. It contains a diverse range of background noise, speaker accents, speech topics, etc. Considering its large size, we randomly select 50,000 samples from its train split for HP dataset generation, where training set contains 47,500 samples and test set contains 2,500 samples.
LRS2 [19]: Lip Reading Sentences 2 (LRS2) is a large-scale publicly available labeled audio-visual dataset, consisting of 224 hours of video clips from BBC programs. We randomly select 42,940 samples from its train split as training set, and the remaining 2,259 samples are used for test set.
ATIS [34]: Airline Travel Information System (ATIS) is a dataset comprising spoken queries for air travel information, such as flight times, prices, and availability. It contains around 5,000 to 5,400 utterances, which are recorded from around 500 to 550 speakers.
CORAAL [47]: The Corpus of Regional African American Language (CORAAL) is the first public corpus of AAL data. It includes audio recordings along with the time-aligned orthographic transcription from over 150 sociolinguistic interviews. To generate HP dataset, we select 1,728 samples as training set and 100 samples as test set.
3.3 HyPoradise (HP) Dataset Statistics
After performing beam search decoding on the selected speech datasets introduced in Section 3.2, we collected more than  $334\mathrm{K}$  pairs of hypotheses list and transcription to form the HP dataset, including training and test sets. The statistics for the HP dataset are given in Table 1, which shows the number
Figure 2: A scalable evaluation of Task-Activating Prompting [100] (TAP) based in-context learning. The demonstration in blue box is drawn from the training set, which is optional for LLMs input.
Figure 3: (a) Structure of H2T-  $f t$ . (b) Reparametrization in H2T-LoRA. Solid box denotes the module is fixed during tuning while dashed box stands for trainable. Blue color denotes the weights has been pre-trained on another dataset.
of pairs and average length in various domains and splits. We would release our generated datasets and kindly call for more hypotheses-transcription pairs toward sustainable community efforts.
4 ASR Error Correction from Hypotheses to Transcription
We hereby introduce a hypotheses-to-transcription (H2T) training scheme utilizing the collected HP dataset to enhance ASR performance with LLM integration. With limited labeled data, in-context learning [100] is employed to form task-specific prompts and in-domain demonstrations: Linguistic knowledge in LLM is exploited without parameter tuning. Furthermore, we present two trainable methods fine-tuning  $(ft)$  and H2T-LoRA to learn the hypotheses-to-transcription mapping when a sufficient amount of labeled data is available.
4.1 Hypotheses-to-Transcription (H2T) Training
In addition to in-context learning, we introduce two parameter-tunable methods to learn hypotheses-to-transcription mapping in a sequence-to-sequence manner: H2T-ft and H2T-LoRA.
H2T-ft denotes fine-tuning all parameters of a neural model with labeled data of each HP domain. Specifically, we introduce a similar method with N-best T5, which utilizes other hypotheses to improve the 1-best hypothesis as shown in Fig. 3. To constrain the decoding space, we add an new item criterion  $\mathcal{L}_{ft} = \sum_{i=1}^{N} \alpha_i \log P(x^{(i)}|x,\theta)$ , where  $x^{(i)}$  is the  $i$ -th hypothesis in N-best list. This item aims to encourage the correction model to preferentially consider tokens into the N-best hypotheses list, preventing arbitrary modification in huge decoding space.  $\alpha_i$  is a hyper-parameter for  $i$ -th hypothesis that decreases with the order ranked by the acoustic model.
H2T-LoRA avoids tuning the whole set of parameters of a pre-trained model by inserting a neural module with a small number of extra trainable parameters to approximate the full parameter updates, allowing for efficient learning of the H2T mapping without affecting the pre-trained parameters of the LLM. H2T-LoRA introduces trainable low-rank decomposition matrices into LLMs' existing layers, enabling the model to adapt to new data while keeping the original LLMs fixed to retain the previous knowledge. Specifically, LoRA performs a reparameterization of each model layer expressed as a matrix multiplication by injecting low-rank decomposition matrices (Fig.3 (b)). As a result, the
Table 2: WER (%) results of H2T-ft and H2T-LoRA in fine-tuning setting. "  $o_{nb}$ " and "  $o_{cp}$ " respectively denote n-best oracle and compositional oracle that are defined in 5.2.
Test Set	Baseline	LMrank	H2T-ft	H2T-LoRA	Oracle
T5	LLaMA	T5	LLaMA	onb	ocp
WSJ	4.5	4.3	4.0	3.8	2.7-40.0%	2.2-51.1%	4.1	1.2
ATIS	8.3	6.9	2.7	3.4	1.7-79.5%	1.9-77.1%	5.2	1.1
CHiME-4	11.1	11.0	7.9	8.2	7.0-36.9%	6.6-40.5%	9.1	2.8
Tedlium-3	8.5	8.0	6.6	5.2	7.4-12.9%	4.6-45.9%	3.0	0.7
CV-accent	14.8	16.0	12.9	15.5	11.0-25.7%	11.0-25.7%	11.4	7.9
SwitchBoard	15.7	15.4	15.9	18.4	14.9-5.1%	14.1-10.2%	12.6	4.2
LRS2	10.1	9.6	9.5	10.2	6.6-34.7%	8.8-12.9%	6.9	2.6
CORAAL	21.4	21.4	23.1	22.9	20.9-2.3%	19.2-10.3%	21.8	10.7
representations generated by the LLM are not distorted due to task-specific tuning, while the adapter module acquires the capability to predict the true transcription from the N-best hypotheses.
Benefiting from efficient training, we can employ a large-scale language model in the H2T-LoRA method, which is expected to understand the task description and capture correlation in the N-best list. Meanwhile, instead of adding an extra training objective in H2T-ft, we constrain the decoding space of H2T-LoRA by adding requirement in task description.
5.1 Language Models Configurations