SlowGuess/NeurIPS_2023_OCR huggingface dataset:
T5 (0.75B~3B): T5 family [72] is a set of encoder-decoder models pre-trained on a multi-task mixture of unsupervised and supervised tasks and for which each task is converted into a text-to-text format. T5 works well on a variety of tasks out-of-the-box by prepending a different prefix to the input corresponding to each task, e.g., for machine translation or text summarization. In this paper, we select T5-large (0.75B) as the correction model in H2T-ft method.
LLaMA (7B~65B): Proposed by Meta AI, LLaMA [84] is a collection of foundation language models ranging from 7B, 13B, 30B, and 65B parameters. It is trained on publicly available datasets exclusively, and shows remarkable efficiency on NLP benchmarks. We select LLaMA-13B for LoRA adaptation in H2T-LoRA method as one best setup under ablations.
GPT-3.5 (175B): Proposed by OpenAI, GPT-3.5-turbo is one of the most advanced large language models, which powers the popular ChatGPT. It has been optimized from the GPT-3 [5] for chat purposes but works well for traditional completions tasks as well. We utilize GPT-3.5-turbo in task-activated in-context learning [100], which conduct zero-shot and few-shot learning experiments with designed task prompt.
5.2 Training and Evaluation
For few-shot settings, the specific task prompts, along with the LLM's responses from task-activated ICL prompting [100], are provided in the Appendix (page 20). For fine-tuning setting, the detailed configuration of H2T-ft and H2T-LoRA are also explained in Appendix. Furthermore, we release some of the pre-trained correction models to allow interested readers to reproduce our results.
We report WER results as the evaluation metric for all methods. Additionally, we report the two oracle WER for comparison, which are 1) the n-best oracle  $o_{nb}$ : WER of the "best candidate" in N-best hypotheses list, and 2) the compositional oracle method  $o_{cp}$ : achievable WER using "all tokens" in N-best hypotheses list. The  $o_{nb}$  can be viewed as upper bound performance of the re-rank based method, while  $o_{cp}$  denotes the upper bound of correction using occurred elements in the list.
Table 3: Cross-domain WER results by task-activated ICL [100] in zero-shot and few-shot settings. "  ${o}_{nb}$  " and "  ${o}_{cp}$  " respectively denote n-best oracle and compositional oracle that are defined in 5.2.
Domain Shift	Test Set	Baseline	n-shot In-Context Learning (ICL)	Oracle
n = 0	n = 1	n = 5	n = 10	onb	ocp
Specific Scenario	WSJ-dev93	9.0	8.5–5.6%	7.8–13.3%	7.7–14.4%	7.1–21.1%	6.5	5.3
WSJ-eval92	7.6	7.3–3.9%	6.6–13.2%	6.6–13.2%	6.3–17.1%	5.5	4.7
ATIS	5.8	5.5–5.2%	5.1–12.1%	5.0–13.8%	4.7–19.0%	3.5	2.4
Common Noise	CHiME4-bus	18.8	17.6–6.4%	16.7–11.2%	16.2–13.8%	15.9–20.7%	16.8	10.7
CHiME4-caf	16.1	14.7–8.7%	14.3–11.1%	13.7–14.9%	13.2–18.0%	13.3	9.1
CHiME4-ped	11.5	10.9–5.2%	9.9–14.4%	9.7–15.7%	9.4–18.3%	8.5	5.5
CHiME4-str	11.4	10.9–4.4%	10.0–12.3%	9.7–14.9%	9.2–19.3%	9.0	6.0
Speaker Accent	CV-af	25.3	24.9–1.6%	24.2–4.3%	23.6–6.7%	22.6–10.7%	23.6	21.7
CV-au	25.8	25.1–2.7%	24.1–6.6%	24.0–7.0%	23.3–9.7%	24.9	21.8
CV-in	28.6	27.6–3.5%	25.6–10.5%	25.0–12.6%	24.4–14.7%	27.1	22.6
CV-sg	26.4	26.5+0.4%	25.0–5.3%	25.1–4.9%	23.7–10.2%	25.5	22.2
Table 4: Case study of ICL. The utterance is drawn from WSJ-dev93 dataset.
Type	Utterance	WER
1stHypo. by AM	Bankers in Hong Kong expect xinnepec to return for more loans as it develops China's petro chemical industry.	16.7
2ndHypo. by AM	Bankers in Hong Kong expect xinepec to return for more loans as it develops China's petrochemical industry.	8.3
Correction by LLM	Bankers in Hong Kong expect Sinopec to return for more loans as it develops China's petrochemical industry.	0
Ground-truth Transcription	Bankers in Hong Kong expect Sinopec to return for more loans as it develops China's petrochemical industry.	-
5.3 Results of H2T-ft and H2T-LoRA
We first report the WER results for H2T-ft and H2T-LoRA in the fine-tuning setting, where the training set of HP is available to learn H2T mapping. Whisper is employed as acoustic model for hypotheses generation, and a vanilla language model  $LM_{rank}$  is trained using in-domain transcription of the training set, and then it re-ranks the hypotheses according to perplexity. From Table 2, we observe that 1) correction techniques achieve significant performance gain in specific scenarios, where H2T-LoRA respectively reduces  $77.1\%$  and  $55.1\%$  relative WER on ATIS and WSJ. 2) WER performances on CHiME-4 and CV-accent demonstrate proposed correction methods improves the robustness of on background noise and speaker accent. Additionally, H2T-LoRA on these two datasets both surpass the upper-bound of re-ranking based method referring to  $o_{nb}$ . 3) In general, H2T-LoRA usually generate better WER results than H2T-ft, as the low-rank adapter allows LLMs to keep pre-trained knowledge and avoid over-fitting problem.
Limitation and Failure Studies. We notice that an over-fitting phenomenon existing in our correction techniques, especially in H2T-  $ft$  where all parameters are tunable. Furthermore, the mean and variance of the utterance length can potentially influence the WER result, since H2T-  $ft$  results on CORAAL (long-form speech) and SwitchBoard (large variance in length) both fail to enhance ASR performance. On LibriSpeech, when the WER is low (1.8% by WavLM), there is less room to correct recognition errors with proposed framework. The experimental results and list the representative failure cases can be found in Appendix Table 6 and Table 7. Given the evidence of ample room for further performance improvement, our proposal thus serves as an appropriate benchmark to assess the contribution of current and future LLMs to ASR.
5.4 In-context Learning Results
We conduct in-context learning experiments in the practical scenario when a well-trained ASR system encounters domain mismatch. To this end, the WavLM is selected as the in-domain acoustic
model, and GPT-3.5 serves as the LLM for correction. We mainly consider common domain shifts of application: specific scenario, common background noise, and speaker accent, where 5-best hypotheses are selected as context input. From Table 3, we can observe that: (1) Without any in-domain data, LLM can benefit from ASR results based on the hypotheses list. This performance gain mainly relies on the linguistic knowledge of LLM and task-activating [100] descriptions (e.g., chains of task hints) in pipeline. (2) A few in-domain pairs effectively enhance the performance gain in terms of WER. From the final output of the reasoning process, we find that LLM attempts to summarize the regulation from the demonstration and then apply it to the given test example. (3) Leveraging the vast knowledge base, LLM can even correct missing tokens that are exclusive from hypotheses list in terms of context information.
To illustrate the third observation, we conduct the case study on WSJ-dev93 in Table 4. According to the ground-truth transcription, two errors (shown as red) are included in  $1^{st}$  hypothesis, where "petro chemical" is wrongly recognized as two tokens perhaps due to the speaking style of the speaker. LLM correct this error since "petrochemical" can be found in  $2^{nd}$  hypothesis. However, "Sinopec" is unseen during ASR training, leading it to be recognized as weird tokens ("xinnepec" or "xinepec") in hypotheses. In this case, LLM shows human-like correction - it successfully infers the correct token based on the pronunciation of "xinnepec", as well as the context of "China's petrochemical". In fact, Sinopec is a petrochemical-related Chinese company.
5.5 Additional Discussion
Effect on Spoken Language Intent Detection. We examine the effect of error correction on a downstream task of spoken intent detection [80] (SID). To this end, we reproduce an BERT-based SID model [13] and respectively feed the 1-best utterance and corrected utterance by H2T-LoRA for comparison. The ablation results on ATIS dataset are reported in Appendix, which shows that our correction technique can also benefit to SID task in terms of detection accuracy. (3) LLM correction based on N-best hypotheses can effectively enhance the downstream SIT result, which achieves comparable accuracy with using ground-truth transcription (97.4% v.s. 97.9%).
Zero-shot Prompting Results. We finally report an initial prompting evaluation on CHiME-4 in zero-shot setting. Considering the task difficulty, T5 and LLaMA are employed for hypothesis correction. For comparison, we also provide the correction results using a far smaller GPT-2 (1.5B) with a 5-gram LM baseline trained by in-domain transcription. We used LLaMA 13B to perform these zero-shot error correction tasks. Using the test set extracted from Whisper, we observed that the zero-shot method did not yield improved results on CHiME-4  $(11.5 \pm 0.5\%)$  and CV-accent  $(14.9\% \pm 1.5\%)$ . This zero-shot pipeline performed less stably on the other test set discussed in Table 2, which we consider a failure case with a standard deviation exceeding an absolute value of  $10\%$  in terms of WER. For T5-based error correction, we noticed that the method also failed to perform zero-shot error correction by using 0.75B.
Future work. We find that LLMs potentially perceive acoustic information during pre-training, as they tend to perform error correction using tokens with similar pronunciation. Therefore, our first future work is including more acoustic information in HP dataset, such as token-level confidence provided by ASR engine. Furthermore, considering different data amount of each domain, more parameter-efficient training methods besides low-rank adaptation should be discussed for LLMs tuning [54], e.g., model reprogramming [102, 31], prompting [10] and cross-modal adaptation [97, 101, 71].
To explore the benefits in speech-language co-learning, this work introduces a new ASR benchmark that utilizes LLMs for transcription prediction from N-best hypotheses. Our benchmark contains a new HP dataset consisting of more than 334K hypotheses-transcription pairs that are collected from 9 different public ASR corpora. In few-shot settings, we demonstrate that LLMs with in-context learning can serve as a plug-and-play back end to effectively alleviate domain shift of ASR. In the fine-tuning setting, our proposed error correction technique based on LLMs achieves betterWER performance than the upper-bound of re-ranking based method, which provides a new paradigm for applying ASR in some challenging conditions, such as background noise and speaker accent. We believe our benchmark and findings provide new and unique insights into LLM-enhanced ASR.
- Did you include the license to the code and datasets? [Yes]
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope? [Yes]
(b) Did you describe the limitations of your work? [Yes] See Section 5.3
(c) Did you discuss any potential negative societal impacts of your work? [N/A]
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
(a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)? [Yes] See Section 3.
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [Yes] See Section 5.1.
(c) Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)? [N/A]
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)? [Yes]
(a) If your work uses existing assets, did you cite the creators? [Yes] See Appendix.
(b) Did you mention the license of the assets? [Yes] See Appendix.
(c) Did you include any new assets either in the supplemental material or as a URL? [Yes]
(d) Did you discuss whether and how consent was obtained from people whose data you're using/curating? [N/A]
(e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content? [N/A]
Have LLMs seen ASR transcription during pre-training?
We raise a concern about LLM correction: is the remarkable correction result owing to the fact that the ASR transcriptions exist in the pre-trained corpus of LLM?
To evaluate this concern for GPT-3.5, we manually create some hypotheses to simulate the utterance in case study in Table 4, and we show the correction results in Table 5.
Table 5: Synthetic hypotheses for LLM correction
ID	Utterance
1	Patrue China and xinnepec own all national oil pipeline infrastructure.
Corrected item: "Patrue China" → "PetroChina", "xinnepec" → "Sinopec"
2	The Legend of Saida is the most anticipated game of this year from nine ten door.
Corrected item: "Saida" → "Zelda", "nine ten door" → "Nintendo"
3	The world error rate is the primary matrix to evaluate ASR performance.
Corrected item: "world" → "word", "matrix" → "metric"
4	He missed the mid nine flight from Singapore to rose island.
Corrected item: "mid nine" → "midnight", "rose" → "Rhodes"
We observe that given arbitrarily fabricated utterances in different scenarios, LLM can still give reasonable correction results, especially on specific terms. Furthermore, we attach the LLM reasoning process for two examples, which shows the utilization of context information:
ID 1: Please note that the transcription assumes "Patrue China" was an error, and "PetroChina" was intended instead. Additionally, "xinnepec" has been interpreted as "Sinopec," which is another major Chinese oil and gas company.
ID 2: Please note that there may have been some confusion with the phrase "nine ten door" as it doesn't match any known company or widely recognized term. As a result, I have inferred that it may have been a misinterpretation or a typographical error. The transcription assumes "Zelda" as a possible replacement for "Saida" based on its similarity to the well-known game series "The Legend of Zelda" developed by Nintendo.
In H2T-ft, the T5 model was pre-trained on C4 dataset that does not include WSJ, but it still achieves significantWER reduction on WSJ. In H2T-LoRA, we do not find the any ASR transcript exists in the pre-trained data of LlaMa according to their papers or public information.
In general, we argue that the improved results of LLMs correction primarily stem from its extensive knowledge base and the ability to utilize context information as part of a formal scientific experimental process. However, the methods through which indirect knowledge is captured and propagated across a large area remain open questions for future investigation. We hope that by initiating this study, the results would encourage more further and ongoing works on knowledge grounding in the future.
Resources for Hypotheses Paradise (HP) Generation
The related toolkit and pre-trained models used to generate HP dataset are listed as follows:
- For ASR model, we used pre-trained WavLM from HuggingFace and the ESPNet toolkit.
- When using WavLM for ASR decoding, we also incorporate an external LM rescoring option, which is trained using extra text data from web<sup>6</sup>.
- Apart from WavLM, we also select Whisper-LargeV27 for ASR decoding.
4https://huggingface.co/espnet/simpleoier_librispeech_asr_train_asr_conformer7_wavlm_large_raw_en_bpe5000_sp
<sup>5</sup>https://github.com/espnet/espnet/tree/master/egs2/librispeech/asr1
<sup>6</sup>http://www.openslr.org/resources/11/librispeech-lm-norm.txt.gz
<sup>7</sup>https://github.com/openai/whisper
Task-Activated Prompting (TAP) for Error Correction Evaluation
We illustrate the given prompt for task-activated prompting [100] (TAP), as well as the response from the frozen pretrained LLM for each turn, as proposed by Yang et al. [100] in a recent work.
$Q$ : Are you familiar with speech recognition?
R: Yes, I am familiar with speech recognition. Speech recognition, also known as automatic speech recognition (ASR) or speech-to-text, is the process of converting spoken language into text. This technology involves using algorithms and machine learning models to analyze and transcribe the acoustic features of spoken words and phrases. Speech recognition has many applications, including voice-controlled assistants, automated phone systems, and transcription services.
$Q$ : Are you familiar with language model rescoring in ASR?
R: Yes, I am familiar with language model rescoring for speech recognition. Language model rescoring is a technique used to improve the accuracy of speech recognition systems. It involves using a separate language model to evaluate the likelihood of a given hypothesise list. This separate model is typically more complex and powerful than the initial language model used for the transcription, and it is used to re-score the transcription based on the probability of the words occurring in the given context. The rescoring process involves taking the output of the initial language model, which is usually based on statistical methods such as Hidden Markov Models, and then applying a more advanced language model, such as a neural network-based language model, to generate a more accurate transcription. This is accomplished by re-ranking the possible transcriptions based on the probabilities assigned by the more advanced language model. Language model rescoring has been shown to significantly improve the accuracy of speech recognition systems, particularly in noisy or challenging environments where the initial language model may not perform well.
$Q$ : Can you give a possible example on language model rescoring with 5-best hypotheses?
$\pmb{R}$ : Sure, here is an example of language model rescoring for ASR with 5-best hypotheses:
1. I want to go to the store.
2. I want to go to the storm.
3. I want to go to the stove.
4. I want to go to the star.
5. I want to go to the storage.
After rescoring, I think the ground-truth of this speech should be: I want to go to the store.
Q: Nice job, i will give you a real example as a demonstration from [target domain]. The 5-best hypothesis is:[hypotheses list from training set], and I expect your output is: [corresponding transcription]. Following this example, can you report the true transcription from the following 5-best hypotheses:? [hypotheses list for inference]
It is noted that the responses from GPT-3.5 vary from random seeds. In our experiments, we utilize the API key from OpenAI, and fix the above template before the last turn error correction.
We observe that many recent works [103] of ICL focus on the selection strategy for in-domain demonstration, which might have an obvious impact on inference results. We leave the demonstration selection as future work, and in our few-shot learning, we manually select those utterances with long lengths according to [64].
Hypotheses-to-Transcription (H2T) Training Configuration
H2T-ft. We employ the T5-v1.1-large pre-trained model (0.75B) downloaded from HuggingFace 8. Compared with the original T5 model, GELU Sevres as activation function in the feed-forward layer to replace ReLU. Furthermore, T5 Version 1.1 was only pre-trained on C4 excluding any supervised training. Therefore, this model has to be fine-tuned before it is applied on a downstream task.
<sup>8</sup>https://huggingface.co/google/t5-v1_1-large
We finetune 20 epochs on each domain of HP dataset with a batch size of 16. To select the best model, we first split a validation set with  $5\%$  data amount of training set. The learning rate varies from  $1 \times e^{-4} \sim 1 \times e^{-3}$  according to data amount of each domain, and AdamW is employed for optimization. The  $\alpha_{1}$  to  $\alpha_{1}$  are set as 0.1, 0.05, 0.05, 0.05 respectively, as the  $2^{nd}$  utterances are usually more informative than others as shown in Fig.1. In practice, we observe the overfitting phenomenon during training. The WER on training set can be lower than  $1\%$ , however, the performance on CORAAL dataset even is even worse than the baseline. In other words, H2T-  $ft$  still has room for improvement by adding some techniques for avoiding over-fitting.
H2T-LoRA. We select LlaMa-13B as the frozen pre-trained model in our method, which is downloaded from HuggingFace  $^{9}$ . The learning rate is set as  $1e^{-4}$ , and the batch size is 128. For the low-rank adapter, we implement by peft  $^{10}$ , where the configuration of rank  $r$  is set as 8. Similarly, we also use T5-v1.1-large as pre-trained model with low-rank adapter for experiments, where the learning rate is set as  $3e^{-4}$  and the lora_  $r$  is set as 16.
We train 10 epochs using AdamW optimizer, and the prompt for LLM is designed as follows:
"Below is a best-hypotheses that is transcribed from an automatic speech recognition system. Write a response to predict the true transcription using the tokens from other-hypotheses.##### best-hypothesis:{1st utterance}##### other-hypothesis:{2nd  $\sim$  5th utterances} ######### Response:"
The prompt template is not unique, and it leaves a slight impact on the final WER result. Additionally, we calculate the WER using Sclite toolkit, which keep consistent with evaluation script of ESPNet $^{12}$ .
$LM_{rank}$  is an Transformer-based language model that is implemented using ESPNet toolkit  $^{13}$ , where the training transcription from each HP domain is utilized for a typical LM training. The Transformer layer of each model varies from 8 to 16 in terms of data amount. The training epoch is set as 20, and Adam is employed as optimizer. The initial learning rate is set as 0.002 with warm up strategy. During decoding, the perplexity of each hypothesis is calculated for re-ranking the N-best list, and the utterance with the lowest perplexity is selected as the final output.
LibriSpeech Results and Failure Cases Study
We list two representative failure cases from LibriSpeech-test-other in Table. 7. For the first case, "ward" is corrected by "warde" as there is an "his" behind it. Additionally, we observe that "warde" also appears in the  $2^{nd}$  hypothesis, so LLM adopts it according to context information. For the second case, LLM directly adopts the  $2^{nd}$  utterance in the N-best list, as "think" does not often appear at the beginning of a sentence from a grammatical perspective. Therefore, as [100] and explained in future work of 5.5, we argue that LLM correction should also consider acoustic information provided by the ASR system, which helps to avoid "over-correction" cases and keeps the fidelity to spoken language.
Table 6: WER (%) results on LibriSpeech dataset. "  ${o}_{nb}$  " and "  ${o}_{cp}$  " respectively denote n-best oracle and compositional oracle that are defined in 5.2.
Test Set	Baseline	Correction with	Oracle
LMrank	H2T-ft	H2T-LoRA	onb	ocp
LS-clean	1.8	1.8	1.8-0.0%	1.7-5.6%	1.0	0.6
LS-other	3.7	3.7	3.9+5.4%	3.8+2.7%	2.7	1.6
Results on Mandarin and code-switching Dataset
We include AISHELL-1 [6] as a Mandarin dataset into HP benchmark, consisting of a training set with 120098 utterances and a testing set with 7176 utterances. We randomly select 20k examples (16.7%) from training set to evaluate the effect of proposed H2T-LoRA. For the foundation model, we
$^{9}$ https://huggingface.co/decapoda-research/llama-13b-hf
10 https://github.com/huggingface/peft
<sup>11</sup>https://github.com/usnistgov/SCTK/blob/master/doc/sclite.htm
$^{12}$ https://github.com/espnet/espnet/blob/master/egs2/TEMPLATE/asr1/asr.sh
<sup>13</sup>https://github.com/espnet/espnet/tree/master/egs2/librispeech/asr1
Table 7: Failure cases corrected by H2T-LoRA. The utterances are drawn from LibriSpeech-test-other.
Type	Utterance	WER
1stHypo. by AM	Yet there was gambling again the second night between ward and several others of his profession.	0
Correction by H2T	Yet there was gambling again the second night between warde and several others of his profession.	6.25
Ground-truth Transcription	Yet there was gambling again the second night between ward and several others of his profession.	-
1stHypo. by AM	Think he really needs it he pursued	0
Correction by H2T	He really needs it he pursued	14.3
Ground-truth Transcription	Think he really needs it he pursued	-