Listen Again and Choose the Right Answer: A New Paradigm for Automatic Speech Recognition with Large Language Models (2024)

Abstract: Recent advances in large language models (LLMs) have promoted generative error correction (GER) for automatic speech recognition (ASR), which aims to predict the ground-truth transcription from the decoded N-best hypotheses. Thanks to the strong language generation ability of LLMs and rich information in the N-best list, GER shows great effectiveness in enhancing ASR results. However, it still suffers from two limitations: 1) LLMs are unaware of the source speech during GER, which may lead to results that are grammatically correct but violate the source speech content, 2) N-best hypotheses usually only vary in a few tokens, making it redundant to send all of them for GER, which could confuse LLM about which tokens to focus on and thus lead to increased miscorrection. In this paper, we propose ClozeGER, a new paradigm for ASR generative error correction. First, we introduce a multimodal LLM (i.e., SpeechGPT) to receive source speech as extra input to improve the fidelity of correction output. Then, we reformat GER as a cloze test with logits calibration to remove the input information redundancy and simplify GER with clear instructions. Experiments show that ClozeGER achieves a new breakthrough over vanilla GER on 9 popular ASR datasets.

heading: 1 Introduction
text: Recent advances in large language models (LLMs) have attracted a surge of research interest thanks to their remarkable language generation and reasoning ability (OpenAI, 2022, 2023; Touvron et al., 2023a,b), which achieve a wide range of success on natural language processing (NLP) tasks (Brown et al., 2020; Wei et al., 2022; Ouyang et al., 2022). Powered by LLMs, latest work (Chen et al., 2023a) proposes a generative error correction (Yang et al., 2023) (GER) benchmark1 for automatic speech
*Corresponding author. 1https://github.com/Hypotheses-Paradise/Hypo2T
rans
recognition (ASR), and they release a HyPoradise dataset2 that contains over 332K pairs of decoded N-best hypotheses and ground-truth transcription in various ASR domains. It has shown great effectiveness in learning the mapping from hypotheses to transcription by parameter-efficient LLM finetuning (Hu et al., 2021), which significantly enhances the ASR result and outperforms typical LM rescoring methods (Mikolov et al., 2010).
However, GER paradigm is also observed to suffer from two limitations. First, LLMs are unaware of the source speech during GER process, which could lead to results that do not match the source speech content. For example, as shown in Fig. 1 (left), the source speech reads the word “Think” at
2https://huggingface.co/datasets/PeacefulData /HP-v0
666
the beginning and followed by “he”, which is correctly recognized by the 1-best hypothesis. Then during the GER process, LLM removes the word “Think”, as this structure of verb plus noun at the beginning of a sentence is not rigorous according to grammar. However, this is not expected as it violates the source speech content. Second, we observe that N-best hypotheses usually only vary in a few tokens. For example, as shown in Fig. 1 (right), all the tokens in candidates are the same except “enjoys”/“enjoy”/“joins”. In this case, it would be information redundant to leverage all of the hypotheses for predicting the ground-truth transcription, which could confuse the LLMs about which tokens to focus on for correction and thus lead to sub-optimal GER performance.
Motivated by the above observations, we propose ClozeGER, a new paradigm for ASR generative error correction. First, we introduce a popular multimodal LLM, SpeechGPT (Zhang et al., 2023a), to receive source speech as an extra input to the GER paradigm. With the powerful cross-modal ability of SpeechGPT, we can now constrain GER to comply with the source speech while correcting the errors in decoded hypotheses. Then, in order to remove the input information redundancy, we reformat it as a cloze test (i.e., a special multiplechoice question) with logits calibration (Kumar, 2022; Wang et al., 2023), where the identical parts across N-best hypotheses are set as the context and the varying parts are set as blanks (each with several options provided). With such clear instructions for error correction, it would be easier for LLMs to perform context reasoning and choose the right answer for each blank rather than predicting the entire sentence from redundant N-best inputs3. Finally, we add a simple post-processing stage to correct the errors in cloze context (i.e., identical parts across N-best list) to further improve the correction result.
Our contributions are summarized as follows:
• We propose ClozeGER, a new paradigm based on multimodal LLM for ASR generative error correction, which receives both source speech and the decoded N-best hypotheses as input to predict the ground-truth transcription.
• We further reformat the generative error correction as a cloze test with logits calibration to remove the information redundancy in N-
3Think if we humans are asked to do GER, which option is easier and efficient, cloze or entire sentence prediction?
best hypotheses input and simplify the GER paradigm with clear instructions.
• Experiment evidence shows that our proposed ClozeGER achieves a new breakthrough over vanilla GER on 9 popular ASR datasets.

heading: 2 Related Work
text: Large Language Models. There is recently a surge of research interests in Transformer-based LLMs, such as ChatGPT (OpenAI, 2022), GPT-4 (OpenAI, 2023) and LLaMA (Touvron et al., 2023a,b). Benefiting from the huge model size and abundant training data, LLMs can well understand the linguistic structures and semantic meanings behind textual data, which shows remarkable performance on a wide range of NLP tasks (Brown et al., 2020; Wei et al., 2022; Ouyang et al., 2022). More recently, researchers have started to explore the potential of LLMs on multimodal tasks by incorporating other modalities into LLMs (Liu et al., 2023; Li et al., 2023; Chen et al., 2023b; Zhang et al., 2023b,c; Gao et al., 2023; Fathullah et al., 2023). Among them, SpeechGPT (Zhang et al., 2023a) is one of the most popular multimodal LLMs that represent speech and text using a unified tokenizer, which enables us to add source speech into the original N-best hypotheses input of the GER paradigm. LM Rescoring and ASR Generative Error Correction. LM rescoring has been widely used in ASR decoding to rerank the N-best hypotheses and yield better 1-best recognition result (Arisoy et al., 2015; Shin et al., 2019; Mikolov et al., 2010). Furthermore, to make full use of all candidatures, recent works employ the entire N-best list for error correction (Leng et al., 2021; Ma et al., 2023). Powered by LLMs, latest work proposes a generative error correction (GER) benchmark (Chen et al., 2023a) to predict the ground-truth transcription from ASR N-best hypotheses and achieves remarkable performance. This work serves as an extension of GER to resolve the existing limitations. Cloze Test with LLMs. As a special format of multiple-choice questions (MCQ), the cloze test provides a context with some blanks, where each blank is provided with several options for selection. Recently, cloze test and MCQ are widely adopted in LLM-centric scenarios (Chiang et al., 2023; Zheng et al., 2023b), as well as numerous LM benchmarks including MMLU (Hendrycks et al., 2020), AGIEval (Zhong et al., 2023), and CEval (Huang et al., 2023). However, recent works
Response
Response
Response1: Make Choices
Response2: Correct Context
observe that LLMs-based cloze test is vulnerable to option position changes due to their inherent “selection bias” (Kumar, 2022; Wang et al., 2023; Pezeshkpour and Hruschka, 2023). In this work, we reformat the GER paradigm as a cloze test for simplification, as well as introduce a logits calibration method to remove the existing selection bias and make LLM a robust cloze solver.

heading: 3 Methodology
text: In this section, we present our proposed ClozeGER paradigm in detail. We first introduce the preliminary knowledge of GER in §3.1, and then we investigate to introduce source speech to GER paradigm with multimodal LLM (§3.2). Finally, we present the new task format of ClozeGER in §3.3.

heading: 3.1 Preliminary: Generative Error Correction
text: We follow the original generative error correction benchmark (Chen et al., 2023a) as shown in Fig. 2 (a). Given an input speech X , the pre-trained ASR model first transcribe it into N -best hypotheses YN = {Y1, Y2, · · · , YN} by beam search decoding. The goal of GER is to learn a hypotheses-totranscription (H2T) mapping MH2T that predicts the transcription Y based on N -best list YN :
Y = MH2T(YN ), (1)
Given the ground-truth transcription Y ∗, we can finetune the LLM to learn MH2T in an autoregressive manner, where the cross-entropy loss
LH2T is formulated as:
LH2T = T∑
t=1
− logPθ(y∗t |y∗t−1, · · · , y∗1,YN ),
(2) where y∗t is the t-th token of Y
∗, and θ denotes the learnable parameters in LLM (i.e., LoRA).

heading: 3.2 GER with Source Speech
text: In order to prevent GER from violating the content of source speech, we incorporate it as an extra input into LLM to improve output fidelity as shown in Fig. 2 (b). So that Eq.(1) should be rewritten as:
Y = MH2T(YN , X), (3) where the N-best hypotheses and source speech are concatenated using the following instructions:
"Below is a speech and its candidate transcriptions from a speech recognition system. Please listen to the speech and revise the candidate transcriptions to generate better final recognition results. ### Speech:{speech units}. ### Candidates:{Nbest hypotheses}. ### Response: "
To jointly process text and speech, we leverage the popular multimodal LLM, SpeechGPT4 (Zhang et al., 2023a), to replace the LLaMA in original GER benchmark. Notably, SpeechGPT is developed by discretizing speech into 1,000 HuBERT units and adding them to LLaMA-7b5 tokenizer, and it then finetunes LLaMA-7b to learn crossmodality mapping. With such multimodal ability, we can enable GER to comply with source speech.
4https://huggingface.co/fnlp/SpeechGPT-7B-cm 5https://huggingface.co/yahma/llama-7b-hf

heading: 3.3 ClozeGER

heading: 3.3.1 Cloze Format
text: Since N-best hypotheses usually only vary in a few tokens, it would be information redundant to send all of them for GER, which could confuse the LLM about which tokens to focus on and thus lead to increased miscorrection. To this end, we simplify the GER paradigm as a cloze test as shown in Fig. 2 (c). Specifically, we set the identical parts across N-best hypotheses as the context and the varying parts as blanks, where each blank is provided with several options. In addition, we also insert a null token ‘<NULL>’ to align the N-best candidates. We design an instruction-following cloze template:
"Below is a speech and its candidate transcriptions from a speech recognition system. The candidates are formatted as a cloze test, where the blanks to fill are indicated by ‘[Blank1]’, ‘[Blank2]’, etc. Each blank is provided with several options indicated by ID letters ‘A’, ‘B’, ‘C’, etc., where
‘<NULL>’ denotes the null token. To generate a better final recognition result, please listen to the speech and write an answer choice for each blank. ### Speech:{speech units}. ### Cloze test: [Blank1] he [Blank2] need it. ### Options: [Blank1]: A. Think; B. <NULL>. [Blank2]: A. rarely; B. really; C. rally. ### Answer choices: "
With such clear instructions for error correction, it would be easier for LLM to perform context reasoning and choose the right answer for each blank than to predict the entire sentence from redundant N-best inputs. In addition, the strong speech understanding ability of SpeechGPT enables ClozeGER to refer to source speech to make better choices.

heading: 3.3.2 Logits Calibration with Prior Estimate
text: Despite the promising performance, most recent works find that LLMs-based cloze test is vulnerable to option position changes due to their inherent “selection bias” (Kumar, 2022; Wang et al., 2023; Pezeshkpour and Hruschka, 2023; He et al., 2023). Similarly, in our experiments, we have observed a strong “selection bias” towards option ‘A’, especially in the cases where ClozeGER makes mistakes. One reason is that most ground-truth options are ‘A’ in the training data6 as the 1-best hypothesis usually enjoys the best quality. As a result, during inference when LLM find it hard to decide the answer choice, it tends to select ‘A’ that can at least guarantee no performance drop, i.e., op-
6Ablation study on debiased training is in Table 4.
tion ‘A’ comes from 1-best hypothesis (baseline). Inspired by prior works on permutation-based debiasing (Wang et al., 2023; Zheng et al., 2023b,a), we propose a logits calibration approach with prior estimation to alleviate this bias during inference.
Formally, we denote the question context as c, the n option IDs (e.g., A/B/C) for one blank as d, and the default option contents (i.e., follow the order of N-best hypotheses) as x. Take the second blank in Fig. 2 (c) as an example, we have c = “[Blank1] he [Blank2] need it”, d = [A, B, C], n = 3, x = [rarely, really, rally]. The concatenation of default option IDs and contents is denoted as o.
Then, we use I to denote a permutation of {1, 2, · · · , n}, and I to denote the set of all possible Is. For better formulation, we denote oI as the concatenation of option IDs and I-permuted option contents, and rI(i) denotes the position of ID i in I . Take the above example to illustrate, assume I = [2, 3, 1], then oI = {A: really, B : rally, C : rarely} and rI(1) = 3, rI(2) = 1, rI(3) = 2. In order to alleviate the selection bias of LLMs towards the option IDs, we have to first formulate it mathematically. One feasible solution (Wang et al., 2023; Zheng et al., 2023b) is to enumerate all permutations of the option contents and average their output distributions for debiasing:
Preal(xi|c, o) = 1 |I| ∑
I∈I Pllm(drI(i)|c, oI), (4)
where i ∈ {1, 2, · · · , n}, and Preal(xi|c, o) denotes the debiased (i.e., real) probability of i-th option content in x. After such enumeration of all possible permutations of option contents, the “selection bias” towards option IDs could be well resolved.
Furthermore, considering calculating full permutations is prohibitively expensive (×n! costs), we leverage the cyclic permutation as an alternative, i.e., I = {(i, i + 1, · · · , n, 1, 2, · · · , i − 1)}ni=1. Take the previous example, we have I = {(1, 2, 3), (2, 3, 1), (3, 1, 2)}. It reduces the computational cost from ×n! to ×n and guarantees one pairing between each option ID and content. However, the inference cost of ×n is still much too high especially in practical scenarios.
Inspired by recent work on MCQ debiasing (Zheng et al., 2023a), it is a promising idea to disentangle the distribution bias of option IDs from the original predictions from LLMs. The insight behind is that the option ID itself is inherently unrelated to the option contents, the option orders,
and the context. Therefore, the LLM predicted distribution over di can be disentangled as a prior distribution of the option ID di and the debiased (i.e., real) distribution of option content of di:
Pllm(di|c, o) ∝ Pprior(di|c)Preal(xi|c, o), (5) where we omit I as only the default order of options needs to be considered during formal inference. The prior distribution Pprior(di|c) indicates the LLM’s selection bias towards option ID di, and the debiased distribution Preal(xi|c, o) indicates the LLM’s real confidence of option content xi.
Inspired by recent work (Zheng et al., 2023a), we calculate the averaged prior distribution P̂prior(di) on validation set Dv to estimate Pprior(di|c). In particular, we perform cyclic permutation Ic for each sample in Dv and send all of them for inference, and then we average their output distributions to obtain the prior distribution of option ID di:
P̂prior(di) = 1 |Dv| ∑
{c,o}∈Dv Pprior(di|c),
Pprior(di|c) = sm ( 1
|I| ∑
I∈I logPllm(di|c, oI)
) ,
(6) where “sm” denotes softmax operation. With the estimated prior distribution, we can perform logits calibration during the inference stage:
P̂real(xi|c, o) ∝ Pllm(di|c, o)/P̂prior(di), (7) In case of the small size of Dv, this logits calibration method would be efficient during inference.

heading: 3.3.3 Post-processing
text: After cloze test with logits calibration, many ASR errors captured by the blanks (i.e., varying tokens between N-best hypotheses) are corrected, but what about those remaining in the question context? For example, as shown in Fig. 2 (c), the ASR model fails to recognize the word “needs”, where all Nbest hypotheses produce “need”. In this case, we need a simple post-processing stage to further correct them, with the following instructions:
"Below is a speech and its candidate transcription from a speech recognition system. Please listen to the speech and correct the candidate transcription. ### Speech:{speech units}. ### Candidate:{cloze result}. ### Response: "
Similar to GER, here we also use SpeechGPT with LoRA finetuning for post-processing. This stage is necessary especially when ASR model does not perform well on current speech domains.

heading: 4 Experiments

heading: 4.1 Setup
text: Dataset. We utilize the HyPoradise (HP) dataset from the original GER benchmark (Chen et al., 2023a) for our experiments, which contains over 332K hypotheses-transcription pairs collected from multiple mainstream ASR corpora. Specifically, each transcription is paired with 5-best hypotheses transcribed from Whisper-Large model (Radford et al., 2023) with beam search decoding. In this work, we select 9 popular ASR corpora from HyPoradise to evaluate the proposed ClozeGER, including WSJ (Paul and Baker, 1992), CommonVoice (Ardila et al., 2019), TED-LIUM3 (Hernandez et al., 2018), SwitchBoard (Godfrey et al., 1992), LibriSpeech (Panayotov et al., 2015), CHiME-4 (Vincent et al., 2016), LRS2 (Chung et al., 2017), ATIS (Hemphill et al., 1990), and CORAAL (Kendall and Farrington, 2021). Since HyPoradise provides 5-best hypotheses for each sample, we follow it to set 5 options for each cloze blank. More statistical details are in Appendix A.
Models. As introduced before, we use SpeechGPT as the LLM in our main experiments, and later on we also try LLaMA-2-7b7 (Touvron et al., 2023b) to verify the effectiveness of ClozeGER paradigm in case of no source speech input. For efficient LLM finetuning, we employ the popular low-rank adapter (LoRA) tuning strategy (Hu et al., 2021), where the rank r is set to 8 and the LoRA is added in the query, key, value, and output layers in each Transformer block (Vaswani et al., 2017). As a result, the number of trainable parameters is only 8.39 M, accounting for only 0.12% of the LLM.
Training and Inference. During finetuning, we employ Adam optimizer (Kingma and Ba, 2014) with a learning rate set to 2e−4 and warmup steps set to 100. The number of training epochs is set to 5, the batch size is set to 256. The maximum input sequence length is set to 1024. For inference, we adopt top-k and top-p sampling strategies at the same time, where k = 40 and p = 0.75. The temperature is set to 0.1, and beam size is set to 4.

heading: 4.2 Main Results
text: Table 1 presents the WER results of ClozeGER with SpeechGPT and LoRA tuning. First, we can observe that vanilla GER achieves significant im-
7https://huggingface.co/meta-llama/Llama-2-7 b-hf
provements over Whisper ASR baseline, and introducing source speech as extra input further enhances the performance. In comparison, our proposed ClozeGER also improves the baseline but underperforms the GER approach. There are two reasons, the cloze test suffers from selection bias and cannot yield satisfactory results, and on the other hand, there are many errors exist in the cloze context due to imperfect N-best list quality (i.e., Whisper is a general ASR model and may not perform well in every specific domain). To this end, we first propose a logits calibration approach to alleviate the selection bias, which results in considerable WER reductions. Furthermore, we add a post-processing stage to correct the errors in cloze context, which moves one step forward and outperforms the GER approach with source speech input, where some results even surpass the N-best oracle.
Table 2 reports the WER results of ClozeGER using LLaMA-2 as a backbone in case of no source speech as input, where we observe similar gains of performance of the proposed ClozeGER over GER baseline. It demonstrates the general effectiveness
of ClozeGER paradigm, as well as the proposed logits calibration and post-processing techniques.

heading: 4.3 Ablation Study and Analysis
text: Why we need logits calibration? We note that ClozeGER only produces limited improvement and even underperforms the GER baseline, where one key reason is the “selection bias” towards option IDs. Take the WSJ dataset as an example, we visualize the distribution of predicted option IDs in Fig. 3, where over 80% of predictions fall on option ‘A’. This phenomenon can be explained by the imbalanced training label distribution8 and its resulted “selection bias” as shown in Table 3. As a result, the proposed ClozeGER yields poor predicting accuracy on options ‘B’ to ‘E’, which limits its final WER performance.
How does logits calibration work? To alleviate this limitation, we propose to estimate a prior distribution to represent the selection bias
8Because top-1 hypothesis enjoys the best quality and is most likely to be the ground-truth choice.
and then remove it from the output logits during inference stage to conduct calibration. As illustrated by the orange bars and yellow lines in Fig. 3, our proposed calibration approach mitigates the imbalance of predicted options and effectively improves their cloze accuracy. As a result, the overall cloze accuracy is significantly improved on various ASR datasets in Table 4, which thus produces better WER performance in the final. More visualization on other datasets is in the Appendix Fig. 5.
Why not do calibration during training stage? One may raise concerns about why we conduct the calibration during inference instead of training stage, where simply shuffling the options seems able to mitigate the selection bias. To this end, we present an ablation study in Table 5 to explore calibration in different stages, where we observe that shuffled training can indeed achieve some improvement over ClozeGER but still lag behind the proposed logits calibration approach. When further
adding the post-processing stage, our calibration also produces better performance, indicating that it is more beneficial to remove selection bias during the inference stage rather than training stage.
This observation may suggest that within our specific framework, the model’s acquisition of selection bias during training is somehow advantageous, as the strategy of arbitrarily selecting ‘A’ would at worst regress to the baseline (top-1 hypothesis) without deterioration. As a result, the task difficulty of ClozeGER is naturally reduced, because it can rely on the heuristic of selecting ‘A’ when feeling uncertain and hard to make the choice. Thereafter, the logits calibration during inference removes such bias by diversifying some of the ‘A’ choices to other options to improve the accuracy.
Why we need post-processing? Table 6 further investigates the role of the postprocessing stage in ClozeGER paradigm. We observe that such post-processing is necessary to further correct the errors in cloze context, which results in promising gains of performance. On the other hand, this phenomenon also reflects the suboptimal quality of N-best hypotheses, according to
the errors in cloze context (see Fig. 2 (c)), as Whisper is a general ASR model that may not generalize well to some specific domains like accents.
The role of ClozeGER and post-processing. One may raise concerns on whether the effectiveness of our approach is all attributed to postprocessing. To this end, we add it onto the GER baseline, which also shows some improvement but still underperforms our ClozeGER, indicating that our proposed ClozeGER paradigm and logits calibration raises the upper-bound performance of GER by correcting errors in a targeted manner. Based on that, the post-processing aims to further correct the errors in cloze context that cannot be resolved by the cloze-test paradigm.
Case study. We illustrate a case study in Fig. 2 to interpret the motivation of our approach. First, we introduce source speech as extra input to improve output fidelity, i.e., avoid removing the word “Think”. Second, we reformat GER as a cloze test to reduce the
task difficulty with clear instructions, i.e., explicitly prompt the LLM to select a word from [“rarely”, “really”, “rally”], which results in an effective correction. Finally, we note that there still exist some errors in the cloze context, e.g., “need”, which cannot be corrected by the cloze-test paradigm. To this end, we design a post-processing stage to further remove them and improve the final output.

heading: 5 Conclusion
text: In this paper, we propose ClozeGER, a new paradigm for ASR generative error correction. First, we introduce a multimodal LLM (i.e., SpeechGPT) to receive source speech as extra input to improve the fidelity of correction output. Then, we reformat GER as a cloze test with logits calibration to remove the input information redundancy and simplify GER with clear instructions. Experimental evidence shows that ClozeGER achieves a new breakthrough over vanilla GER on 9 popular ASR datasets. Further analysis verifies the effectiveness of different modules in our framework.
Limitations
This work introduces an extra input of source speech to improve the output fidelity, which achieves some improvements but is somewhat limited since we only employ a new prompt for LLMs to exploit the source speech. In future, we may investigate more advanced multimodal prompting techniques for it and also combine them with our proposed cloze-test paradigm for further improvements. In addition, we believe it should also be beneficial to further investigate the reasons for the sub-optimal performance of cloze-test paradigm, as well as integrate the calibration and post-processing stages as an end-to-end pipeline in future work.
Ethics Statement
This work does not pose any ethical issues. All the data used in this paper are publicly available and under the following licenses: the Creative Commons BY-NC-ND 3.0 License, Creative Commons BY-NC-ND 4.0 License, Creative Commons BYNC-SA 4.0 License, Creative Commons Attribution 4.0 International License, Creative Commons (CC0) License, the LDC User Agreement for NonMembers, the TED Terms of Use, the YouTube’s Terms of Service, and the BBC’s Terms of Use.

heading: Acknowledgements
text: This research is supported by the National Research Foundation Singapore under its AI Singapore Programme grant number AISG2-TC-2022-004. The computational work for this article was partially the High Performance Computing Centre of Nanyang Technological University, Singapore.

heading: A HyPoradise Dataset Details
text: A.1 Hypotheses Generation We employ the HyPoradise (HP) dataset9 from original GER benchmark (Chen et al., 2023a), which contains over 332K pairs of N-best hypotheses and ground-truth transcription. The hypotheses are generated using Whisper-Large (Radford et al., 2023)
9https://huggingface.co/datasets/PeacefulData /HP-v0
beam search decoding, where the beam size is set to 50. After removing repetitive utterances, the top-5 hypotheses with the highest probabilities are selected as the final N-best list. The HyPoradise dataset is built by carrying out this decoding strategy on multiple popular ASR datasets as introduced in §A.2. As a result, the detailed statistics of HyPoradise dataset is illustrated in Table 7.
A.2 ASR Corpora Selection For ASR corpora selection, we follow original benchmark (Chen et al., 2023a) to cover common ASR scenarios, e.g., noise and accents. Consequently, the following corpora with evident domain characteristics are collected to build the HP dataset. WSJ (Paul and Baker, 1992): The Wall Street Journal (WSJ) is a widely-used benchmark for speech recognition. It includes read speech from speakers in a controlled environment, with a focus on business news and financial data. The train-si284 split (37,514 samples) is utilized to generate HP training set. The dev93 (503 samples) and eval92 (333 samples) splits are combined to build test set. CommonVoice (Ardila et al., 2019): CommonVoice 5.1 is a publicly available dataset for automatic speech recognition. It contains speech recordings from diverse speakers in over 60 languages. To generate HP dataset, they randomly select 51,758 samples from its train-en split with various accent labels, including African, Australian, Indian, and Singaporean. Then, it is separated into two parts to build training (with 49,758 samples) and test (with 2,000 samples) sets respectively. TED-LIUM3 (Hernandez et al., 2018): TEDLIUM3 is a speech dataset recorded from TED talks. It contains a diverse range of background noise, speaker accents, and speech topics. Considering its large size, they randomly select 50,000 samples from its train split for HP dataset generation, which is then separated into training (47,500 samples) and test (2,500 samples) sets. SwitchBoard (Godfrey et al., 1992): The SwitchBoard corpus is a telephone speech dataset collected from conversations. It focuses on North American English and involves over 2,400 conversations from around 200 speakers. They randomly select 36,539 samples from its train split to generate HP training set, as well as 2,000 samples from the eval2000 split to generate HP test set. LibriSpeech (Panayotov et al., 2015): LibriSpeech is a public corpus of read speech from audiobooks,
including 1,000 hours of labeled speech data from diverse speakers, genders, and accents. To generate HP training data, they exclude some simple utterances from its train-960 split that yield 0% WER, which results in 88,200 training samples. The testclean split (2,620 samples) is used for HP test data.
CHiME-4 (Vincent et al., 2016): CHiME-4 is a dataset for far-field noisy speech recognition. It includes real and simulated noisy recordings in four noisy environments, i.e., bus, cafe, pedestrian area, and street junction. Its tr05-real-noisy (9,600 samples) and test-real (1,320 samples) splits are used to generate HP training and test data, respectively.
LRS2 (Chung et al., 2017): Lip Reading Sentences 2 (LRS2) is a large-scale publicly available audiovisual dataset, consisting of 224 hours of video clips from BBC programs. They randomly select 42,940 samples from its train split as training set, and the rest of 2,259 samples are used for test set.
ATIS (Hemphill et al., 1990): Airline Travel Information System (ATIS) is a dataset comprising spoken queries for air travel information, including flight times, prices, and availability. It contains 4,773 utterances recorded from over 500 speakers, which are separated into two parts to build training (3,964 samples) and test (809 samples) sets.
CORAAL (Kendall and Farrington, 2021): The Corpus of Regional African American Language (CORAAL) is the first public corpus of AAL speech data. It contains audio recordings along with the time-aligned orthographic transcriptions from over 150 sociolinguistic interviews. To generate HyPoradise dataset, they select 1,728 samples as training set and 100 samples as test set.
A.3 Validation Set Selection
As mentioned in §3.3.2, our logits calibration method requires a validation set to calculate the
prior distribution P̂prior(di). To this end, we reserve a small portion of training samples to build the validation set. To save the computation cost and time, we randomly select 100 samples from each ASR corpus in Table 7 for prior estimation (Wu et al., 2021). Relevant ablation study is illustrated in Fig. 4, where we observe that around 100 validation samples are sufficient to estimate a reliable prior distribution for logits calibration on most datasets.

heading: B Examples of Cloze test
text: Table 8 presents several examples of cloze test built from CHiME-4 test-real set, where each example contains the context and several options.

heading: Future Work
text: In future, we may investigate more advanced multimodal prompting techniques for it and also combine them with our proposed cloze-test paradigm for further improvements. In addition, we believe it should also be beneficial to further investigate the reasons for the sub-optimal performance of cloze-test paradigm, as well as integrate the calibration and post-processing stages as an end-to-end pipeline in future work