SlowGuess/NeurIPS_2023_OCR huggingface dataset:
employ Chinese LlaMa2-7b from Huggingface $^{14}$ , and keep other settings consistent with H2T-LoRA in this paper. Notably, ASR on Mandarin dataset is usually evaluated by character error rate (CER), as character is equal to word in Chinese. We also evaluate H2T-LoRA on ASRU [78] that is a Mandarin-English code-switching dataset. Each utterance in ASRU contains 8.6 Chinese characters and 1.6 English words on average, and the transcripts cover many common fields, including entertainment, travel, daily life, and social interaction. Similarly to AISHELL-1, Chinese LlaMa2-7b is selected as foundation model, and 15k Hypotheses-Transcription pairs are randomly used in H2T training. Additionally, the evaluation metric for code-switching ASR is mixed error rate.
From the results reported in Table 8, we observe that H2T-LoRA demonstrates its generalization on both Mandarin and code-switching dataset, which respectively achieves  $20.6\%$  CER and  $24.5\%$  MER reductions. Furthermore, H2T-LoRA exhibits remarkable data efficiency since the training pairs is equivalent to less than 20 hours of speech data.
Table 8: Mandarin and code-switching results on AISHELL-1 (CER) and ASRU (MER) dataset, where 20k and 15k Hypotheses-Transcription pairs are respectively used in H2T training.
Dataset	Language	Baseline	H2T-LoRA	\( o_{nb} \)	\( o_{cp} \)
AISHELL-1	Man	6.3	5.0-20.6%	4.1	3.1
ASRU	Man & Eng	11.0	8.3-24.5%	8.6	6.5
Results on Spoken Language Intent Detection (SID) task
We first train an intent detection model using the transcription of ATIS training set, as the intent label is available for each example. Then, during testing, we respectively feed the  $1^{st} \sim 5^{th}$  utterances in Whisper hypotheses list, utterance after correction, and ground-truth transcription as input text for intent detection. The accuracy results are reported in Table 9.
Table 9: Accuracy (%) results of intent detection with different input on ATIS test set.
Textual input	nthutterance in Hypotheses list, n = 1 2 3 4 5	After Correction	Oracle
Acc. (%)	94.9	95.5	94.2	94.3	94.2	97.4+2.5%	97.9
$^{14}$ https://huggingface.co/ziqingyang/chinese-llama-2-7b
We observe that: (1) When we use corrected text for intent detection, the accuracy is  $97.4\%$  which achieves an absolute improvement of  $2.5\%$  over  $1^{st}$  utterance in the hypothesis list. (2)  $2^{nd}$  utterance is more suitable for intent detection than  $1^{st}$  utterance in terms of accuracy. This phenomenon validates the case (ii) from a perspective other than WER, where the discarded utterances in the N-best hypotheses might be better than the selected utterance.
Preliminary Results on Zero-shot Prompt-based Error Correction
To examine the zero-shot ability of LLM, we propose a framework that requires a targeted LLM to perform either (i) ranking-based error correction or (ii) single-sentence generative correction. We follow the self-activated prompting method mentioned in the previous appendix section to repurpose the language model in the form of zero-shot error correction, without providing instructions. Using the same decoding test set from WavLM and Whisper, a 5-gram language model (coefficient of 0.1) combined with its acoustic model score showed a  $2.95\%$  WER relative improvement. This result is slightly worse than the  $\mathrm{LM}_{\text{rank}}$  baseline. The current limitations on the results regarding the zero-shot abilities of LMs could be attributable to the model scale. The zero-shot or emergent abilities of these models have been reported [100] to be more significant when the parameter scale of the LLM exceeds 100B.
Hypotheses Paradise (HP) Dataset Visualizations
We have open-sourced a Colab example<sup>15</sup> for HP dataset visualizations and analysis. First, same as Fig. 1, we visualize and analyze the information in N-best hypotheses from both utterance- and token-levels. Fig. 4 illustrate more visualizations on CHiME-4 test sets, where we can observe valuable information in N-best hypotheses.
Figure 4: Probabilities of the case (i) and (ii) on CHiME-4 test set, similar to the right part of Fig. 1.
Furthermore, we also visualize and compare the word frequency in N-best hypotheses and ground-truth transcription in Fig. 5, where we can observe some but limited gap between them.
N-best Hypotheses Distribution
Fig. 6 visualizes the distribution of N-best hypotheses generated by different-sized Whisper models, i.e., from 'tiny' to 'large'. We can observe very limited diversity in the N-best hypotheses generated by Whisper models. Considering such high monotonicity, we only collect the top-5 hypotheses to form our HP dataset.
Though the proposed HP benchmark provides a new paradigm of generative error correction for ASR, we analyze and discuss the limitations of this work from the following perspective:
<sup>15</sup>https://colab.research.google.com/drive/1traA2scdnmAKFq6yIEZhHwrhCBVxB2ig
Figure 5: Top-10 word frequencies in N-best hypotheses and ground-truth transcription of CHiME-4 test set.
Figure 6: N-best hypotheses distribution with different Whisper models. Each color denotes an unique hypothesis, 'GT' denotes the ground-truth transcription. The sample is selected from LibriSpeech test-clean set, i.e., '1089/134691/1089-134691-0005.flac'.
- Evaluation metric. As an ASR error correction benchmark, HP employs WER as the primary metric to evaluate the system performance. Nevertheless, prior work [2] has pointed out that WER can be too coarse-grained for describing the performance of ASR models. Furthermore, [82] raise community awareness regarding the problems caused by the optimistic bias toward ASR accuracy. In the future, we aim to provide more annotations for spoken language, e.g., entity spans and dependency structure. Accordingly, a comprehensive evaluation framework can be established to assess the quality and interpretability of output from the LLM-enhanced ASR system.
- Robustness in reality. HP benchmark covers mainstream domains where ASR tasks are usually deployed. However, as shown in [57], no single validation or test set from public datasets is sufficient to measure transfer to real-world audio data. Since all test sets of HP benchmark are drawn from existing ASR corpus, despite enhancing the WER performance, we are unable to ascertain the extent to which it can mitigate the gap between well-trained ASR models and real-world application scenarios. Furthermore, considering the discrepancy between spoken language and written language, more efforts are required from both speech and NLP communities to build a human-like robust ASR system beyond single modality [18].
With recent advances in using large-scale neural language models to solve problems once believed to be challenging for machines to learn and understand, we believe it is timely to move to the next
milestone: providing publicly accessible n-best hypotheses as transcription resources from LLM decoding. This motivation inspires this work, offering a collection of hypotheses paradise, inspired by in-context learning.
- Who may benefit from this research: Researchers working on speech technology and language model based error correction; as well as the users using the related techniques for responsible and reproducible machine learning technology.
- Who may be put at disadvantage from this research: When our work revealed that open-source hypotheses can be used to generate malicious recognition, we understood the responsibility of properly explaining the results to the public and providing reproducible evaluations. We have discussed terms of use for reusing these hypotheses with legal and regulatory experts, addressing potential risks and concerns.
- Whether the task/method leverages biases in the data: To alleviate possible bias in the data and model, we have made efforts to design reproducible metrics and to evaluate a wide variety of reproducible data sources and training configurations. We have also conducted user studies to highlight potential bias in "terms of use" provided in our Github repo.
- Who will be supporting/hosting/maintaining the dataset? Hypotheses Paradise has been actively maintained by the authors of this paper. We are still actively updating the dataset that focus on specific ASR scenario, which are noise-robust ASR and multi-lingual ASR. In INTERSPEECH 2023, we will have a tutorial to introduce the related Hypotheses Paradise-V2 with some excited experimental results. Furthermore, we also open the link to collect more hypothesis-transcription pairs from public.
- How can the owner/curator/manager of the dataset be contacted? To contact the main developers, we encourage users to use our emails: {chen1436, yuchen005}@e.ntu.edu.sg, huckiyang@gatech.edu.
- Is there an erratum? Users can use GitHub to report issues/bugs, and we would actively improve the codes accordingly. We also have a HuggingFace Model card under an non-profit organization in https://huggingface.co/datasets/PeacefulData/HP-v0.
- Will the dataset be updated? Yes, we are actively updating Hypotheses Paradise codes and data sources. Users could get information and the newly updated version through our GitHub repository.
- If the dataset relates to people, are there applicable limits on the retention of the data associated with the instances? No for the Dataset.
- Will older versions of the dataset continue to be supported/hosted/maintained? Yes, we will keep the old version that generated by Whisper. All versions can be found on our GitHub repository
- If others want to extend/augment/build on/contribute to the dataset, is there a mechanism for them to do so? We maintain Hypotheses Paradise on GitHub and we encourage all users to share their ideas to extend Hypotheses Paradise to more speech recognition cases. Users can use GitHub to report issues/bugs, and send us emails to discuss solutions.
This research is supported by the National Research Foundation, Singapore under its AI Singapore Programme (AISG Award No: AISG2-100E-2022-102). The computational work for this article was partially performed on resources of the National Supercomputing Centre, Singapore (https://www.nssc.sg). The corresponding author is supported by Wallace H. Coulter Fellowship.