Whispering LLaMA: A Cross-Modal Generative Error Correction Framework for Speech Recognition
Authors: Srijith Radhakrishnan
Chao-Han Huck Yang
Sumeer Ahmad Khan
Rohit Kumar
Narsis A. Kiani
David Gomez-Cabrero
Jesper Tegnér
Venue: EMNLP 2023 Main (2023)
Keywords: Multimodal Learning
Parameter-Efficient Adaptation
Speech Recognition
Generative Error Correction
TL;DR: We propose a new cross-modal fusion method integrating large-scale pre-trained speech (Whisper) and language (LLaMA) models for Generative Error Correction.

Abstract: We introduce a new cross-modal fusion technique designed for generative error correction in automatic speech recognition (ASR). Our methodology leverages both acoustic information and external linguistic representations to generate accurate speech transcription contexts. This marks a step towards a fresh paradigm in generative error correction within the realm of n-best hypotheses. Unlike the existing ranking-based rescoring methods, our approach adeptly uses distinct initialization techniques and parameter-efficient algorithms to boost ASR performance derived from pre-trained speech and text models. Through evaluation across diverse ASR datasets, we assess our fusion technique, demonstrating a 37.66\% improvement in word error rate (WER) relative performance compared to the n-best Oracle. To encourage future research, we have made our code and pre-trained models open source at [https://github.com/Srijith-rkr/Whispering-LLaMA](https://github.com/Srijith-rkr/Whispering-LLaMA)

Whispering LLaMA: A Cross-Modal Generative Error Correction Framework for Speech Recognition
Srijith Radhakrishnan 1.2,6 Chao-Han Huck Yangl.4, Sumeer Ahmad Khan'.6, Rohit Kumar', Narsis A. Kiani,, David Cobrer-a Jesper N. Tegner
King Abdullah University of Science and Technology MManiaal Institute of Technology
GGeorgia Institute of Technology *NVIDIA Research
KKarolinska Institute
SSDAIA-KASST Center of Excellence in Data Science and Artificial Intelligence
srijithrkr@gmail. com;
huckiyang@gatech. edu
Abstract
2011) of language structures beyond the knowledge of transcribed audio present in the ASR model's pre- pre-training data, thereby improving performance on unseen words. Furthermore, adapting the two- pass paradigm to accommodate domain shifts (Li et al., 2023; Liu et al., 2021; Yu et al., 2023) is much easier as only the language model needs to be finetuned on the the need for spoken transcription corpus, which can be particularly beneficial for unser-resoureed or endangered spoken languages.
We introduce new cross-modal fusion tech- nique designed for generative error correc- tion in automatic speech recognition (ASR). Our methodology leverages both acoustic in- formation and external linguistic representa- tions to generate accurate speech transcrip- tion contexts. This marks a step towards fresh paradigm in generative error correction within the realm of n-best hypotheses. Unlike the existing ranking- runking-based rescoring methods, our approach adeptly uses distinct initializa- tion techniques and parameter-efficient algo- rithms to boost ASR performance derived from pre- pre-trained speech and text models. Through evaluation across diverse ASR datasets, we assess our fusion technique, demonstrating a 37.66% improvement in word етгоr rate (WER) relative performance compared to the n-best Oracle. To encourage future research, we have made our code and pre-trained mod- els open source at https: thub. com/ Sri Srijith-rkr/Whisper ing-erin..
dataset. This alleviates
The recent emergence of conversational abilities in large language models, such as ChatGPT (Ope- nAI, 2023a) and GPT-4 (OpenAI, 2023b), has further sparked interest in leveraging the repre- sentational power of large pre-trained models for more complex tasks involving diverse data modali- ties (Yang et al., 2021b; Chang et al., 2023). More- over, this new research direction also introduces a set of unique challenges related to considering information from other input modalities, such as acoustic and visual conditions (Peng et al., 2023; Zhang et al., 2023), in which could enrich using context beyond text-only input.
1 Introduction
End-to-end (E2E) trained speech models have demonstrated statc-of-the-art performance on Au- tomatic speech recognition (ASR) tasks. Several methods (Xia et al., 2017; Guo et al., 2019; Hu al., 2021b; Yang al., 202 Salazar et al., 2020) have widely adopted a two-pass rescoring paradigm leverage upon language models to fur- ther enhance the capabilities of these models. the two-pass paradigm, the first pass ASR system "gen- erates" n-best hypotheses using an E2E acoustic model, while the second pass "re-ranks" these hy- potheses by incorporating a language model (L.M).
Recognizing speech signals is task that neces- sitates both acoustic information (Hu al., 202 la; Hung al., 2023) (e.g., speaking environments) and linguistic information (Meng et al., 2023; Chen et al., 2023b,c) (e.g., context and domains). Ef- ficiently amalgamating or integrating representa- tion learning from acoustic modeling into language modelling to bolster its performance represents a notably intricate research domain that warrants fur- ther exploration. In this paper, we present token- level fusion framework, merging two foundation (large-scale pre-trained) models into recognition eтror correction paradigm, with the objective of enhancing the performance of ASR systems.
This two-pass reranking approach has several notable advantages over single-pass End-to-End (E2E) ASR systems (Amodei et al., 2016; Chan et al., 2016). Firstly, the subsequent large language model often captures a more comprehensive un- derstanding (Stooke et al., 2023; Tur and De Mori,

---

GLLAMA
Kataper
A'w (Adapiorynase)
Selisttentir LLatss
SelvAttniin Wrsper
Kuaa
Language Embeddings (M.)
Podding
SA
S
M down
Sicu
M
(H2)
Layer Output
M,
Leamaale Matrix
Frozen
Trainable
Figure I: Illustration of proposed generative ASR error correction with trainable token (Ma) and fusion mechanism inside a self-attention layer described in Section 3.2. detailed model-wise illustration discussed in Fig 2.
2 Related Work on ASR Post-processing
language transformer model, to generate error- corrected transcripts by utilizing the n-best hy- potheses via prompt (illustrated in Appendix, Fig 5) and audio representations via our proposed frame- work as input.
Transformer-based language models (Shin al., 2019; Salazar al., 2020) approach the two-pass paradigm by utilizing the summation of negative log-likelihoode of individual tokens from the lan- guage model to re-score the n-best output. Recent works on deliberation method (Hu et al., 2020; Prabhavalkar et al., 2018) 8) and audio-attention based rescoring (Futami et al., 2021; Gandhe and Rastrow, 2020; Tanaka et al., 2021) in improv- ing ASR-LM rescoring with the incorporation of acoustic features. Recent works on decoder prompt- ing (Yang et al., 2023a) and encoder-decoder based error correction (Chen et al., 2023a; Ma et al., 2023) have demonstrated benefits in using an external lan- guage model for reducing the transcription error rate. Meanwhile, how to inject or fuse represen- tations from large acoustic model into another language model remains under investigation.
Whisper utilizes the encoder of Transformer model to derive features from audio input, which then fed into the decoder through multi-headed cross-atiention, enabling auto-regressive prediction (Wang et al., 2023; Irie et al., 2022). The encoded features provide information from audio input via cross-attention, while the decoder's 's self- attention attends previous tokens using a key-value caching mechanism.
token
We fuse the audio features and the Whisper lin- ear layers that generate the key and value pairs in the decoder's cross- cross-attention mechanism to the LLaMA model to inject audio information. The inherent self-attention modules in LLaMA com- bined with the added cross- cooss-attetionn module make it analogous to the Whisper decoder. An overview of the proposed method is presented in Appendix, Fig. 2.
3 Method
We diseuss the model architecture and the intuition behind the proposed feature combination Sec- tion 3.1. The cross-modal fusion mechanism and weight initialization are explained in Section 3.2 and Section 3.3, respectively.
3.2
Cross-Modal Fusion Mechanism
We introduce our mechanism in Fig 1. To efficient fine-tune large models, incorporate two resid- ual adapter (Houlsby et al., 2019; Radhakrishnan et al., 2023; Chen al., 2023d; Yang et al., 2023b) modules (A), and Aty) after the self-attention mod- ules (SAf)) of the frozen LLaMA model at each layer. The first variable Ai, represents the adapter in layer used to fine-tune the LLaMA model using scaled dot product attention mechanism. The sec- ond variable Aiy refers another adapter in layer i used to fuse Whisper features with the LLaMA model by following an autoencoder mechanism. In each A we incorporate a learnable matrix
3.1 Generative Error Correction for ASR
Our approach combines two pre-trained models, Whisper (Radford et al., 2022) and LLaMA (Tou- vron et al., 2023), to facilitate generative error cor- rection (Yang et al., 2023a; Chen et al., 2023a). Firstly, we employ Whisper, multi-task encoder- decoder-based transformer (Vaswani al., 2017) speech model trained on 680.000 hours of multi- lingual data, to encode audio representations and generate transcripts of n-best hypotheses. Sec- ondly, we utilize LLaMA, decoder-based large

---

Mj E RNoxNN, Ng denotes the dimensionality of the adapter embeddings, while Nr indicates the dimensionality of LLaMA embeddings. The language embedding feature extracted from the pre- trained LLM is represented by Hÿ, for each layer.
Table I: Dataset sample statistics are provided with alias names. The Science & Technology category of GigaSpeech (Chen et al., 2021) is divided into two sub- sets: GSss (small) and GSsss (medium). to evaluate performance differences with respect to data size.
We repurpose the frozen LLaMA linear layers K ilama and Litama from the LLaMA self-attention S to transform M3 into key and value pairs, thus reducing the number of trainable parameters. We also reuse the query tensor from the frozen LLaMA attention module SAJ, to compute A as shown below; S represents the Softmax:
Dataset
Train
Test
ATIS (Hemphill et al., 1990)
4978
893
GigaSpeech: Entertainment (GSg)
4709
1000
People & Blogs (GSp)
6802
1000
Science & Technology (GSss)
6908
1000
Science & Technology (GSsM)
10323
1000
adjust the shape of K whisper and V whisper to en-
S Giaaa (Itt) (ltm
Ki iioma (M3)" )
able computation of Multi Head Attention (MHA) with Q Uama (H2) from the frozen LLaMA model as before to obtain its adaptable self-attention head (SA as:
Vilama (M))
vdk
(1) To integrate the audio representations and key- value tensors from the Whisper decoder cross- attention module into the LLaMA model, we in- troduce two additional linear frozen transforma- tions (K* whisper and V whissper) at each layer of the LLaMA model. These transformations are initial- ized with the respective weights from the cross- attention module of the Whisper decoder. By ap- plying the audio representations to these additional linear transformations, we generate the valuv pairs that mirror the ones produced by Whisper. We then utilize the second adapter module A to add trainable components to learn cross-modal rep- resentation. We apply learnable projection matrix to down project the obtained key and value pairs. Where Nw denotes the size of the Whisper encoded audio representations (x). We then apply the SiLU activation function (Elfwing et al., 2018) followed by a learnable up-projection Nw. xNw, to compute trainable output:
Vhoma (Ht) (K Wohispee
T
V Vanteisper (5)
S
ddg
Then, we utilize a gated fusion mechanism, WWispering-CLahh (WL), to fuse all the mod- ules together as shown below:
SAic SAL d A1 + Aw SAW, (6)
where dy and Aw are learnable scalars.
3.3 Weight Initialization
Maoun R Nw Nw
The latent dimensions of the Whisper and LLaMA models are different, making it necessary to re- shape the Whisper tensors to match the shape of the LLaMA model while preserving the latent structure and information inherent in the Whis- per model. Tensors are shaped in the format of [B. NH, T, HS), which denotes the Batch size, Number of heads, context length and Head Size, respectively. The last two dimensions undergo transformation during the attention mechanism. Hence in order to preserve the Whisper latent structure, We initialize a matrix of zeros of shape C I Hitame XTwhisper Suame and fill the principal diagonal of the last two dimensions with ones. We then place K¹ and V on the top left corner of the padding template. We further initialize the projec- tion matrices Mi Maown: Mt on the second adapter module Aly as identity matrices. The proposed framework encounters significant losses and fails to converge unless this initialization strategy is fol- lowed to preserve Whisper's latent representations.
M E
Aw (x) < SiLU (x Je-Maannch Mamono) M
(2)
Using this setup, we transform the key-value pair at each layer to merge the hidden representation (H₂) from the output of the Whisper frozen pre- trained encoder with decoder from LLaMA:
R whisper f Aly (Ku
whissper (HA));
(3)
Vi whisper
Al (Vi Whhiispprr (HA)).
(4)
Once we obtain the corresponding Whisper key and value pairs, we apply the padding mechanism described in 3.3 to preserve the latent structure of the Whisper Key and Value embeddings and

---

Table 2: The experimental results are presented terms WER without text normalization. The performance of our proposed framework is reported in rows 2 4. Oracle refers to the candidate among the n-best hypothesis with the lowest word error rate compared to the ground truth. Rows 5 9 represent different ablation experiments on the best- performing model, WLM. The WERR is measured relative to the oracle performance as shown B.2
#
Method
#Para.
ATIS
GSg
GSp
GSss
GSSM
WER Aug
WERR (1)
1
Oracle (Ist-pass)
13.76
28.22
22.84
23.93
19.5
21.64
2
WL2
26.40M
2.04
21.76
19.21
20.55
11.6
15.03
30.52
3
WLM
7.97M
1.77
21.61
16.20
18.02
9.82
13.48
37.66
4
WLs
4.89M
1.89
22.24
17.23
19.157
10.185
14.144
34.62
5
WLM w/o masking
4.89M
3.94
27.56
18.10
21.71
12.79
20.04
22.25
6
WLM w/o HA
4.89M
253.20
123.19
203.44
376.81
256.44
242.61
-1020.68
7
WLM w/o init.
4.89M
405.83
500.58
414.34
461.63
390.64
434.60
-1907,45
8
WCM w/o SAw
1.22M
1.66
24.99
18.734
20.73
10.86
15.39
28.83
9
Big-scale Adapter
4.91M
1.45
23.65
16.59
19.93
10.62
14.45
33.21
4
Experimental Setup
uation. We select domain-specine subsets in Gi- gaSpeech and focus three specific categories: Entertainment, People and Blogs, and Science and Technology. To explore performance variations with respect to the number of data points, we fur- ther divide the Science and Technology category into two subsets. Table I provides detailed infor- mation on the number of training points per dataset. We chose Whisper-Ting to generate the n-best hy- pothesis baseline to establish robust evaluation en- vironment that aligns more closely with real-world settings dealing with sub-optimal hypotheses. By employing Whisper-Tiny we mimic a weak acous- tic model with lower-quality hypotheses. Feeding LMs with better-quality hypotheses from Whisper- Large would make the generative error correction task less challenging for LM adaptation and does not explore the model's 's performance under practi- cal settings where our method is intended to be em- ployed. However, emphasize that method remains effective when starting with a Whisper- Large hypothesis in Appendix E.
4.1
Models
For our experiments, we utilize the LLaMA-7B model architecture. As we instruct the language model with the generated hypotheses (as explained in Section 1.3.1) to perform generative error cor- rection, we initialize our model weights with Al- paca (Taori et al., 2023), model fine-tuned from LLaMA-7B, utilizing 52,000 instruction-folloiingg demonstrations to enable instruction following abil- ities. To extract audio representations from input audio clips, we employ Whisper-Large V2, a model with 1.55B parameters trained on 620,000 hours of audio data. Additionally, we employ Whisper- Tiny, a model with 70M parameters, for generating our transcripts, as described in the subsequent sec- tion 4.2. We name our model Whispering LLaMA (WL) and train three variants with our proposed framework with No 10 and 8, 16, 32 named W.,. (large), WLss (medium), WCs (small), re- spectively. We design WCr with two separate Aw adapter modules for key and value, respectively. WLM and WLs use the same Aw adapter in sec- tion 3.2 to reduce trainable parameters.
For each audio clip, we generate 200 hypotheses using a top-k value of 200 and a randomly selected temperature between the range of sequently, we filter out redundant sentences and select the top 15 with the highest log probability.
Sub-
4.2
Dataset
We curate our own transcripts by leveraging two datasets: the Airline Travel Information System (Hemphill et al., 1990) (ATIS) and Gi- gaSpeech (Chen et al., 2021). ATIS consists of audio recordings of individuals querying flight in- formation. GigaSpeech, contains audio from audio- books, podcasts and You'ube videos on diverse topics. ATIS represents a semantically correct, domain-specific dataset, while GigaSpecch repre- sents a more noisy, real-world setting in our eval-
4.3 Training Pipeline
The input to our model consists of the encoded audio representations extracted from the Whisper- Large model, accompanied by the 15-best tran- scripts generated by Whisper-Tiny. We employ the prompt template used by the Alpaca model shown in Appendix Fig We utilize the Adam optimizer (Kingma and Ba, 2014) and experiment
with learning rates of x 10 2 x 10 and

---

x 10-4, selecting the optimal value. The model is trained for 25 epochs, employing early stopping to prevent overfitting. Training is conducted on two Nvidia A 110 GPUs to leverage efficient parallel processing. An effective batch size of 32 is used, and weight decay of 1 x 10-2 is applied.
explore the significance of our weight initialization mechanism by replacing it with random initializa- tion (Row 7). Both of these ablation studies vali- date our intuition, demonstrating that the method utilizes acoustic features effectively and highlight the importance of the initialization mechanism in preserving the latent structure of the acoustic em- beddings. For further insights, please refer to Ap- pendix D. We also remove the Whisper adapter (SAw) module for an text feature only baseline performance using adapters (Row 8). Since the dis- parity between the number of trainable parameters is high, we train another model with an increased adapter context dimension of N₂ 4Ng (Row 9).
4.3.1 LLM Prompting Examples for ASR
We employ the Alpaca (Taori et al., 2023) prompt template, as illustrated in Fig. 5 of the Appendix, to generate the n-best hypotheses. This template features an instructional segment designated by the Instruction tag, which offers guidance to the model. Essential contextual data required by the model is housed under the Input tag. The prompt concludes with the Response tag, directing the model enact the specified instruction within the supplied input context. Rather than adopting the re- cent advances of Task Activaaci Prompting (Yang et al., 2023a) (TAP), we opt to feed the LLM with its task-specific data (e.g.. speech recognition in our instance). Our alternative approach facilitates second-past -ooonddpssssae error correction, mitigating the latency issues observed in the extensive context windows of the TAP-based generative ASR error correction.
Conclusion
We propose a novel framework to leverage the ex- ternal knowledge from LLM to improve the tran- scription accuracy of ASR systems. Our frame- work presents a parameter-efficient way to integrate large foundational Speech and Language models to achieve competitive WERR improvements. We further conduct extensive ablation experiments to validate intuitions and open source our code and sewtrrainnd-weiighn to the research community.
4.4 Performance Studies
Limitation
Results from our experiments have been reported in Table 2. The WLm model achieves the best per- formance with a word-error-rate relative (WERR) of 37. 37.66%, defined in B.2. A comparison be- tween WL,, and WLM indicates that having sepa- rate adapter modules for key and value pairs does NOT result in performance improvements. Further dataset-specific analyses are detailed in Appendix B. The models exhibit better performance on the Gigaspeech with more in-domain data.
Using large models such as LLaMA is intuitive, as it provides a comprehensive comprehension of language structure owing to its intemet-scaled pre- training. However, deploying these systems and conducting research with them in real-world sce- narios is challenging due to their computationally intensive nature. In our approach, we aim to de- sign our framework to be parameter-efficient by re-using multiple model components with adapters for model fusion. Nonetheless, incorporating au- dio representations into the training pipeline ex- tends the training duration by 394.76%. This un- derscores the significance of alignment issues (Yen et al., 2023). Furthermore, our proposed solution demonstrates a need for larger volume of data to achieve optimal performance despite having a modest parameter count of only 7.97M to integrate foundational models. During our experimentation, we encountered issues related to over-fitting on datasets. To mitigate this problem, we trained with reduced learning rate and monitored the Word Er- ror Rate (WER) performance throughout the train- ing process and selected the model checkpoint with the best performance to implement early stopping.
4.5 Ablation Studies
We empirically discover that masking the prompt except for the ground truth in the cross entropy loss function significantly improves the perfor- mance. We attribute this improvement to the model's enhanced capacity to grasp accurate se- mantics, achieved by refraining from penalizing the model for erroneous sentences found in the n- best hypotheses. Row 5 represents the performance of WLM without masking. We further investigate if the proposed framework is utilizing the audio representations from Whisper by substituting them with random tensors generated from a normal dis- tribution as the input (Row 6). Additionally, we

---

References
Charles T Hemphill, John Godfrey, and George R Doddington. 1990. The atis spoken language sys- tems pilot corpus. In Speech and Natural Language: Proceedings of a Workshop Held at Hidden Valley, Pennsylvania, June 24-27, 1990.
Dario Amodei, Sundaram Anantanaaaaanaan Rishita Anubbai, Jingliang Bai, Eric Battenberg. Carl Case. Jared Casper, Bryan Catanzaro, et al. 2016. Deep speech 2: End-to-end speech recognition english and mandarin. In International conference ma- chine learning, pages 173-182. PMLR.
Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzebski. Bruna Morrone, Quentin De Laroussilhe, Andrea Gesmundo, Mona Attariyan, and Sylvain Gelly, 2019. Parameter-efficien transfer learning for nip. In In- ternational Conference on Machine Learning, pages 2790-2799. PMLR.
William Chan, Navdeep Jaitly, Quoc Le, and Oriol Vinyals. 2016. Listen, attend and spell: neural network for large vocabulary conversational speech recognition. In Proc. ICASSP, pages 4960-4964
IBEE.
Hu Hu, Xuesong Yang. et al. 2021a. Redat: Accent- invariant representation for end-to-end ast by domain adversarial training with relabeling. In Proc. ICASSP, pages 6408 00--6112. IEEE.
Kai-Wei Chang, Yu-Kai Wang, Hua Shen, Tu-thing Kang, Wei-Cheng Tseng, Shang-Wop Wen Li, and Hung- Lee. 2023. Speechprompt v2: Prompt tun- ing for speech classification tasks. arXiv preprint arXiv:2303.00733.
Ke Hu, Ruoming Pang. Tara N Sainath, and Trevor Strohman. 2021b. Transformer based deliberation for two-twoss speech recognition. In Proc. SLT, pages 68-74. IEEE.
Chen Chen, Yuchen Hu, Chao-Han Huck Yang. Sabato Macro Siniscalchi, Pin-Yu Chen, and Eng Siong Chng. 2023a. Hyporadise: An open base- line for generative speech recognition with large lan- guage models. arXiv preprint 2rX9v:2300.15700
Ke Hu, Tara N Sainath, Ruoming Pang, and Rohit Prab- havalkar. 2020. Deliberation model based two-pass end-to-end speech recognition. In Proc. ICASSP, pages 7799-7803. IEEE.
Guoguo Chen, Shuzhou Chai, Guanbo Wang. Jiayu Du, Wei-Qiang Zhang. Chao Weng, Dan Su, Daniel Povey, Jan Trmal, Junbo Zhang, et al. 2021. Gi- gaspeech: An evolving, multi-domain asr corpus with 10,000 hours of transcribed audio. In Proc. Interspeech 2021.
Yun-Ning Hung, Chao- Han Huck Yang, Pin-Yr Yu Chen, and Alexander Lerch. 2023. Low-resource music genre classification with cross-modal neural model reprogramming, In Proc. ICASSP, pages 1-5. IEBE,
Zih-Ching Chen, Chin-Lun Fu, Chih-Ying Ying Liu, Shang- Wen Daniel Li, and Hung-yi yi Lee. 2023b. Exploring efficient-tuning methods in self-supervised speech models. In Proc. SLT, pages 1120-1127 IEEE.
Kazuki Irie, Róbert Csordás, and Jürgen Schmidhu- ber. 2022. The dual form of neural networks re- visited: Connecting test time predictions to training patterns via spotlights of attention. In International Conference on Machine Learning, pages 9639-9659. PMLR.
Zih-Ching Chen, Yu-Shun Sung, and Hung-yi Lee. 2023c. Chapter: Exploiting convolutional neural network adapters for self-supervised speech models. In Proc ICASSP Workshop, pages 1-5. IEEE
Diederik P Kingma and Jimmy Ba. 2014. Adam: A method for stochastic optimization. arXiv preprint arXiv: 1oXiv:1412.6980.
Zih-Ching Chen, Chao-Han Huck Yang. Bo Li, Yu Zhang, Nanxin Chen, et al. 2023d. How to es- timate model transferability of pre-trained speech models? In Proc. Interspeech 2023.
Ke Li, Jay Mahadeokar, Jinxi Guo. Yangyang Shi, Gil Keren, Ozlem Kalinli, Michael I. Seltzer, and Duc Le. 2023. Improving fast-slow encoder based transducer with streaming deliberation. In Proc. ICASSP, pages 1-5. IEEE
Stefan Elfwing, Biji Uchibe, and Kenji Doya. 2018. gegmoid-weighter linear units for neural network function approximation in reinforcement learning. Neural Networks, 107:3-11.
Linda Liu, Yile Gu, Aditya Gourav, Ankur Gandhe, Shashank Kalmane, Denis Filimonov, Ariya Ras- trow, and Ivan Bulyko. 2021. Domain-aware neural language models for speech recognition. In Proc. ICASSP, pages 7377. IEEE
Hayato Futami, Hirofumi Inaguma, Masato Mimura, Shinsuke Sakai, and Tatsuya Kawahara. 2021. Asr rescoring and confidence estimation with electra. In Proc. ASRU, pages 380-387. JEEE.
Rao Ma, Mark Gales, Kate Knill, and Mengjie Qian. 2023. N-best 15: Robust asr error correction using multiple input hypotheses and constrained decoding space. Proc. Interspeech.
Ankur Gandhe and Ariya Rastrow. 2020. Audio- attention discriminative language model for asr rescoring. In Proc. ICASSP, pages 7944-7948. IEEE,
Jinxi Guo, Tara N Sainath, and Ron J Weiss. 2019. A spelling correction model for end-to-end speech recognition. In Proc. ICASSP, pages 5651-5655. IBEE.
Zhong Meng, Weiran Wang, Rohit Prabhavalkar, et al. 2023. Jeit: Joint end-to-end model and internal lan- guage model training speech recognition. In Proc. ICASSP, pages 1-5. IEEE.

---

OpenAl. 2023a. Chatgpt. https://openai.c com/ blog/chatgpt/.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. Advances neural information processing systems, 30.
OpenAI. 2023b. Gpt-4 technical report.
Puyuan Peng. Brian Yan, Shinji Watanabe, and David Harwath. 2023. Prompting the hidden talent of web- scale speech models for zero-shot task generalization. arXiv preprint arXiv:2305. 10055.
Siyin Wang, Chao-Han Huck Yang, Ji Wu, and Chao Zhang. 2023. Can whisper perform speech-based in- context learning. arXiv preprint arXiv:2309 00000.
Rohit Prabhavalkar, Tara N Sainath, Yonghui Wu, Patrick Nguyen, Zhifeng Chen, Chung-Cheng Chiu, and Anjuli Kannan. 2018. Minimum word error rate training for attention-based sequence-to-sequenee models. In Proc. ICASSP, pages 4839-4843. IEEE.
Shinji Watanabe, Takaaki Hori, et al. 2018. Espnet: End- to-end speech processing toolkit. Proc. Interspeech 2018.
Yingee Xia, Fei Tian, Lijun Wu, Jianxin Lin, Tao Qin, Nenghai Yu, and Tin-Yan Liu. 2017. Deliberation networks: Sequence generation beyond one-pass de- coding. Advances in neural information processing xystems, 30.
Alec Radford, Jong Wook Kim, Tao Xu, Greg Brock- man, Christine Mcl.eavey and Ilya Sutskever. 2022.
Robust speech recognition via large-scale weak su- pervision. arXiv preprint 2rXiv22212.043355
Chao-Han Huck Yang, Yile Gu, Yi-Chieh Liu, Shalini Ghosh, Ivan Bulyko, and Andreas Stolcke. 2023a. Generative speech recognition error correction with large language models and lask-activating prompting. IEEE Proc. ASRU 2023.
Srijith Radhakrishnan, Chao-Han Huck Yang, et al. 2023. A parameter-efficient learning approach arabic dialect identification with pre-trained general- purpose speech model. In Proc. Interspeech 2023, pages 1958-1962.
Chao-Han Huck Yang. Bo Li, Yu Zhang. Nanxin Chen, Rohit Prabhavalkar, Tara N Sainath, and Trevor Strohman. 2023b. From english to more languages: Parameter- model reprogramming for cross- lingual speech recognition. In Proc. ICASSP, pages 1-5, IEEE
Julian Salazar, Davis Liang, Toan Q Nguyen, and Ka- trin Kirchhoff. 2020. Masked language model scor- ing. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 2699-27 2712.
Joonbo Shin, Yoonhyung Lee, and Kyomin Jung. 2019. Effective sentence scoring method using bert for speech recogmition. Asian Conference on Machine Learning, pages 1081-1093, PMLR.
Chao-Han Huck Yang. Linda Liu, Ankur Gandhe, Yile Gu, Anirudh Raju, Denis Filimonov, and Ivan Bulyko. 2021a. Multi-task language modeling for improving speech recognition of rare words. In Proc. ASRU, pages 1087-1093, IEEE.
Adam Stooke, Khe Chai Sim, Mason Chua, Tsendsuren Munkhdalai, and Trevor Strohman. 2023. Internal language model personalization of c2e automatic speech recognition using random encoder features. In Proc. SLT, pages 213-220. IEEE
Chao-Han Huck Yang, Yun- Yun Tsai, and Pin- Yu Chen. 2021b. Voice2series: Reprogramming acoustic mod- els for time series classification. In International Conference on Machine Learning, pages 11808- 11819, PMLR.
Tomohiro Tanaka, Ryo Masumura, Mana Ihori, Akihiko Takashima, Takafumi Moriya, Takanori Ashihara, Shota Orihashi, and Naoki Makishima. 2021. Cross- modal transformer- based neural correction models for automatic speech recognition. Proc. Inter- speech 2021. ISCA.
Hao Yen, Pin-Jui Ku, Chao-Ham Huck Yang, Hu Hu, Sabato Marco Siniscalchi, Pin- Yu Chen, and Yu Tsao. 2023. Neural model reprogramming with similarity based mapping for low-resource spoken command classification. Proc. Interspeech 2023.
Rohan Taori, Ishaan Gulrajani, Tianyi Zhang, Yann Dubois, Xuechen Li, Carlos Guestrin, Percy Liang. and Tatsunori B. Hashimoto. 2023. Stanford alpaca: An instruettion-oolloiin llama model. https:// github. ctmub..comttaatt latissullaassttafrrrraea
Yu Yu, Chao-Han Huck Yang, Jari Kolehmainen, et al. 2023. Low-rank adaptation of large language model rescoring for pafameter-efficient speech recognition. IEEE Proc. ASRU 2023.
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Manie-Anne I Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. 2023. Llama: Open and effi- cient foundation language models. arXiv preprint arXiv:2302. 13971.
Zhuosheng Zhang, Aston Zhang, Mu Li, Hai Zhao, George Karypis, and Alex Smola. 2023. Multi- modal chain-of ohothhtght reasoning language mod- els. arXiv preprint arXiv:2302.00922.
Gokhan Tur and Renato De Mori. 2011. Spoken lan- guage understanding: Systems for extracting seman- tie information from speech. John Wiley & Sons.

---

A Appendix
B.2 WERR
Word error rate relative is calculated as
this Appendix, We investigate the performance difference between datasets in Section B, and pro- vide illustrations of the model-level architectural design in Section C. Section D provides more in- sight into the results from ablation studies and we report Whisper Large Hypothesis baseline in Sec- tion E.
WERR(1) Oracle(i) -WER() x 100 (7) Oracel(i)
where Oracle(i) refers to the average Oracle per- formance in terms of WER and W WER(() refers to the average performance of a particular method.
C Proposed Architecture Illustrations
B Dataset Analysis
We present a model-level overview of our proposed method described in Section 3.2 in Fig 2. We add two modules into each layer of the LLaMA model. The LLaMA adapter and the Fusion adapter which refer Ap and Aw. respectively. We initialize the Fusion adapter with the weights from the Whis- per cross-attention module in the decoder model. LLaMA takes the encoded features generated by the Whisper encoder and the n-best hypothesis gen- erated by the Whisper in prompt format as input to generate the error-corrected response.
We report the WER from our experiments before and after text normalization on Table 3. We con- vert the model prediction and the ground truth to lower-case and remove punctuation during text nor- malization. The ATIS dataset is not impacted by text normalization because the dataset does con- tain any punctuation. It only contains contractions such as "I'd like" instead of "I would like" " ATIS consists of audio recordings of individuals query- ing automated airline travel inquiry systems for flight information. We believe the lack of punctua- tion and the consistent structure present within the ATIS dataset enables improved WER performance compared to GigaSpeech. The Gigaspeech dataset contains punctuation and lacks consistency within the dataset because has diverse categories and sources such as audiobooks, podeasts and YouTube videos.
D Failure Case Studies of Generative ASR with Whispering-LLaMMA
Since the WER error rate in row 6 (WLM w/o audio representations and row 7 WW w/o ini- tialization) of table is high and provides no insight into model performance, we present the training loss graphs the best performing model ( (WCM) with and without audio representations in Figure 4. The model is not able to converge below certain threshold without audio representations. Addition- ally, we include the training loss graphs of WLM with and without our initialization mechanism in Figure 3. Without our initialization mechanism, the latent structure of the Whisper encoder embedding is not preserved, leading to an inability to converge.
B.1 More Discussion on Ground Truth Match Rate
During dataset generation, we remove the ground truth if it is present among the Whisper generated best hypotheses. This allows us introduce new metric called Ground Truth Match Rate (GTMR). GTMR calculates the percentage of predictions generated by the model that exactly match the ground truth. This metric indicates the model's ability to learn the structure of the dataset. The GTMR of our experiments before and after text normalization is reported in Table 5. The model is able to learn the structure of the dataset better with more data points, as observed from the perfor- mance difference between GSss and GS5M. It can also be observed that the model able to learn the simpler structure of ATIS much better than other GigaSpecch datasets.
E Whisper Large Decoding Baseline
We report the results of using the hypothesis generated by Whisper Large to train our best- performing model (WLM) on GigaSpeech En- tertainment (GSg) and Science and Technology (GSss) datasets on Table 4. By leveraging the LLaMA model with the proposed generative error correction mechanism, we are able to match the performance of the Whisper Large model with 1.5 billion parameters by using a Whisper-Ting model with just 70 million parameters. Using the hypothe- ses generated by Whisper Large results in a higher

---

Generative Speech Transcription
LLaMA Decoder
Audio Representations
LLaMA
MIPP
Whisper Decoder
Whisper Encoder
Fusion Adapter (Aw)
Whisper MLP
LLaMA Adapter (Ai)
Whilsper MLP
Whisppr
Cross-Attention
LLaMA Self-Attention
Whisser
Self-Attention
Whisper
Self-Attention
LLaMA
MLP
Whilsper MLP
Fusion Adapter (Aw)
Whisper
Whilsper
LLaMA Adapter (Au)
Self-Attention
Whisper
Cross-Attention
LLaMA Self-Attention
Whisper Self-Attention
N-Best Hypotheses Hi ripe your
Prompt Inputs
Hell wero
Input Audio Clip
Figure 2: Whispering-LLaMA model-overview of proposed adaptation pipeline described in Section 3.2
Yoin
Table 3: The experimental results in terms of Word Er- Rate (WER), before and after text normalization. We convert all text to lowercase and remove the follow- ing punctuation [".", "-", "?", "'"J. Rows 1-3 represent before text normalization, and Rows 4-6 represent after text normalization.
Allie
2000
Method
ATIS
G8g
68p
G8gs
GS
WER (J)
Figure 3: Train loss of WLr (Row WLa without initialization (Row 7) on the Entertainment dataset
WC2
200
21.76
19.21
20.55
11.6
15,03
WCn
137
21.61
16.20
18.02
9.82
13.48
WLs
2.11
23.60
17.13
20.16
10.73
14.75
WL1
13.36
10.35
WCM
1.77
17.71
10.83
12.00
6.07
9.67
WCs
1.89
15.21
11.49
13.03
6.41
9.61
F Reproducibility Resources
We have open-sourced the pre-trained model weights and code, available at https://ggi thub. http:srijithhrrrrr///sssrrrrnggr
1250
Our fu- ture plan includes integrating this baseline into both Espnet (Watanabe et al., 2018) and HyPo- radise (Yang et al., 2023a; Chen al., 2023a) accommodate broader range of use cases.
Figure 4: Train loss of WL (Row 3) vs WLM without audio representations (Row 6) on the Entertainment dataset
Acknowledgement
WERR as expected. This finding confirms the ef- fectiveness of the proposed approach, particularly in the context of Whisper Large generated N-best hypotheses.
The authors thank Maxim Lvov and anonymous reviewers for their feedback on the draft.

---

Model Input
#*# Instruction:
You ASR transcript selector. You have few transcripts generated by automatic speech recognition model. Your task to generate the most likely transcript from them. the generated transcripts have grammatical or logical errors, you will modify them accordingly to produce the most accurate and coherent transcript.
### Input:
so that it can carry its momentum to the logic that he can carry his moment tour with the logic
that he that he can carry his momentum to with an logic that he can carry his momentum true within logic that he carries momentum through with logic that it can carry moment and through with the logic that can carry moment of truth with the logic that it can carry a moment on tour with the logic that it can carry moment on true with the logic that it can carry momentum tool within logic that it carry
carry his momentum through with the logic
moment tour with the logic
that can carry at moments and through
the logic
that it can carry his moment on tour with a logic that it can carry his moment on tour with the logic
### Response:
Model Output
that It can carry its momentum through with logic.
Figure Illustration of Alpaca prompt template used our proposed framework
Table 4: Results of employing a Whisper Large generated hypothesise baseline in comparison to the proposed
Whisper Tiny hypothesis baseline
Method
GS
GS
WFRAvg() (+)
WERR(1)
Oracle with Whisper Tiny
28.22
23.93
26.08
WLs with Whisper Tiny
21.61
18.02
19.82
24.01
Oracle with Whisper Large
21.78
18.09
19.93
WLM with Whisper Large
15.59
12.77
14.18
28.86
Table 5: The experimental results in terms of Ground Truth Match Rate (GTMR), Before and after text normalization. Rows 1-3 represent before normalization, and Rows 5-7 represent after text normalization. Row 4 and denote the average GTMR across datasets, and column W ER denotes the average GTMR across each method
#
Method
ATIS
GS g
GSp
GSss
GS5M
WER ())
1
WLL
86.1
26.5
24.1
24.8
36.2
39.5
2
WLM
88.3
26.3
31.4
28.8
41.0
43.17
3
WLs
87.5
25.2
27.6
26.6
38.6.1
41.10
4
Avg
87.3
26.0
27.7
26.7
38.6
5
WC
86.1
46.9
46
45.5
56.4
56.2
6
WLM
88.3
47.0
54.3
49.2
62.7
60.3
7
WLs
87.5
44.3
49.5
46.3
60.0
57.5
8
Avg
87.3
46.0
49.9
47.0
59.7


Reviews:
[{"review_id": "6PQn5kJYZ4", "reviewer": "Reviewer_v94L", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "The authors appreciate the reviewer's effort in reviewing the submission and are deeply grateful for recommending the paper to be discussed at the conference. We have provided a related discussion to address their concern regarding the reproducibility of our proposed framework. \n\n**_In the interest of reproducibility, we will open source the model code, training script, inference script and pre-trained weights under the MIT license._** \n\n***\n#### **Regarding initialisation mechanism**\nThe initialisation mechanism in our proposed framework ensures that the latent structure and information inherent in the audio features are preserved during injection to the LLaMA model and **are implemented as part of the training script** yet to be open-sourced (currently included in supplementary materials). The results in Table 1 (rows 2-4) and the ablation studies in Row 7 (WL-M w/o init.) consistently showcase the stability and effectiveness of the proposed initialization mechanism.\n\nWe further conducted 3 runs with our best-performing model $WL-M$ on the GigaSpeech Entertainment ($GS_{E}$), People & Blogs ($GS_{P}$) and Science & Technology ($GS_{S}$) datasets to investigate the consistency in results and have reported the WER below. \n\n|     | Run 1  | Run 2  | Run 3  | Mean   | Std   |\n| --- | ------ | ------ | ------ | ------ | ----- |\n| $GS_{E}$ | 21.496 | 21.529 | 21.518 | 21.514 | **0.017** |\n| $GS_{M}$ | 18.01  | 17.969 | 17.958 | 17.979 | **0.027**|\n| $GS_{S}$ | 16.498 | 16.282 | 16.255 | 16.345 | **0.133** |\n***\n#### **Regarding Early stopping**\n\nEarly stopping is an established countermeasure against overfitting with Adapters, as demonstrated in prior work [1], and is easily reproducible. We also **commit to integrating the early stopping feature** into training scripts when we open-source our work. \n\nWe sincerely hope that these measures address any concerns surrounding the reproducibility of our work.\n\n***\n**References**\n\n\n[1] : Thuy-Trang Vu, Shahram Khadivi, Dinh Phung, and Gholamreza Haffari. 2022. Domain Generalisation of NMT: Fusing Adapters with Leave-One-Domain-Out Training. In Findings of the Association for Computational Linguistics: ACL 2022, pages 582–588, Dublin, Ireland. Association for Computational Linguistics.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The described approach is new and interesting and should be discussed at the conference.", "reasons_to_reject": "It appears that it will be very difficult to reproduce the results - the authors point out themselves that a lot of fiddling has to be done with initialization, monitoring WER on the test set and stopping the training at the right moment. ", "questions_to_authors": "", "paper_topic_and_main_contributions": "The paper describes an attempt to combine the Whisper ASR encoder with the LLama LLM. In contrast to conventional methods like n-best reranking, the two models are fused by means of \"cross-modal fusion\", i.e. the whisper's audio features are integrated into LLama's frozen self-attention module using trainable adapter modules followed by a weighted fusion. Experimental results indicate significant reductions in WER compared to an oracle n-best selection for ATIS and GigaSpeech test sets.", "excitement": 3, "reproducibility": 2, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "DydQhudlMg", "reviewer": "Reviewer_YmwX", "summary": "", "questions": "1. What is the purpose of using Alpaca in place of a pure LLaMA?\n\n2. Why the whisper-tiny results are corrected instead of whisper-large results?", "limitations": "", "rating": 3, "confidence": 4, "soundness": 2, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "The authors sincerely thank the reviewer for their dedicated effort in reviewing the submission. Their attention to detail contributed significantly to enhancing the manuscript's quality and provided performance-driven insight that benefited from the review cycle.\n\nWe have provided a related discussion to address their questions below. \n\n***\n#### **Addressing and clarifying discrepancies in the manuscript.**\na) In response to your observation on line 272 regarding the reported WERR value of 32.83%, we acknowledge this discrepancy. The accurate value should be 37.66%, reflecting the performance improvement of the $WL-M$ model from Table 2.\n\nb) Your observation on line 273 pertaining to the absence of a $WL$ model reference in our tables is correct. The intended reference should have been $WL-M$ rather than $WL$, reflecting the model under discussion. \n\nWe deeply appreciate your attention to detail. We apologise for this error and assure you that we will rectify these mistakes in our final version.\n\n***\n#### **Q1: Purpose of using Alpaca in place of a pure LLaMA?**\n\nThe LLaMA-**1** [1] model open-sourced by Meta has **no instructional fine-tuning** [2] and, hence, is not aligned towards following instructions. In our proposed framework, we **instruct** the language model with the generated hypothesis  (as illustrated in Figure 5) to perform generative error correction. Hence, we use Alpaca [3],  A version of the LLaMA model that has been fine-tuned on instructional datasets, as it can adhere to provided instructions better than LLaMA.\n\n***\n#### **Q2: Why the whisper-tiny results are corrected instead of whisper-large results?**\n\nWe chose the Whisper Tiny generated hypothesis due to the following reasons:\n\n**Better Evaluation Criteria:** The proposed method tries to leverage the rich pretrained knowledge in Large Language Models as text normalisation to improve the transcription accuracy of audio models. In this paradigm, providing the Language model with better-quality hypothesis makes the generative error correction task less challenging for the language model, and does not explore the model's performance under practical settings where our method is intended to be employed. As real-world use cases may encompass scenarios where the hypotheses from acoustic models might exhibit lower quality.\n\n**Research Reproducibility:** We adopted the Whisper Tiny model to ensure research reproducibility. Generating multiple hypotheses using the Whisper Large model proves computationally intensive for most research environments. To illustrate, Generating hypothesis for the $GS_{S}$ dataset (11,323 data points) takes 1 hour 48 minutes by the Whisper Tiny model in a single NVIDIA A100-80GB GPU, while it takes around 37 hours with Whisper Large.\n\n\n\nAs per the reviewer’s interest, we also agree that having a model trained on the Whisper Large hypothesis would be a great assessment to the community and would enhance the comprehensibility of the paper. We have attached the results of using the hypothesis generated by Whisper Large for training our best-performing model (WL-M) on GigaSpeech Entertainment ($GS_{E}$) and Science & Technology ($GS_{S}$) datasets to investigate the performance. We could not run the experiments across all the datasets due to limited time. However, **_We commit to adding a related discussion_** to the camera-ready submission that depicts the performance when using the Whisper Large model to generate hypotheses across all datasets.\n\n\n| |$GS_{E}$|$GS_{S}$|$WER_{Avg}\\downarrow$|$WERR\\uparrow$|\n|:----|:----|:----|:----|:----|\n|Oracle with Whisper Tiny (Proposed method) |28.22|23.93|26.08|- |\n|WL-M with Whisper Tiny (Proposed method) |21.61|18.02|19.82|**24.01**|\n|Oracle with Whisper Large|21.78|18.09|19.93|-|\n|WL-M with Whisper Large|15.59|12.77|14.18|**28.86**|\n\nOur results showcase the effectiveness of our proposed method in benefiting both the Whisper Large and Whisper Tiny hypothesis-trained models. Using the hypothesis generated by Whisper Large results in a higher WERR as expected. We sincerely hope that these findings will convincingly showcase the advantages of our approach to the reviewer.\n\n***\n**References**\n\n[1]: Hugo Touvron and Thibaut Lavril and Gautier Izacard and Xavier Martinet and Marie-Anne Lachaux and Timothée Lacroix and Baptiste Rozière and Naman Goyal and Eric Hambro and Faisal Azhar and Aurelien Rodriguez and Armand Joulin and Edouard Grave and Guillaume Lample. 2023. LLaMA: Open and Efficient Foundation Language Models\n\n[2]: Chung, H.W., Hou, L., Longpre, S., Zoph, B., Tay, Y., Fedus, W., Li, E., Wang, X., Dehghani, M., Brahma, S. and Webson, A., 2022. Scaling instruction-finetuned language models. arXiv preprint arXiv:2210.11416.\n\n[3]: ''tatsu-lab/stanford_alpaca\" repository in GitHub", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The paper addresses the issue of ASR-LLM integration which is a problem of practical important that raised to the significance in the recent years.\n\nThe method seams to be promising. However, it should be evaluated in a more demanding setting.", "reasons_to_reject": "1. The paper uses whisper-large to improve the results of whisper-tiny. I expected that the authors would show the improvement to WER and WERR of whisper-large with the use of their method. Evaluating on whisper-tiny leads to overly optimistic results. Whisper-tiny is just a \"toy\" model to be used in tutorials and examples. In any real application one of the larger Whisper models would be used. \n\n2. The performance analysis in Section 4.4 is not compatible with the results reported in  tables. In particular:\n\na) line 272: 32.83 WERR is not reported in Table 2.\nb) line 273: There is no WL model mentioned in any of the tables.", "questions_to_authors": "1. What is the purpose of using Alpaca in place of a pure LLaMA?\n\n2. Why the whisper-tiny results are corrected instead of whisper-large results?", "paper_topic_and_main_contributions": "The paper approaches the problem of correcting speech recognition errors with the use of of LLMs.\nThe authors propose to utilize cross-attention tensors from Whisper (a SOTA ASR model) as a part of the LLM decoder (in this case LLaMA) and then use the resulting adapted LLM for speech error correction.", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Typos_Grammar_Style_and_Presentation_Improvements": "line 225: \"with of our\"\n\n"}, "extra_text": {}}, {"review_id": "0Faz1TxGfg", "reviewer": "Reviewer_7p1W", "summary": "", "questions": "1. Why is adapter A_L^i introduced? It does not fuse any acoustic information.\n\n2. Did you try the LoRA adapter? It's a more widely used adapter for fine-tuning LLMs.\n\n", "limitations": "", "rating": 4, "confidence": 3, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "The authors wish to express their heartfelt appreciation for awarding a soundness score of 4 and acknowledging the originality of our work. We have provided a related discussion to address the reviewer's questions below.\n***\n#### **Regarding using Whisper Tiny for n-best hypothesis**\nWe chose the Whisper Tiny generated oracle baseline due to the following reasons:\n\n**Uniform training and testing evaluation:** We use the Whisper Tiny model to generate the N-best oracle baseline as we trained our models with the hypothesis generated by the Whisper Tiny model. \n\n**Better Evaluation Criteria:** The proposed method tries to leverage the rich pretrained knowledge in Large Language Models as text normalisation to improve the transcription accuracy of audio models. In this paradigm, providing the Language model with better-quality hypothesis from Whisper Large makes the generative error correction task less challenging for the language model, and does not explore the model's performance under practical settings where our method is intended to be employed. As real-world use cases may encompass scenarios where the hypotheses from acoustic models might exhibit lower quality.\n\n**Research Reproducibility:** We adopted the Whisper Tiny model to ensure research reproducibility. Generating multiple hypotheses using the Whisper Large model proves computationally intensive for most research environments. To illustrate, Generating hypothesis for the $GS_{S}$ dataset (11,323 data points) takes 1 hour 48 minutes by the Whisper Tiny model in a single NVIDIA A100-80GB GPU, while it takes around 37 hours with Whisper Large.\n\nAs per the reviewer’s interest, we also agree that having a model trained on the Whisper Large hypothesis would be a great assessment to the community and would enhance the comprehensibility of the paper. We have attached the results of using the hypothesis generated by Whisper Large for training our best-performing model (WL-M) on GigaSpeech Entertainment ($GS_{E}$) and Science & Technology ($GS_{S}$) datasets to investigate the performance. We could not run the experiments across all the datasets due to limited time. However, **_We commit to adding a related discussion_** to the camera-ready submission that depicts the performance when using the Whisper Large model to generate hypotheses across all datasets.\n\n\n| |$GS_{E}$|$GS_{S}$|$WER_{Avg}\\downarrow$|$WERR\\uparrow$|\n|:----|:----|:----|:----|:----|\n|Oracle with Whisper Tiny (Proposed method) |28.22|23.93|26.08|- |\n|WL-M with Whisper Tiny (Proposed method) |21.61|18.02|19.82|**24.01**|\n|Oracle with Whisper Large|21.78|18.09|19.93|-|\n|WL-M with Whisper Large|15.59|12.77|14.18|**28.86**|\n\nOur results showcase the effectiveness of our proposed method in benefiting both the Whisper Large and Whisper Tiny hypothesis-trained models. We sincerely hope that these findings offer valuable insights into the validity of performance comparison.\n\n***\n#### **Q1: Why is Adapter $A_L^i$ introduced?**\n\nThe proposed method is centred around the utilisation of two adapter modules: the $A_L$ adapter aims to fine-tune the LLaMA model, while the $A_W$ adapter focuses on the integration of acoustic features. Using only the $A_W$ adapter would apply transformations to audio features from the Whisper model, but it would not introduce any trainable components inside the language model to use the input audio features. Consequently, the necessity for the $A_L$ Adapter remains, as we still need trainable parameters inside the language model injected by $A_L$ to fine-tune the LLaMA Model.\n***\n#### **Q2: Did you try the LoRA adapter?**\n\nWe greatly appreciate your recommendation regarding the LoRA adapter. We did not incorporate it during the time of this submission. However, through subsequent experimentation in our Lab, we found that the Adapter approach still performs better than LoRA. This could be because LoRA has no nonlinearity to map complex boundaries. We have attached some results (WER) from our LoRA implementation below for reference. \n\n|                                           | *$GS_{E}$*  | *$GS_{S}$*  | *$GS_{M}$* |\n| ----------------------------------------------- | ------ | ------ | ----- |\n| WL_M (Proposed method) | 21.61  | 18.02 | 9.82 |\n|                 Replacing Adapters with LoRA of rank 16                                | 23.38 | 20.36 | 10.09 |", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. This paper proposed a novel and effective fusion mechanism for incorporating acoustic features into LLMs.\n\n2. The results are promising and the ablation study is detailed.", "reasons_to_reject": "1. The oracle n-best hypo baseline is a smaller model, which reduces the validity of performance comparison.", "questions_to_authors": "1. Why is adapter A_L^i introduced? It does not fuse any acoustic information.\n\n2. Did you try the LoRA adapter? It's a more widely used adapter for fine-tuning LLMs.\n\n", "paper_topic_and_main_contributions": "This paper introduces a cross-modal technique for correction in ASR, which combines both acoustic data and linguistic data. This is a new paradigm to inject acoustic features from a large acoustic model to a large language model. Experiments verify the effectiveness of this method.", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: **Originality:** 

This work is the logical next step for integrating LLMs with ASR models. Many similar prior works have explored this space, albeit only very recently. This work differs from others, for instance (Ma et al. 2023), primarily in the scale of the model  (Llama) and data (Gigaspeech) used for fine-tuning the error-correction model, as well as in the type of LLM (decoder only). The more novel aspect of this work is the fact that it gives LLMs more direct access to acoustic representations, through the Whisper encoder, rather than just through n-best lists or ASR outputs, through cross-attention.
 
**Significance:**

This work, as well as the topic of leveraging LLMs for ASR error correction is very significant.

**Clarity:**

The paper is well-written though there are a number of minor grammatical errors that could be fixed. As pointed out by some of the reviewers, some corrections to tables need to be made.

**Pros:**
   - A new method addressing an incredibly important problem
   - Thorough experimentation
   - Promising results

**Cons:**
   - This method should be evaluated in other settings.
   - Some minor concerns about reproducibility.

**General Comments**
Large pre-trained models such as Whisper and Llama are trained on large amounts of web-scraped corpora. Without knowledge of the training data, It is difficult to ascertain to what extent data leaking (training data in the test sets) is a problem. Evaluation sets must be carefully selected to ensure conclusions hold across data sets, and to limit the potential for data leakage. In this study most results are presented on web-scraped corpora. the results from ATIS are very encouraging, but it would be useful to see results on more datasets.

The authors seem to have addressed many of the concerns about using Whisper Tiny as well reproducibility.
Decision: Accept-Main