reviews: 
Comments: This paper builds upon the recently proposed “generative error correction” (GEC) paradigm for speech recognition (ASR). The previous GEC work used a text-only LLM to map from a list of ASR hypotheses to a single transcription. In this paper, the authors propose to add acoustic information to this mapping process by including speech representation in the GEC. They experiment with several such “fusion” strategies (so-called early, mid, and late fusion), and finally show that late fusion with uncertainty awareness works best. Evaluations are conducted on a number of ASR benchmarks, showing significant WER improvements compared to GEC or ASR-only baselines, and the method is also shown to generalize to audio-visual ASR.
Strengths: 1. Using LLMs for ASR error correction is an interesting idea and has shown strong results in the past. ASR models often perform badly on rare words such as named entities, and LLMs can be a useful component to improve recognition of such words. In the earlier GEC work, only the top hypotheses of the ASR model was used for the final transcription. Since these hypotheses may not always contain information from the correct word, it is natural to also use the ASR information more directly. With these considerations, the ideas in this paper are relevant and a logical continuation in this line of work.

2. Incorporating modality-specific uncertainty using the UADF approach is a novel and interesting technique, and seems to provide reasonable improvements compared to static weighting of modalities. For example, in Table 4, we see that particularly on very low SNR conditions, there is a large improvement in WER when using UADF instead of static weights. I also appreciate that the authors show ablation results and analysis for the UADF strategy in Section 5.2.

3. In addition to traditional ASR benchmarks, the authors show that their UADF strategy can be used for audio-visual ASR, where it gives a small improvement over AV-HuBERT.
Weaknesses: My main concerns about the paper are about the presentation (in terms of relation with prior work) and comparison with baselines in the experiments, which I will detail below.

### This model in the context of prior work

In order to make this point clear, let me briefly summarize some key events in the history of the use of language models (LMs) in ASR.
- ASR was formulated as a noisy channel model using the Bayes rule P(W|X) = P(X|W)P(W). The two distributions were named an acoustic model and a language model, respectively. The LM was trained separately on source text, and only used for decoding [1].
WFST-based decoding provided a way to create a incorporate n-gram LMs into the decoding graph for efficient **first-pass decoding** [2].
- Incorporating larger n-gram LMs was hard (since the decoding graph explodes), so researchers used them instead in **second-pass rescoring** in both offline and on-the-fly settings [3, 4], by subtracting the original LM scores and adding back the larger LM’s scores.
- The Bayesian formulation still made sense in the era of hybrid HMM-DNN model, even when the acoustic models were discriminatively trained, since the scores could be interpreted as pseudo-likelihoods by subtracting an appropriate prior, and so the same decoding/rescoring framework carried over.
- In the era of “end-to-end” ASR, the models are trained to directly estimate P(W|X), and so log-linear interpolation with an external LM is not interpretable in the Bayesian sense anymore. Yet, practitioners still do this and call the process “fusion” (e.g. shallow fusion, cold fusion, etc. [5, 6]) — this is analogous to using LMs in the first-pass decoding.
- Analogous to the second-pass rescoring, we can obtain candidates using beam search on an ASR model, and then re-rank them with an externally trained LM (e.g. as done in the original LAS paper). So far, this “rescoring” step only manipulates the hypotheses, without considering the acoustic information. Let’s call this model (A).
- More recently, a two-pass E2E ASR model was proposed which has an encoder shared between a streaming RNN-T model and a full-context LAS decoder [7]. The idea was to perform a first-pass decoding with the RNN-T head, and use the decoded hypotheses along with the encoder representations to rescore using the LAS head in a *acoustic-guided rescoring* technique.
- In [8], the authors further built upon this two-pass model by proposing a **deliberation network**, which is an LAS decoder that attends to both the acoustic representations as well as the first-pass hypotheses, to generate the output. Let’s call this model (B). 

The Generative Error Correction (GER) model (which this paper extends) is analogous to model (A), with the exception that it formulates the rescoring problem as a many-to-one sequence transduction task, as opposed to a simple re-ranking task. 

The proposed model is similar to model (B), as is clear from equation (2). Its goal is to generate a new sequence given acoustic representations and an N-best list. In fact, the “early” and “mid” fusion methods in the paper are, in my opinion, exactly the same as a deliberation model, with the exception that the decoder is frozen instead of being jointly trained. 

Situating late fusion is a little more complicated since the fusion only happens at the scoring stage, and it is quite confusing to me exactly how to think about it. This “late fusion” is, in a way, similar to shallow fusion methods in E2E ASR, with the exception that a GER model is used in place of the external LM. The other view is how the authors present it: GER with additional acoustic information.

While it is completely acceptable to use new terminology that is more in keeping with broader advances in LLMs, I think it would be beneficial to situate the proposed method in the context of the above ASR+LM strategies to make the paper more widely accessible. In summary, the use of LMs in ASR can be through either first-pass decoding or second-pass rescoring (this paper falls into the latter). Second pass rescoring may be modeled as re-ranking or sequence generation, and methods may or may not use the original acoustic information. This kind of taxonomy and categorization can be used to clarify the contribution of this work. 

[1] Jelinek, Frederick. “Continuous speech recognition by statistical methods.” Proceedings of the IEEE 64 (1976): 532-556.

[2] Mohri, Mehryar et al. “Speech Recognition with Weighted Finite-State Transducers.” (2008).

[3] Ljolje, Andrej et al. “Efficient general lattice generation and rescoring.” EUROSPEECH (1999).

[4] Sak, Hasim et al. “On-the-fly lattice rescoring for real-time automatic speech recognition.” Interspeech (2010).

[5] Chorowski, Jan and Navdeep Jaitly. “Towards Better Decoding and Language Model Integration in Sequence to Sequence Models.” Interspeech (2016).

[6] Sriram, Anuroop et al. “Cold Fusion: Training Seq2Seq Models Together with Language Models.” Interspeech (2017).

[7] Sainath, Tara N. et al. “Two-Pass End-to-End Speech Recognition.” ArXiv abs/1908.10992 (2019): n. pag.

### Evaluation problems

If we accept the above categorization of models, the paper can be viewed in 2 ways: (i) shallow fusion of ASR and GER, and (ii) GER with acoustic information. As such, the proposed method should be compared against the following baselines: (1) end-to-end ASR, (2) shallow fusion of ASR and LLM, and (3) GER-only. The authors have compared against (1) and (3), but not against (2), which may be important to show how using GER improves compared to simply using a LLM with shallow fusion.

This is important because training the GER itself requires generating N-best hypotheses for the whole training data, which is computationally expensive, and it can only be used in an offline manner, while simple shallow fusion can be done on-the-fly. As such, we expect to see significant WER gains when using the GER module.

Another concern, which is a common issue when using LLMs, is the following: *How can we ensure that the test data was not seen by LLAMA?* In particular, the authors show in Table 3 that we get a large improvement on CHiME-4 when using the proposed method. CHiME-4 is essentially WSJ read outdoors, and WSJ consists of text from newspapers, which are in the public domain. It would be more useful to conduct experiments on a closed dataset, such as CHiME-5, which is less likely to have been memorized by LLAMA.

### Other minor comments

1. Some sections of the paper use a lot of flowery language which can be avoided. For example, in the last paragraph in Section 1, extraneous terms such as “a pioneering UADF technique”, “adroitly allocates”, and “conspicuously outperforming” can be made concise. In general, authors should stick to reporting rather than embellishing.
2. In Section 2 (paragraph 1), the authors mention that language models have been used in ASR for two decades, but the oldest citation is only from 2019. I think they should conduct a more thorough literature review in order to perform correct credit attribution.
3. The use of the terms “early”, “mid”, and “late” fusion in the paper may be confusing for readers familiar with ASR methods such as shallow, cold, and deep fusion. The authors should consider either changing *fusion* to another word, or making the difference explicit in the paper.
Recommendation: 6: marginally above the acceptance threshold
Reviewer confidence: 5: You are absolutely certain about your assessment. You are very familiar with the related work and checked the math/other details carefully.
Soundness/correctness: 3 good
Clarity: 3 good