Kati8/ICLR_2023_2025 huggingface dataset:
authors: Sabato Marco Siniscalchi
reviews: 
Comments: To integrate acoustic information into the speech recognition output error correction using large language models (LLMs), the authors of this paper compare three fusion methods: early, mid, and late fusions. To further improve late fusion, this paper investigates adding uncertainty and proposes a new approach, Uncertainty-Aware Dynamic Fusion (UADF), introduced to integrate acoustic information, significantly enhancing word error rate (WER) and showing potential in audio-visual speech recognition.
Strengths: Overall, this paper's structure is easy to follow.
It reminds me of previous milestone works about RNN decoders: Shallow fusion, Cold fusion, and Deep fusion. And this paper also has the potential to be a milestone. 

The UADF is novel in this paper, and the improvements are guaranteed shown in Tables 1 and 2.
Weaknesses: ICLR conference is less specialized for speech researchers than ICASSP and INTERSPEECH. A brief and precise background introduction to the speech recognition framework is needed.

For the three frameworks in Figure 1, it needs to be clarified: where is N-best from? The N-best and the X_tok, X_enc, and X_dec are from different types of models according to the descriptions. It is better to mention these differences in the figures.

"language models have been widely utilized in ASR tasks over the past two decades," maybe there is a better way to express it; some earlier efforts are ignored.

Additionally, I believe you want to demonstrate that the proposed method is also applicable to audio-visual tasks. This requires a more detailed description; merely using a single paragraph here is insufficient.
Recommendation: 10: strong accept, should be highlighted at the conference
Reviewer confidence: 4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.
Soundness/correctness: 3 good
Clarity: 3 good
reviews: 
Comments: This paper introduces a late fusion strategy called "UADF" that combines the modalities of LLM and ASR to enhance generative error correction. The research demonstrates promising results across multiple datasets.
Strengths: 1. Integrating acoustic information with LLM is a promising strategy that enhances the capabilities and potential of LLMs in ASR tasks. This approach leverages the strengths of both acoustic processing and advanced language modeling, providing a more robust framework for GER.
2. The paper conducts a comprehensive examination of different fusion strategies, delving into detailed analyses of each approach.
3. The proposed UASF yields promising results across datasets with varying conditions, including ASR and VASR.
Weaknesses: I believe it's not entirely fair to make comparisons with GER given the use of acoustic information.  It would be better to add some comparisons of the results with LLM scoring.
Recommendation: 6: marginally above the acceptance threshold
Reviewer confidence: 3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.
Soundness/correctness: 3 good
Clarity: 3 good
reviews: 
Comments: Recent studies showed that large language models (LLMs) can be successfully used to map the N-best hypotheses list generated by an ASR system to the predicted output transcription to improve ASR accuracy. But LLM is not trained with acoustic information. This paper proposed a method named Uncertainty Aware Dynamic Fusion (UADF) to infuse acoustic information to the auto-regressive decoding process with ASR decoder and LLM predictions. Specifically, it uses late fusion to combine the probability output by ASR decoder and LLM to predict the recognition results. In this process, temperature scales are applied to the logits of ASR decoder and LLM to match up the model confidence and recognition accuracy. Besides, the ASR and LLM combination weights are decided by LLM entropy so when LLM’s uncertainty is larger, more compensation from the ASR model will be used to decide the recognition results. The proposed method showed obvious WER reduction for several ASR tasks.
Strengths: The paper proposed several methods applied to the fusion of ASR decoder and LLM output to improve performance. The paper compared the proposed method with several existing methods and did some ablation study to analyze the effectiveness of the proposed method. It also did experiments to prove the generalization of the proposed method. All the results showed the effectiveness of the proposed method.
Weaknesses: •	Some contents are not accurate or clear. 
o	In eq. (9), the entropy of LLM output is not accurate since for a discrete distribution, the entropy should be the sum of -p(y_t,i)logp(y_t,i). 
o	In figure 2, it’s clear the probability of “all” from ASR is high. But it didn’t show where is the probability of “all” from LLM ? And what’s the meaning to show the blue cross in the left bottom part and the green cross in the right bottom part? 
o	In the experiments part, for all late fusion experiments, the ASR model needs to be retrained with the training data. But it’s not clear whether all the training data are used to get a unified model or for each task (WSJ, ATIS, Chime-4), the model is updated only with the training data from this task. 
o	The results in table 2 showed that without calibration, the proposed UADF did not perform better than the static method (WER 1.39 vs. 1.36). This means calibration is crucial for UADF, so the tuning of temperature t1 and t2 becomes a key to make it work. But the paper didn’t give enough information on these two parameters, such as what’s the optimal value, should we tune them for different tasks and so on.
Recommendation: 6: marginally above the acceptance threshold
Reviewer confidence: 4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.
Soundness/correctness: 2 fair
Clarity: 2 fair
reviews: 
Comments: The authors present an approach they describe as Uncertainty Aware Dynamic Fusion (UADF) that 1. calibrates the LLM score at the token level accounting for the over-confidence of the neural network when doing auto-regressive decoding, 2. a time-changing uncertainty uncertainty that dynamically adjusts the decision-level fusion of text-based LLM and ASR scores. The results show significant improvement over strong baseline models such as Whisper and wav2vec or Hubert on a variety of tasks. Further the authors demonstrate that the results can be applied in an AVSR task with the dynamic uncertainty improving results in noise over a static fusion.
Strengths: Overall the work is well motivated and demonstrates strong empirical results.
Weaknesses: The point about calibration in using LLMs for rescoring of ASR hypotheses may be novel, however both calibration of hybrid or neural network ASR model scores and the use of strong language models combined with ASR scores are not novel approaches and have been widely used within speech. 

The paper seems to desire establishing the term "modality laziness" as a technical way of describing when multimodal system underachieve compared to a unimodal system; in this case I would describe this as a simply a poorly designed system. There are multiple papers in the literature that describe effective ways of combining audio and visual modalities to yield an improve AVSR system over any single modality. Look for the term "modality drop out" and work on audio-visual conformers or AV-hubert.

Applying the entropy of the LLM predictive posterior eqn (9)  as a weighting factor for the dynamic fusion neatly correlates with the desired property of: when the LLM is more certain, then the ASR weight of the ASR can be less, and when the LLM is uncertain, then the ASR weight is higher.  However, it doesn't stem from the statistical approach of ASR where 
Y_T = argmax Y_n p(X|Y_n; M_am) P(Y_n; M_lm ) where the first term in the argmax is the acoustic score and the second term the text-based LLM score, which could be computed with a LLM. In this light, without a Bayesian motivation, one might consider UADF as a hueristic method of combining LLM scores with ASR.

Lastly, the results here are all conducted on speech recognition tasks. Its unclear if such an approach can be applied to non-speech tasks or is of interest to the broader ICLR community.
Recommendation: 5: marginally below the acceptance threshold
Reviewer confidence: 3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.
Soundness/correctness: 3 good
Clarity: 3 good
reviews: 
Comments: This work addresses the integration of acoustic signals into speech recognition error correction via large language models by examining three distinct fusion strategies: early-stage, intermediate, and late-stage fusion. Building upon late-stage fusion, the authors introduce uncertainty considerations and develop Uncertainty-Aware Dynamic Fusion (UADF), a novel technique for incorporating acoustic information that yields substantial improvements in word error rate (WER) and demonstrates applicability to audio-visual speech recognition scenarios.
Strengths: The paper is well-organized and straightforward to navigate. The work draws meaningful parallels to influential prior research on RNN decoder architectures such as Shallow fusion, Cold fusion, and Deep fusion, and demonstrates similar potential for becoming a landmark contribution in the field. The proposed UADF methodology represents a genuine innovation, with empirical validation through comprehensive results presented in Tables 1 and 2 demonstrating clear performance gains.
Weaknesses: As an ICLR submission targeting a broader machine learning audience rather than the speech-specialized venues like ICASSP and INTERSPEECH, the paper would benefit from a more thorough and accessible introduction to the fundamentals of speech recognition systems. Regarding the three fusion architectures presented in Figure 1, clarification is needed regarding the origin of the N-best list. Since the N-best list and the various feature representations (X_tok, X_enc, X_dec) are derived from different model architectures according to the text, these distinctions should be made explicit within the visual representation. The statement concerning language models' widespread adoption in automatic speech recognition over the past two decades may warrant revision to provide more comprehensive acknowledgment of earlier foundational work. Furthermore, to substantiate the claim that the method extends effectively to audio-visual speech recognition, substantially more detailed exposition is necessary, as the current treatment with a single paragraph provides insufficient support for this assertion.
Recommendation: 10: strong accept, should be highlighted at the conference
Reviewer confidence: 4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.
Soundness/correctness: 3 good
Clarity: 3 good