Kati8/ICLR_2023_2025 huggingface dataset:
authors: Yuchen Hu
authors: CHEN CHEN
authors: Chao-Han Huck Yang
authors: Ruizhe Li
authors: Chao Zhang
authors: Pin-Yu Chen
authors: EngSiong Chng
reviews: 
Comments: In a previous study, generative error correction (GER) is achieved by learning the mapping from ASR N-best hypotheses to ground-truth transcription through efficient LLM finetuning. This paper extends this idea and focuses on noisy conditions. To avoid the cross-modality gap, the authors propose a novel idea to extract a language-space noise embedding from the N-best list to represent the noise conditions of source speech. Furthermore, in order to enhance its representation ability of audio noise, a knowledge distillation (KD) approach via mutual information estimation (MINE) is employed. The experiments show that the proposed method can significantly outperform the conventional LM rescoring baseline. Several additional experiments are also included in the Appendix which provide more insight into the proposed RobustGER. Overall, the paper is very clear and well-written. It describes the problem and explains the solution well. The experiments done are reflective of the proposed model's performance.
Strengths: 1)	Novel idea to apply LLM for noise-robust ASR.
2)	Extract language-space noise embedding with knowledge distillation based on mutual information.
3)	Good performance improvement.
4)	Plenty of experiments and ablation studies.
5)	Insightful discussions, such as t-SNE visualization, and the relationship between noisy speech and n-best list diversity.
Weaknesses: The way to extract the audio noise embedding is from the ASR encoder (i.e., Whisper Large-V2). This may only make sense for the Whisper ASR, as a recent paper [r1] pointed out that the noise-robustness of Whisper does not come from noise-invariant, but recognizes speech conditioned on the noise type. In summary, the Whisper encoder is a suitable model to extract noise information. On the other hand, other ASR models may not have such ability and they achieve noise-robustness through the noise-invariant encoder. If this is the case, those ASR encoders may not be suitable to extract audio noise embedding. A discussion and simple experiment about this would be great.


[r1] Gong, Y., Khurana, S., Karlinsky, L., & Glass, J. (2023). Whisper-at: Noise-robust automatic speech recognizers are also strong general audio event taggers. arXiv preprint arXiv:2307.03183.
Recommendation: 8: accept, good paper
Reviewer confidence: 4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.
Soundness/correctness: 3 good
Clarity: 4 excellent
reviews: 
Comments: This work extends an established benchmark of generative error correction with a new "HyPoradise" dataset, in order to enable LLMs to perform error correction. The study presents application on noisy-robust speech recognition and claims that it reaches up to 53.9% improvement on word error rate.

The study itself follows the latest trend of research, where LLM is used to
Strengths: 1. The work itself holds certain level of novelty, with good review on earlier literatures on both error correction in ASR and LLM.
2. The methodology is clearly presented, along with the novelty of the paper.
3. With some ambiguities in the middle, the paper itself clarifies the idea with experiments subtlely.
Weaknesses: 1. I think the topic of error correction might be a poor fit to the conference. But perhaps I am wrong on this so correct me if so.
2. There lacks the practical discussion on additional workload, especially on resources.
3. The description of building the embedding space is somehow confusing in particular terms. For example, in Section 4.2.1 - what is "diversity similar to variance"?

Minor issues:
1. Section 5.4 - What is Table 14?
2. I suggest to put the definition of embedding a bit earlier from the beginning of Section 4. Otherwise, Figure 2 looks a bit confusing.
Recommendation: 6: marginally above the acceptance threshold
Reviewer confidence: 4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.
Soundness/correctness: 2 fair
Clarity: 3 good
reviews: 
Comments: This work addresses noisy conditions by proposing a language-space noise embedding derived from ASR hypotheses to aid denoising with large language models (LLMs)-based generative error correction (GER). A knowledge distillation strategy further enhances noise representation. Tests on various LLMs show up to 53.9% improvement in word error rate with limited training data, demonstrating the effectiveness of the proposed noise embedding and denoising ability of LLMs.
Strengths: From a generally purposed GER benchmark to a more focused noise-robust problem, it is a suitable extension in depth and kind of milestone using LLM for robust ASR.
Weaknesses: The illustrations of (b) GER with audio-space denoising (Zhang et al., 2023b; Fathullah et al., 2023) and (c) GER with language-space denoising (ours) are a little challenging to follow. Is denoised audio directly fed to the LLM adapter, or is there something else you want to express?

We noticed that the HP database only comes from the n-best of a few models. Is it possible to introduce more diverse system outputs from various models?

According to the method in the article, Section 4.3 should be the most important part, relatively speaking. Unfortunately, the space allocated to it in the article is too cramped—too many things to fit into this small section, which is not very reader-friendly.
Recommendation: 10: strong accept, should be highlighted at the conference
Reviewer confidence: 4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.
Soundness/correctness: 4 excellent
Clarity: 3 good
reviews: 
Comments: This paper addresses the noise-robust ASR. The method is based on the LLM-based generative error correction (GER), but contrary to the existing approaches, it extract the noise information from the N-best hypotheses of transcription languages, not directly from the audio. The proposed method significantly outperformed the existing baseline and advanced the area of noise-robust ASR.
Strengths: Originality:
While it would normally be better to reduce the noise before ASR or to estimate the noise directly from the acoustic data, the idea of doing noise estimation from N-best transcription hypotheses is interesting. It is interesting that this method avoids the difficulty of cross-modal fine-tuning by doing so. It is also compelling because it seems that humans actually perform similar processing in noisy environments.

Quality:
The paper is rich in evaluation, and the improvement interval is significant compared with the existing baselines. Also, it provides theoretical hypotheses why the proposed method works better that sound convincing.

Clarity:
The description appears complete. I have not read all the details, but I get the impression that this paper is very well organized.

Significance:
The task addressed is clearly significant because it has many practical applications. The novel approach presented in this paper is also be interesting, and I think it can potentially be applied in other modalities.
Weaknesses: - It was very difficult to find anything to explicitly criticize about the technical content of the paper. There may be flaws and room for improvement, but that is no longer something to do at the peer review stage of this paper.
- If I had to say something, I was concerned that the notation seemed sometimes inconsistent.
Recommendation: 8: accept, good paper
Reviewer confidence: 3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.
Soundness/correctness: 4 excellent
Clarity: 4 excellent