Let’s Fuse Step by Step: Generative Fusion Decoding Algorithm with LLMs to Enhance Speech Recognition and Text Image Recognition
Authors: Chan-Jan Hsu, Yi-Chang Chen, FengTing Liao, Pei-Chen Ho, Yu-Hsiang Wang, Po-Chun Hsu, Da-shan Shiu
Accepted at: acl25


Reviewer paper summaries: This paper proposes a late-fusion based approach to integrate large language models (LLMs) into text recognition processes (such as ASR and OCR). 

To address the differences between the tokenizers of LLMs and recognition models, the authors introduce a method to convert token-level probabilities into byte-level probabilities, enabling the LLM and the recognition model to assist each other during decoding without retraining the decoder of the recognition model. 

The effectiveness of the proposed method is demonstrated on both ASR and OCR tasks.
Summary of strengths: The authors propose a novel late-fusion based approach to leverage LMs to assist ASR and OCR without requiring additional training.
Summary of weaknesses: * The paper lacks a comparison with Chen et al. (2024b). The authors claim that the primary contribution of this work is proposing a training-free fusion method to integrate LLMs for assisting recognition tasks like ASR. However, they do not provide a comparison with prior training-based methods, such as Chen et al. (2024b), even though this method is mentioned in the paper.

* The authors should conduct experiments on more widely-used datasets, such as Aishell or the Hyporadise used in Chen et al. (2024b), to better evaluate the effectiveness of the proposed approach.


> Chen Chen, Ruizhe Li, Yuchen Hu, Sabato Marco Sinis- calchi, Pin-Yu Chen, EngSiong Chng, and Chao- Han Huck Yang. 2024b. It’s never too late: Fusing acoustic information into large language models for automatic speech recognition. In The Twelfth International Conference on Learning Representations.
Comments, suggestions and typos: ## Questions
*I am unclear about the rationale behind the setting of k in Equation 8. When the recognition model has already generated a byte sequence of length t, the LLM only scores the sequence from 1 to t−k. If this is the case, does it imply that converting probabilities from token-level to byte-level might not be necessary?
Overall assessment scores: 2.5
Metareview score: 3
Metareview: The paper introduces Generative Fusion Decoding (GFD), a novel methodology to integrate LLM rescoring in the ASR and OCR autoregressive decoding mechanism even in the condition of a different textual tokenization used by the LLM and the ASR/OCR model. To do so, GFD derives byte-level probabilities from token-level ones, which can then be combined.
Summary of reasons to publish: The strengths of the work are:

 - the method is novel and does not require re-training procedures, offering a method to seamlessly exploit LLM capabilities and learned knowledge in the ASR and OCR tasks;
 - handling the combination of the probabilities of mismatched tokenizations is an important contribution that may be a source for inspiration for other tasks and works;
 - the paper also analyzes the possibility to leverage the method with complex prompt to adapt the rescoring target (in-context learning);
 - the attached software with permissive license can be useful for the community to reproduce the work and build on top of it.