CIEASR:Contextual Image-Enhanced Automatic Speech Recognition for Improved Homophone Discrimination

Venue: acmmm_org_ACMMM_2024_Conference
Published: 2024-04-09 09:50:09

Abstract: Automatic Speech Recognition (ASR) models pre-trained on large-scale speech datasets have achieved significant breakthroughs compared with traditional methods. 
  However, mainstream pre-trained ASR models encounter challenges in distinguishing homophones, which have close or identical pronunciations. 
  Previous studies have introduced visual auxiliary cues to address this challenge, yet the sophisticated use of lip movements falls short in correcting homophone errors. 
  On the other hand, the fusion and utilization of scene images remain in an exploratory stage, with performance still inferior to the pre-trained speech model.
  In this paper, we introduce Contextual Image-Enhanced Automatic Speech Recognition (CIEASR), a novel multimodal speech recognition model that incorporates a new cue fusion method, using scene images as soft prompts to correct homophone errors.
  To mitigate data scarcity, we refine and expand the VSDial dataset for extensive experiments, illustrating that scene images contribute to the accurate recognition of entity nouns and personal pronouns.
  Our proposed CIEASR achieves state-of-the-art results on VSDial and Flickr8K, significantly reducing the Character Error Rate (CER) on VSDial from 3.61\% to 0.92\%.

Main paper: 
 related work 2.1 pre-trained speech models: 
 The field of speech recognition significantly progressed with the introduction of unsupervised pre-training techniques such as Wav2Vec 2.0 [4] , HuBERT [17] , w2v_BERT [11] , etc. The unsupervised pretrained speech models analyze vast amounts of data to capture the underlying patterns of speech. However, they require an additional fine-tuning step due to the lack of a decoding mapping to useful outputs. Whisper [37] is a speech recognition model pre-trained in a weakly supervised fashion across large-scale speech datasets. Goron et al. [14] , Moor et al. [30] , Zhang et al. [53] , among others, adopt Whisper as a backbone for downstream tasks due to its flexible Encoder-Decoder architecture and robust speech pre-training capabilities. In this work, Whisper is adopted as our foundation model to obtain strong speech recognition performance.Traditional speech recognition methods have primarily focused on robustly and effectively transferring speech to text using only auditory inputs [15, 16, 47, 50] . The integration of multimodal cues into ASR systems has been identified as a potent means to enhance robustness and accuracy [2] . Commonly used cues include lip movements [28, 34, 40, 41, 49] and gestures [29] . Notably, lip movements require precise synchronization of video frames and speech segments to be effective [35] , and cannot adequately address the problem of correcting homophone errors. Recent studies have increasingly investigated the application of readily accessible visual contexts, including images and videos, to enhance recognition performance [13, 26, 31, 36, 42, 43] . Srinivasan et al. [42] found that visual context can help ASR systems recover masked speech of objects by using images as auxiliary signals. Based on this discovery, Srinivasan et al. [43] automatically detects object proposals and directly grounds speech into regions of an image, Ma et al. [26] adopts a self-supervised pre-trained text-video embedding model to extract visual information and improve the recognition of specific words. Other applications include extracting visual context by a visual detector for embodied agents [36] and exploring various crossmodal fusion schemes to combine visual and linguistic information to make speech recognition more accurate and versatile [31] .Multimodal models, such as GPT-4V [1] and Gemini [46] , have demonstrated considerable versatility across diverse fields. Existing multimodal models primarily leverage Visual-Language Pretraining (VLP) techniques to align vision and language representations [7] . Typically, VLP models that emphasize multimodal understanding comprise three components: Modality Encoder, LLM Backbone, and Input Projector which transforms encoded visual features into linguistic embedding space [52] . This methodology is adaptable to other modalities, such as audio and video, through modifications to the encoding mechanism [9, 22, 51] . LLaVA [23] , Qwen-VL [5] , Qwen-Audio [9] , among others, use an MLP to map encoded features to textual feature space. Flamingo [3] fuses compressed input feature sequence with text through cross attention. Q-Former [20] extracts relevant features from the encoded features with fixed-length learnable queries and treats the selected features as cross-modal prompts, thereby enhancing the model's interpretative and generative capabilities. 
method 3.1 model architecture: 
 The CIEASR model is a novel approach for addressing homophone errors in speech recognition by leveraging the contextual information of scene images. It incorporates scene image cues as soft prompts into the speech recognition framework, distinct from traditional fusion approaches. As illustrated in Figure 2 , raw images are encoded by a Visual Encoder. Encoded image features are distilled and condensed into high-level semantic features through a Q-Former. These semantic features are aligned with the textual prompt interface space of the Whisper model through a projection layer, enabling the speech recognition system to gain nuanced insights derived from images without fine-tuning the Whisper model itself. This approach maintains Whisper's multilingual and robust capabilities while integrating image understanding. Specifically, the handling of scene images via Q-Former is presented in section 3.2 while the fusion strategy and the speech decoder are presented in section 3.3To enhance the speech decoder's capacity for effective interpretation of image information, we employ the Q-Former [20] for the strategic extraction of semantic visual features. This approach aligns encoded visual representations seamlessly with the textual space. The training of Q-Former is divided into two stages:The first phase involves imagetext contrastive learning, where images 𝑖 are transformed into fixedlength image embeddings 𝐸 𝑖 by a visual encoder EVA_CLIP [45] . 𝐸 𝑖 interact with learnable queries 𝑄 𝑖 through cross attention. Training is conducted using image-text pairs with a contrastive loss, which includes Image-Text Contrastive Learning (ITC), Image-grounded Text Generation (ITG), and Image-Text Matching (ITM). The training in the first phase effectively extracts semantic information from images and compresses the length of the image embeddings.After extensive training with paired image-text data endows the Q-former with the ability to extract image representations, Q-Former is linked to a frozen Whisper decoder through a fully connected layer to establish a bridge between the image space and the textual space of the Whisper decoder. At this stage, our model performs autoregressive predictions of textual labels Y corresponding to both speech and image inputs. The detailed Speech Decoding process is presented in Section 3.3.The addition of visual context 𝑋 𝐼 from Q-Former significantly enhances the decoder's understanding of semantic content within speech signals, bridging the gap between visual inputs and textual transcriptions. This integration approach ensures the generation of textual labels coherent with both the acoustic information and visual context, leading to improved accuracy and robustness in speech recognition tasks.In large-scale pre-trained models employing autoregressive decoding, the critical function of text prompts in determining model outputs has been well-documented. Research in prompt engineering [18, 19, 48] demonstrates that meticulously designed prompts can significantly enhance the performance of generative models. Drawing on these insights, our study explores using contextual images as soft prompts in the Whisper framework, aiming to harness their potential to refine the model's output.Though Whisper may not strictly be a Large Language Model (LLM), its autoregressive decoding aligns with LLM functionalities, endowing it with substantial inferential capabilities. With scene image cues provided by Q-Former, the Whisper decoder essentially assumes the role of a language model, possessing capabilities for inference and error correction. It produces textual predictions 𝑌 that mirror speech transcriptions 𝑆, utilizing image semantics 𝑋 𝐼 as prompts for enriched speech transcription.The origin speech 𝑠 is first transformed into a log-magnitude Mel spectrogram and then preprocessed by the Speech Processor, detailed in section 4.2. Then the processed speech signal 𝑆 is extracted to a series of acoustic features 𝑋 𝑆 by the Whisper Encoder, which are then fed into the decoder using cross attention. This attention mechanism allows for a seamless blend of auditory and visual data in generating speech transcriptions. We have adapted the prompt interface of the Whisper decoder, shifting from text token IDs to image embeddings 𝑋 𝐼 . Utilizing the vocabulary of Whisper, we convert text label IDs into text embeddings 𝑌 , allowing them to be concatenated with 𝑋 𝐼 .For a text label 𝑌 with a length of 𝐿, the decoding process of the Speech Decoder can be represented by the equation (1):EQUATIONIn this equation, 𝑌 represents the final output transcription text. 𝑋 𝑆 denotes the speech features obtained from the Speech Encoder, while 𝑋 𝐼 refers to the aligned image features derived from the Visual Encoder and Multimodal Alignment Module. 𝑋 𝑠𝑝 represents special tokens controlling the model's output, including language specification, task definition, and whether to use timestamps. During the decoding process, 𝑋 𝑆 is inputted into each decoder block through cross-attention mechanisms. The special tokens 𝑋 𝑠𝑝 and text embeddings generated from previous outputs 𝑌 𝑖< 𝑗 are concatenated with the image representations 𝑋 𝐼 and jointly fed into the Speech Decoder module for decoding. 
experiments 4.1 datasets: 
 We conduct our experiments on the multimodal dataset VSDial and Flickr8K.VSDial is a synthetic dataset consisting of 120,000 images. Each sample includes one image, its English description, and ten rounds of dialogue surrounding the image. The images are sourced from COCO and VisDial, while the synthetic speech is generated from VSDial using Fairseq to convert each image's corresponding ten segments of text questions into speech. The image caption serves as a linguistic cue for speech recognition. We have expanded the synthetic speech to include both questions and captions. We develop two versions of the caption in Chinese and English. Using Paddle-Speech for multi-speaker voice synthesis, we address the issue of the original VSDial dataset neglecting English words interspersed within Chinese sentences, ensuring comprehensive language processing. The expanded dataset VSDial-caption is set to be accessible for download under an open-source license, facilitating broader research utilization. For distinction, we refer to the original VSDial set as VSDial-question.Flickr8K is a real speech dataset, serving as a subset of the larger image corpus Flickr30K, wherein captions are articulated by human voices. It encompasses 8,000 images, each associated with five unique English-recorded captions.Flickr8K totally contains a total of 30,000 audio samples for training and 5,000 samples each for validation and testing.We conduct experiments on the original dataset VSDial-question, our expanded dataset VSDial-caption, and the real speech dataset Flickr8K.For every ten pairs of questions and answers of an image in VSDial-question, only one question is randomly used, resulting in a total of 120,000/2,000/8,000 training/validation/testing samples, respectively. For the Flickr8k dataset, one corresponding speech sample is randomly selected for each image, resulting in a total of 8,000 samples for one training epoch.We conduct experiments utilizing VSDial-caption across three linguistic settings: Chinese, English, and a random selection of Chinese and English speech utterances which is analyzed in section 4. 4 .In practical training, we employ a ViT model with a resolution of 224 × 224. We initialize a Q-Former model which has completed stage one training, and the Whisper large-v2 model, with both the ViT and Whisper models being frozen. Consequently, our model is highly lightweight, with a parameter count of 1.0B when using CIEASR and 40k when using learned queries alone.For speech preprocessing, speech utterances are uniformly resampled to 16,000 Hz and transformed to a log-magnitude Mel Spectrogram, computed using 25ms windows with a stride of 10ms. Inputs undergo global scaling to ensure they range between -1 and 1, achieving an approximately zero mean across the pre-training dataset.Generally, for speech in logographic languages such as Chinese, researchers pay more attention to the Character Error Rate (CER), whereas for alphabetic languages like English, the Word Error Rate (WER) is of greater concern. We provide both metrics for reference. In speech recognition systems, both scene image cues and textual cues offer vital contextual information. Textual cues within ASR systems deliver a focused subset of semantic information. In our experiment, image captions serve as highly pertinent textual cues for speech utterances linked to the respective image, thus qualifying as textual cues. Moreover, scene image cues not only furnish semantic information exceeding that provided by textual cues, encompassing aspects like background, texture, and spatial relationships but also reduce the high costs associated with manually designing textual cues in speech systems.Upon examining the differences between scene image cues and textual cues, we find that scene images offer abundant visual information. This wealth of information further enhances automatic recognition performance, specifically resulting in a 1.5 percentage point improvement of CER compared to using textual cues in the same language.In an analysis comparing the use of textual cues across various languages, we observe that employing Chinese textual prompts leads to an improvement in the performance of Chinese speech recognition. In contrast, the effect of cross-lingual textual prompts appears to be less significant. Quantitatively, utilizing textual cues in the same Chinese language enhances performance by 1.2 percentage points of CER, whereas different English language cues lead to a mere 0.3 percentage point increase, underscoring a limitation of textual cues in a cross-lingual context.Furthermore, a common phenomenon is observed. In both our model and the VILAS model, the performance of simultaneously using both image and textual cues tends to be slightly lower compared to using scene image cues alone. This may be attributed to the scene image encapsulating the semantics of the textual cues described in the image, rendering the textual cues redundant in this context. However, the potential for further enhancing ASR system performance by simultaneously using scene images and textual cues that provide non-overlapping semantic information is an area left for future research in our work on the field of multimodal complementary fusion. As shown in Table 2 , the following conclusions can be drawn:• From the Chinese experiments, it can be deduced that due to the dataset's characteristics that the translated captions contain a significant number of numerals and phrases, the performance with the introduction of images is slightly lower than in the question experiments. However, there is still a considerable improvement of 1.3 percentage points of CER. • The performance in the English experiments meets expectations, aligning with the higher WER and lower CER typically observed in English sentence recognition, with an enhancement in performance upon introducing images; there is a 2.2 percentage point improvement in WER. • We also add mixed-language experiments and find that the performance of the ASR system lies between the Chinese and English experiments. This suggests that utilizing the Whisper multilingual pre-trained model as the backbone for the ASR system provides certain robustness to mixed languages. In conclusion, contextual images, as a form of visual cue, can effectively enhance the entity information recognition capability of pre-trained language models, all without the need for any finetuning and efficiently improving system practicality with low computational resources. Compared with textual cues, image cues, as a denser form of information, can provide more context knowledge independent of text, and image information does not have the cross-language limitations of textual prompts. The preliminary mixed-language experiments in Chinese and English also align more closely with the applications of real-world speech recognition, offering a solution for complex multilingual mixed speech recognition.In the context of the real-world speech dataset Flickr8K, the pretrained Whisper model showcases a notable advantage in English speech recognition, surpassing the performance of most conventional models. These traditional models often conclude that incorporating visual cues fails to offer additional improvements. Ni et al. [31] suggests that this phenomenon occurs because speech serves as the primary input for ASR systems, with visual cues playing As shown in Table 3 , our model enhances the ASR capabilities built on the Whisper by integrating visual cues as soft prompts. Given that the visual component functions independently and does not contribute directly to the loss computation, it effectively serves as a soft prompt for speech decoding. This approach to cue integration minimizes the disruption caused by redundant multimodal information, emphasizing the supportive function of multimodal cues instead. Our findings offer fresh perspectives on cue integration techniques, promising valuable directions for future research in multimodal speech recognition.Figure 3 : Ablation study on learned prefix matrix: exploring the impact of trainable prefix on model performance.In our approach, we froze the Whisper model to ensure its original configuration remains unaltered. Recent studies [21, 24, 25] have indicated that adding a prefix to pre-trained models significantly enhances their performance, and our approach of incorporating an image soft prompt similarly introduces a prefix-like element.To discern the unique contribution of contextual visual cues beyond the impact of prefix addition, we design an ablation study utilizing learnable parameters that mimic the form of the image prompt but function as blank cues. This setup aims to neutralize the potential performance enhancements attributed solely to the prefix structure, allowing for a focused examination of the impact of visual cues. The rationale behind this method is to isolate and quantify the specific advantages that contextual images confer on speech recognition, thereby clearly distinguishing between the effects of prefix addition and the substantive enhancement provided by visual information. This approach enables a detailed analysis of the effectiveness of visual cues while keeping all other model hyperparameters and training conditions constant, as detailed in Figure 3 .Specifically, we define a learnable parameter matrix with the same shape as the Q-Former queries, which can be considered a blank image cue. This approach aims to isolate the effect of introducing this prefix matrix on the overall ASR performance of the Whisper model. Our experiments meticulously assess the impact of integrating contextual visual cues alongside the structural addition of a prefix to the Whisper model. The experimental results are presented in Figure 4 .We can conclude that:• The introduction of a prefix prompt has been observed to enhance the performance of speech recognition systems, particularly in synthetic speech datasets. Synthetic speech collections harbor more distinct speech domain features, which can be effectively captured and learned through the utilization of prompts. Similarly, although real-world speech datasets present a higher level of complexity, they also exhibit domain-specific characteristics, such as speakers' vocal habits and regional dialects, which still hold potential for further investigation in future research endeavors. • Compared to prefix prompts, image prompts can further enhance performance, especially within Chinese speech datasets, as indicated by the first two columns in the table. This improvement can be attributed to the widespread presence of homophones in Chinese. In section 4.7, we delve deeper into the corrective capabilities of image prompts concerning homophones. Through the comparison with the blank image experiment, we observe its adaptability to the features of the speech dataset, which can enhance the ASR recognition accuracy to a certain extent.Pronoun Recognition We investigate the contribution of contextual images to the recognition of specific types of vocabulary by analyzing the accuracy of noun recognition as well as pronoun recognition.We use the Spacy model to identify nouns and pronouns in 8,000 samples each from the VSDial-question and VSDial-caption test sets and 5,000 samples from Flickr8K test sets. From this, we calculate the Noun Recognition Rate (NRR) and Pronoun Recognition Rate (PRR), with the results presented in Table 4 and Table 5 .The VSDial-caption, being a comprehensive description of images, frequently utilizes numerals and thus contains more nouns and fewer pronouns. Conversely, VSDial-question primarily poses questions about the most prominent entities in images, containing a higher frequency of personal pronouns and exhibiting stronger significance. In English, the phenomenon of homophonic nouns and pronouns is less common, hence the system's correction capability is not significant for the Flickr8K dataset.The provision of contextual image cues in the VSDial-question dataset resulted in improvements of 3.66% for NRR and 10.85% for PRR; in the VSDial-caption dataset, these cues led to enhancements of 1.75% for NRR and 2.75% for PRR. Utilizing a prefix also significantly boosts performance, primarily in the case of abstract nouns, as well as demonstrative pronouns, relative pronouns, and indefinite pronouns, where images cannot provide auxiliary support.To more vividly illustrate the impact of scene image cues, we delve into error examples from VSDial in the absence of such cues, highlighting the corrective potential of image cues. This analysis is clearly depicted in Figure 5 .The VSDial-question dataset exhibits a high correction rate for erroneous samples, achieving 75% for nouns and 82% for pronouns. Conversely, the VSDial-caption dataset shows correction rates of 50% for nouns and 36% for pronouns. This discrepancy arises because the question format directly inquires about the image, focusing more on the most prominent entities within it, and includes a greater number of personal pronouns that can be corrected. On the other hand, the caption format provides a comprehensive description of the image, prioritizing a thorough depiction of entities and tending to describe using numerals rather than pronouns. We can conclude that contextual images make a significant contribution to the identification and correction of entity information in homophone errors within the Chinese speech dataset. Our method offers a viable and rapid solution for the recognition of homophones, which can alleviate to some extent the high costs and inconvenience of artificial auxiliary cues in ASR systems.In order to further analyze our CIEASR model for enhanced transparency and interpretability, we have integrated the Grad-CAM [39] Figure 6 : The visualization of activation maps between speech and the scene image.technique to examine the impact of image information on the probability of label predictions made by the whisper decoder. More precisely, we initially acquire the probability distribution 𝑦 𝑐 for each word as predicted by the whisper decoder for the 𝑐 𝑡ℎ token, and compute the gradient of 𝑦 𝑐 with respect to the final feature map 𝐴 extracted by the vision encoder, expressed as 𝜕𝑦 𝑐 𝜕𝐴 . Subsequently, the gradients derived via backpropagation are averaged across each channel dimension for every pixel value-reminiscent of Global Average Pooling-to determine the significance weights for each channel, delineated by Equation 2:EQUATIONHerein, 𝑍 signifies the count of pixels within the feature map, and 𝐴 𝑖 𝑗 symbolizes the pixel value at the 𝑖, 𝑗 position of the 𝑘 𝑡ℎ feature map.Utilizing the importance weights ascertained heretofore, the channel features of the feature map are weighted accordingly, and through the application of a ReLU function, the activation map is procured, as demonstrated in Equation 3:EQUATIONThe utilization of the ReLU function is intended to exclusively consider those pixels that positively influence the token 𝑐.During the practical analysis, we eschewed token-level predictions in favor of utilizing the predicted probabilities of each individual word to construct the activation map. This methodology enabled us to scrutinize the impact of each pixel on the predictive accuracy of every word. We curated a selection of images that exemplified the model's proficiency in correcting errors. By leveraging the Grad-CAM approach, we illustrated the critical image regions that were instrumental in the model's sequential word predictions.As shown in Figure 6 , the Chinese speech means "How many giraffes are there?" Without scene images, "giraffe" is confused with the phonetically similar "scene objects". When our scene images are presented, our model is adept at correcting errors to predict the correct word "giraffes". Furthermore, by analyzing the activation map, it becomes evident that the image region corresponding to the giraffes plays a pivotal role in the prediction of the word "giraffes", while this area does not attract attention during the prediction of other words. This indicates that the error correction capability of our model is attributed to the presence of giraffes in the image.In a similar vein, in the second example, the completely identical pronunciation of third-person pronouns (he/she/it) in Chinese poses a challenge for pure speech recognition systems to determine the semantic gender based solely on the speech signal itself. However, the scene image provided offers visual cues about the person's gender, and it is observed that this specific region contributes significantly to the accurate prediction of the correct pronoun "he". In contrast, when predicting the incorrect pronouns "she" or "it", there is almost no attention paid to the image (in our system, the probabilities of "she" and "it" are extremely low).The focus areas of the activation map provide excellent interpretability for our method of integrating scene images as soft prompts. The fusion of contextual visual information is profoundly effective, especially for homophone confusion errors involving entity nouns and pronouns. 
conclusion: 
 In this study, we introduce CIEASR, a pioneering approach to ASR that leverages unsupervised pre-trained speech models and enhances them with visual contextual information, particularly scene images, to address the challenge of homophone discrimination-a task where traditional speech-only models falter. By integrating image semantic information as soft prompts, our model not only improves interpretability but also paves the way for innovative cue fusion methods in speech recognition. The efficacy of CIEASR is underscored by its achievement of state-of-the-art results on both VSDial and Flickr8K datasets. Our comprehensive analyses further validate that scene images significantly contribute to the precise recognition of entity nouns and Chinese personal pronouns with identical pronunciations. Moving forward, we will further investigate how to more effectively integrate an increased range of multimodal cues into speech recognition systems under the paradigms of pre-training and prompts, aiming to study the complementary mechanisms between multimodal information.  
: 
 Previous studies have demonstrated the potential benefits of integrating visual cues such as lip movements [41, 49] and gestures [29] into ASR systems. They are suitable for enhancing speech recognition systems' noise robustness and recognition performance in scenarios such as multi-person meetings and intelligent cockpits [8, 32, 54] . However, these methods require strict synchronization of timestamps, posing a relatively challenging requirement for real-world applications. Because relying solely on lip movements allows for the transcription of spoken content into text [10, 12, 27] , such visual cues essentially act as a visual representation of speech, rather than offering visual semantic information. Therefore, these approaches remain inadequate for tackling the challenge of homophone discrimination.A viable solution leverages semantic-level visual context information to supplement and enhance referential information missing in speech, thereby clarifying the actual meaning of homophones. Previous work has shown that contextual images can enhance the robustness and accuracy of ASR systems. Srinivasan et al. [43] adopts the Faster-RCNN model [38] to obtain fine-grained visual representations and conducts a detailed analysis of the alignment between visual and auditory elements, confirming the reliance of multimodal speech recognition models on relevant visual contexts during prediction. Ni et al. [31] utilizes context-dependent visual and corresponding linguistic cues to correct homophone errors distinguishable from scene images. However, their methods of integrating cues, predominantly through concatenation [33] , attention mechanisms [43] or Multi-Layer Perceptron (MLP) fusion modules [31] , are in a developing stage of utilizing contextual cues. These methods do not yet match the recognition performance of the cue-less speech pre-training model Whisper in terms of recognition capabilities.To effectively address the challenge of correcting homophone errors, it is essential to better utilize the semantic information within scene images. Inspired by the multimodal alignment strategies of Multimodal Large Language Models (MLLMs) [6, 20, 23] , we integrate scene images as soft prompts, diverging from the traditional methods of integrating cues. Specifically, we utilize the Q-Former [20] , an efficient image-text alignment model, to extract semantic information from scene images which provides richer content than textual prompts such as key phrases, captions, and descriptions. To address the scarcity of tri-modal alignment data among images, speech, and text, we revise and expand the VSDial dataset. Furthermore, we exploit the prompt interface of the Whisper model to integrate image-derived soft prompts in a seamlessly natural manner.The performance of our proposed model demonstrates on both the synthetic dataset VSDial and the real dataset Flickr8K, with speech recognition error rates decreasing from 3.61% and 2.42% to an impressive low of 0.92% and 2.05%, respectively. We indicate that when it comes to speech recognition involving homophones, the relevant entity information provided by contextual images can accurately identify key vocabulary within the speech. Our contributions are summarized as follows:• We introduce CIEASR, a novel multimodal ASR model that enhances the contextual visual comprehension of ASR systems, offering an effective solution to the challenge of homophone discrimination. • We propose a new method of cue integration, incorporating scene image cues as soft prompts under the pre-training paradigm which yields significant improvements in the recognition of entity nouns and pronouns. • We revise and expand the VSDial datasets and conduct comprehensive experiments on both VSDial and Flickr8K, demonstrating the significance of scene images in ASR systems. CIEASR achieves new state-of-the-art results on these datasets. 
 
 Appendix:  


Decision: All reviewers agree that the proposed method is interesting and innovative. All reviewers comment that utilizing scene images as soft prompts to correct homophone errors in ASR is novel and contribute significantly to the increased performance of the presented approach. The reviewers commented that generalisation capability of the model should be further investigated (in terms of different languages, accents, etc.). The rebuttal addressed majority of these additionally raised questions. After reading the rebuttal, one of the reviewers upgraded the final score. One reviewer lowered their score but the additional concerns raised by that reviewer are not about any of the major quality points of the presented work but represent a further request for additional auxiliary explanations. The authors are therefore recommended to include a detailed explanation of computational complexity, diversity of the VSDial and Flickr8K datasets, and a discussion on how various multimodal inputs influence performance. Given a general appreciation of the work by the reviewers, I believe that the paper will be of interest to the audience attending ACM MM and would recommend a presentation of the work as a poster.

Reviews: review: paper_summary: paper_summary: The paper introduces CIEASR, a multimodal speech recognition model that utilizes scene images as soft prompts to correct homophone errors in Automatic Speech Recognition (ASR). It refines the VSDial dataset and demonstrates significant improvements in Character Error Rate (CER) on both VSDial and Flickr8K datasets.
strength_weakness: strength_weakness: 1. The approach of using scene images as soft prompts is innovative and addresses the challenge of homophone discrimination effectively.
2. The paper is well-written, with clear explanations of the model architecture and the fusion strategy for integrating visual cues.
limitations: limitations: 1. The paper could explore the model’s performance across a wider range of languages and dialects to ensure broader applicability.
2. While the paper claims state-of-the-art results, a more detailed comparison with existing methods, including quantitative metrics, would strengthen the evaluation.
paper_summary: paper_summary: This paper introduces a novel multimodal speech recognition model CIEASR to correct homophone errors by incorporating scene images as soft prompts.
strength_weakness: strength_weakness: This paper is easily followed and provides a clear description of the methods. 
A novel multimodal ASR model, called CIEASR, is introduced to solve the challenge of homophone discrimination in the ASR systems by incorporating contextual visual comprehension. 
An expanded dataset VSDial-caption is conducted to support the experiments.
limitations: limitations: 1. While the authors have introduced a new framework aimed at addressing the challenge of homophone recognition, it appears that all components of the framework are constructed based on existing technologies. There is a notable absence of technical innovation within the proposed system.
2. It is suggested to provide a detailed description of the fusion process between visual cues and speech signals. A comprehensive elucidation of this process is essential to enhance the perceived technical innovation of the work. Additionally, the term 'learned queries' mentioned in Figure 3 requires clarification. 
3. Although the paper reports experiments conducted across three datasets, the comparative methods are limited. To strengthen the validity of the findings and to provide a more comprehensive evaluation of the proposed approach, it is recommended that the authors incorporate a broader range of comparative methods. 
4. The manuscript currently lacks a dedicated section detailing the comparative methods, which is crucial for ensuring the fairness and transparency of the experimental evaluation. It is recommended that the authors include a section that thoroughly describes the comparative methods and the experimental setup.
5. Furthermore, the integration of textual information within the framework is not clearly explained. It would be beneficial for the authors to elucidate whether the incorporation of textual data is facilitated through a pre-trained model or another mechanism.
6. Table 1, according to the reviewers' understanding, the textual information is more direct than visual cues, implying that using text as a cue should also yield significant performance improvements. Furthermore, the combination of text and image encompasses richer information, therefore, it should obtain better results compared to the image cue. To ensure the validity of the experiments, it is imperative that the authors provide a detailed account of the fusion techniques and methodologies for the text and image+text cues.
paper_summary: paper_summary: The context images are applied to enhance the performance of the ASR task.
strength_weakness: strength_weakness: The scene image cues serve as soft prompts and integrates into pre-trained Whisper model, obtaining significant improvements in the recognition of entity nouns and pronouns.
limitations: limitations: In general, the proposed model architecture only integrates existing well-designed neural networks, please clarify the rationale of the model design related to the task in this work.
As the key in this work, the feature fusion (prompt process) across different modalities is required to be further clarified.
The comparative methods in this work are limited, please consider more recent works as baselines to validate the proposed approach.
The implementation details of the proposed approach, as well as the selective baselines, are also required to provide to improve the reproducibility.
In addition, more analysis is required to understand the work process of the proposed model.
paper_summary: paper_summary: This paper presents an advanced approach to improving speech recognition by integrating audio, visual, and semantic features to address homophone errors. The authors transform speech into a Mel spectrogram and extract acoustic features using the Whisper model's encoder. These features are enhanced through cross attention, blending auditory data with semantic information from scene images processed by the Q-Former model. The adapted Whisper decoder utilizes these image-derived soft prompts alongside text embeddings to generate accurate transcriptions. Evaluations on expanded VSDial and Flickr8K datasets show significant performance improvements.
strength_weakness: strength_weakness: 1. **Clarity and Detail**:
   - Clear explanations of the methodology, including preprocessing steps, integration of multimodal data, and detailed experimental setups.

2. **Problem Statement and Motivation**:
   - Addresses the challenge of homophone errors in speech recognition, providing a strong motivation for integrating visual and semantic information.

3. **Innovative Contributions**:
   - Novel use of Q-Former for extracting semantic information from images and integrating it as soft prompts in the Whisper model.

4. **Evaluation**:
   - Comprehensive evaluation on both synthetic (VSDial) and real-world (Flickr8K) datasets, demonstrating significant improvements in CER and WER.

5. **Relevance**:
   - Practical implications for real-world speech recognition, particularly in multilingual and visually challenging environments.
limitations: limitations: 1. **Computational Complexity**:
   - The integration and processing of multimodal data require significant computational resources, which may limit scalability and real-time application.

2. **Limited Generalization Discussion**:
   - The paper lacks a thorough discussion on how well the proposed method generalizes to diverse real-world conditions, such as various languages, accents, and spontaneous speech.

3. **Scope of Ablation Study**:
   - The ablation study focuses on visual prompts but does not explore the impact of different types of visual and semantic information, limiting the understanding of various influencing factors.