Intuitive Multilingual Audio-Visual Speech Recognition with a Single-Trained Model
Authors: Joanna Hong
Se Jin Park
Yong Man Ro
Venue: EMNLP 2023 Findings (2023)
Keywords: audio-visual speech recognition
speech recognition
multimodal
multilingual
TL;DR: Design a multilingual audio-visual speech recognition with a single trained model

Abstract: We present a novel approach to multilingual audio-visual speech recognition tasks by introducing a single model on a multilingual dataset. Motivated by a human cognitive system where humans can intuitively distinguish different languages without any conscious effort or guidance, we propose a model that can capture which language is given as an input speech by distinguishing the inherent similarities and differences between languages. To do so, we design a prompt fine-tuning technique into the largely pre-trained audio-visual representation model so that the network can recognize the language class as well as the speech with the corresponding language. Our work contributes to developing robust and efficient multilingual audio-visual speech recognition systems, reducing the need for language-specific models.

Intuitive Multilingual Audio-Visual Speech Recognition with Single-Trained Model
Joanna Hong, Se Jin Park, Yong Man Ro School of Electrical Engineering, KAIST 2, ymro) @ kaiist.cckkr
Abstract
2023) has presented cross-lingual visual speech rep- resentation learning and shows multilingual models with more data outperform monolingual ones. The other works, MuAViC (Anwar et al., 2023) and MixSpeech (Cheng et al., 2023), have newly intro- duced multilingual audio-visual corpus for speech recognition and speech-to-text translation task.
We present novel approach to multilingual audio-visual speech recognition tasks by intro- ducing single model on multilingual dataset. Motivated by human cognitive system where humans can intuitively distinguish different lan- guages without any conscious effort or guid- ance, we propose model that can capture which language is given as an input speech by distinguishing the inherent similarities and differences between languages. To do so, we design prompt fine-tuning technique into largely pre-trained audio-visual representation model so that the network can recognize the language class as well as the speech with the corresponding language. Our work contributes to developing robust and efficient multilingual audio-visual speech recognition systems, reduc- ing the need for language-specins models.
While the recent multilingual speech recogni- tion studies have shown remarkable advances, they have only focused on pre-training the multilingual speech recognition model or audio-visual speech representation model, followed by fine-tuning with the specific language (Zinonos et al., 2023). This is due to the imbalance of dataset language distribu- tion and each language's distinctive characteristic. However, producing a language-specift speech recognition model can be time-consuming and in- efficient. Most importantly, it does not correspond to a real-life situation, where humans intuitively recognize the language when others are speaking.
1 Introduction
With the great advancements of deep learning, au- tomatic audio-visus speech recognition (AVSR) technology has achieved remarkable progress (Mroueh et al., 2015; Afouras et al., 201 8a; Baevski et al., 2020; Kim et al., 2022; Ma et al., 2021; Hong et al., 2023). It utilizes multimodal inputs, including both audio and visual cues, providing sev- eral advantages to the deep learning- based speech recognition branch. One of the benefits is that can accurately recognize speech in noisy environ- ments, such as crowded restaurants or corrupted video conferencing situations. This capability is critical for advancing the field of automatic speech recognition technology in the future.
Inspired by the human understanding perspec- tive, in this paper, we design a single model multi- lingual audio-visual speech recognition framework that the model can not only determine which lan- guage is taken into the input speech but also recog- nize the speech correctly. To do so, we newly intro- duce an audio-visual guiding linguistic representa- tion extractor. With the largely trained audio-visual speech representation model (Shi et al., 2022), we fine-tune the model by utilizing prompts so that the model can extract comprehensive linguistic infor- mation from the audio and video inputs. We set only a small amount of downstream task-specific parameters as the learnable parameters for extract- ing linguistic representation into the input space so that comprehensive linguistic information be produced. Furthermore, we consider the im- balanced distribution issue that the multilingual datasets contain, with some languages having sig- nificantly fewer samples than others. Inspired by (Li et al., 2022), we suggest a weighted objective
Nevertheless, outstanding performances in audio-visual speech recognition have been mostly shown in monolingual datasets, particularly in En- glish. Few recent studies have started focusing on multilingual speech recognition tasks, but they are still in their infancy. One work (Zinonos et al.,
*Corresponding Author.

---

"prediction"
Visual Front
Multilingual Transformer Decodertp
Audio-Visual Transformer Encoder
Fa
Multilingual Classifier
class
Audio Audio-Visual Prompts
Figure 1: Overall architecture of the proposed multilingual audio-visual speech recognition model.
function in order to balance each language distribu- tion. By training model diverse set of lan- guages, aim to capture their inherent similarities and differences, allowing our model to recognize and transcribe speech multiple languages with greater accuracy and efficiency. We validate ef- fectiveness of our proposed model using MuAViC (Anwar et al.. 2023), a multilingual audio-visual corpus for robust speech recognition and robust specch-to-text translation providing 1200 hours of audio-visual speech in 9 languages. Therefore, work contributes to developing efficient and robust multilingual audio-visual speech recognition sys- tems. We also believe that our proposed approach has several potential benefits: reducing the need for language-specific models, improving the per- formance of speech recognition in low-resource languages, and enabling more effective multilin- gual communication.
visual transformer encoders. We will explain the detailed aforementioned techniques in the follow- ing subsections.
2.1 Linguistic Representation Extractor
The first thing when recognizing one's speech in multinational society is to distinguish the language the speaker is presenting. When identifying the language of the speech, it important verify the speaker's accent, pronunciation, and representative vocabulary from the speaker's facial movements and audio information. Inspired by the intuitive human understanding procedure, we design a lin- guistic representation extractor from the largely pre- trained audio-visual speech representation model. We add trainable continuous embeddings, so-called prompts, the original sequence of input features in order to fine-tune the pre-trained model relevant to recognizing multilingual input signals.
2 Methodology
2.1.1
Prompt fine-tuning for Linguistic Representation Extractor
Given the input multilingual video sequence, F. .Iz E L where L, H, W. and C are the frame length, height, width, and channel sizes, respectively, and the paired input audio sequence, Ta SRS where S represents the length of audio, we design a model that properly recognizes the given input video and audio with the correct language. We aim to utilize both visual and audio information so that the proposed architecture can successfully recognize not only the language but also the content of the input video. To this end, we initially propose language prompt adopted from the largely pre-trained audio-visual speech representation model (Shi et al., 2022), which call it linguistic representation extractor. Further, we design multilingual transformer decoder given the inputs language class, linguistic representa- tion, and the combined features from the audio and
For the input multilingual video sequence æ. and audio sequence Ca, audio features fa C T and visual features fo E Txxx,, are extracted from through Audio Front and Visual Front, respectively.
f. Fo(x)) and fa = Fa(aa.
(1)
The audio features fa and the visual features fo are concatenated followed by layer normalization and linear projection, producing the audio-visual fea- tures far Y') Then, we apply Audio-Visual prompts into every layer of the Audio- Visual Trans- former Encoder V, along with the audio-visual features fav- For each audio-visual prompt, we sign language-representatie prompts, P E RRuxd which are trained to extract linguistic-relativ fea- tures from the pre-trained audio- representa-

---

tion model. Here, n is the number of prompts.
N represents the total length of ground-truth text. Then, the total objective can be written as follows,
(Favs)) Vi(Pav.-11.fm 1) (Par,trfanLL) v (Pou,L 1
(2)
Lete log Pa (1(l)))
(8)
(3)
Lott log (o(p)2))))
(9)
for the i-th layer Vi where 1,2...., L. We now call eav = (Pav,I-fav,L) . E ( T)xd audio- visual prompt embedding feature.
Lrotal a Lete + a) Latt + 3 Celases
(10)
where a and 3 are weight parameters for balancing three loss terms.
2.1.2 Multilingual Classifier for Linguistic Class Prompt
2.2.1 Objective Funetions for Balancing the Language Distribution
After extracting the audio-visual prompt embed- ding feature Cas, we firstly train the prompt tuning module by updating the gradient of the learnable parameters of the prompts with the classification loss. As indicated Figure 1, we propose a mul- tilingual classifier in order to distinguish the input language class. The multilingual classifier C' con- sists of four blocks of 1D convolution layer, batch normalization, and Relu activation, followed by two linear layers with Relu activation.
An objective function weight is designed to balance the distribution of language data, due to the issue of language imbalance in the multilingual dataset. This weight is calculated as the inverse root of the data distribution ratio r for each language in each mini-batch:
(11)
logitpred = C(eav).
Therefore, the updated total objective function can (4)
be re-written as follows,
The output language class is logitgred E IR.
Liota! = (ax Late + (1 - a) Lose + 3 Letass).
where represents the number of languages for training. Then, the logit is updated through cross- entropy objective function:
(12)
The rationale behind this design comes from the ob- servation that the multilingual dataset often exhibits an uneven distribution of samples across different languages. Thus, when updating the loss during training, becomes crucial to employ balancing loss function with smaller weight for languages that contain a larger number of samples and a larger weight for languages that have fewer samples.
Leaass = plogiittr gt)
(5)
Therefore, by updating the correct language label, the model can be provided with guidance when recognizing the multilingual speech correctly in the backbone network that will be expressed in further sections.
By incorporating this weight into the objective function, the model is encouraged to assign greater importance to underrepresented languages during the learning process. This approach aims to miti- gate the adverse effects of language imbalance and prevent the model from being biased toward dom- inant languages. Thus, the model becomes more capable of effectively recognizing and transcribing speech in languages with limited available data.
2.2 Objective Functions
The proposed multilingual AVSR framework is trained in an end-to-end manner. For the objec- tive function, we utilize joint CTC/attention (Kim et al., 2017). CTC (Graves et al., 2006) loss is defined as:
(6)
3 Experimental Setup
with an independent assumption each output, and attention- based loss is defined as:
3.1
Datasets
The MuAViC dataset (Anwar et al., 2023) is mul- tilingual audio-visual corpus consisting of roughly 1,200 hours of transcribed data spanning 9 lan- guages: English, Arabic, German, Greek, Spanish, French, Italian, Portuguese and Russian. It is col- lected from TED and TEDx talk recordings, where
Pa(y x)) 11
(7)
Here, the current prediction determined by previous predictions and inputs, thus including the learning of the internal language model, where

---

Type
Model
De
En
Es
Fr
1t
Pt
Ru
Avg
Monolingual [R6]
99.24
53.93
25.85
2.21
16.66
26.26
20.07
19.97
32.77
33.00
Multilingual [R6]
90.49
56.01
37.96
18.99
22.97
21.82
22.34
45.50
39.51
Clean
Proposed Model (Latt)
91.48
52.60
45.84
3.39
20.28
23.76
21.45
23.58
57.36
37.75
Proposed Model (Cau + Cete)
90,94
48.10
42,41
2.47
18.04
21.52
19.67
20.80
54.12
35.34
Monolingual [R6]
100.19
70.49
50.81
6.50
40.72
44.87
47.90
42.30
49.48
50.36
Multilingual [R6]
98.65
74.41
62.98
42.11
41.86
47.22
44.86
65.93
59.75
Noisy
Proposed Model (Catt)
100.017
70.56
66.43
10.68
43.48
41.87
47.98
46.73
74.00
55.75
Proposed Model (Cate + Lete)
99.66
65.92
61.94
9.01
38.94
37.37
42.48
42.13
68.99
51.83
Table 1: WER (%) comparisons with the previous multilingual audio-visune speech recognition methods on clean and noisy settings.
native or non-native speakers (only one speaker most of the time) deliver public speech on stage and cameras capture stage scenes switching among different viewpoints.
ative) has been made in one of the low-resourced language, German (DE). Such results demonstrate that the proposed method effectively enables mul- tilingual audio-visual speech recognition umi- fied framework, improving the performance in low- resource languages.
3.2 Implementation Details
We use largely pre-trained visual frontend, audio frontend, and audio-visual transformer encoder (Shi et al., 2022), trained on LRS3-TED (Afouras et al., 201 and VoxCeleb2 English (Chung et al., 2018). We fine-tune the encoding models guided by the prompts and the multilingual classifier in an end-to-end manner, such that the prompts learn the meaningful content, i.e., the language class of the input speech and how the speech delivered, for making the correct prediction the multilingual scheme. We set a 0.1 and 8 = 10.0. For train- ing and testing, we follow the same noise injection protocol as (Anwar et al., 2023).
Noisy Environment. In the second section of Ta- ble 1, we evaluate the proposed method in noisy setup, The proposed model achieves average WER of 47.18, excluding English while the previous mul- tilingual model achieves 59.75 WER, which is a 4.31 % relative improvement. The performance gap between the previous monolingual model (Anwar et al., 2023) is even smaller with average of 2.4% relative WER. We contribute such improve- ment the allows the model to embed eagguag--specfffi fea- tures from the audio-visual input, simulating the monolingual framework.
prompt which
4 Experimental Result
5 Conclusion
the experimental result in Table 1, we report the performances of our proposed model with attention loss only and both attention and CTC loss along with the previous multilingual model (Anwar et al., 2023) performance. The monolingual model refers the model that separately trained each lan- guage and the multilingual refers the model that is jointly trained on all the 8 non- hon-English languages. The previous model did not include En in their mul- tilingual model, so remains blank. Note that we test the provided trained model for reporting previous work performance.
We introduce the multilingual audio-visual speech recognition model by training a single model on diverse set of languages. The proposed model finetunes largely trained audio-visual represen- tation model with prompts provide meaningful language information. It has presented a promising starting point for future research endeavors.
A Acknowledgement This work was partially sup- ported by the National Research Foundation of Ko- rea (NRF) grant funded by the Korea government (MSIT) (No.NRF-2022R1 A2C2005529), and Insti- tute of Information & communications Technology Planning & Evaluation (IIPP) grant funded by the Korea (MVI) (No.2022-0-00124, De- velopment of Artificial Intelligence Technology for Self- gellovingoi Competency- Aware Learning Ca- pabilities).
Clean Environment. We evaluate audio-visual speech recognition in a clean environment. As shown in the first section of Table 1, our proposed model outperforms the previous model (Anwar et al., 2023) in several languages, where the great- est improvement (7.91 WER reduction, 14% rel-

---

References
Pingchuan Ma, Stavros Petridis, and Maja Pantic. 2021. End-to-end audio-visual speech recognition with con- formers. In ICASSP 2021-2021 IEFE International Conference Acoustics, Speech and Signal Process- ing (ICASSP), pages 7613-7617. IEEE.
Triantafyllos Afouras, Joon Son Chung, Andrew Senior, Oriol Vinyals, and Andrew Zisserman. 2018a. Deep audio-visual speech recognition. IEEE transactions on pattern analysis and machine intelligence.
Youssef Mroueh, Etienne Marcheret, and Vaibhava Goel. 2015. Deep multimodal learning for audio-visual speech recognition. In 2015 IEEE International Con- ference on Acoustics, Speech and Signal Processing (ICASSP), pages 2130-2134, IEEE.
Triantafyllos Afouras, Joon Son Chung, and An- drew Zisserman. 2018b. Lrs3-ted: a large-scale
dataset for visual speech recognition. arXiv preprint
Mohamed Anwar, Bowen Shi, Vedanuj Goswami, Wei- Ning Hsu, Juan Pino. and Changhan Wang. 2023. Muavic: A multilingual audio-visual corpus for ro- bust speech recognition and robust speech-to-text translation. arXiv preprini arXiv:2303.00068..
Bowen Shi, Wei-Ning Hsu, Kushal Lakhotia, and Ab- delrahman Mohamed. 2022. Learning audio-visua speech representation by masked multimodal cluster prediction. arXiv preprint axXiv:2201.02 184..
Andreas Zinonos, Alexandros Haliassos, Pingchuan Ma. Stavros Petridis, and Maja Pantic. 2023. Learning cross-lingual visual speech representations. arXiv preprint 203.05233..0945..
Alexei Baevski, Yuhao Zhou, Abdelrahman Mohamed, and Michael Auli. 2020. wav2vec 2.0: A framework for supervude learning of speech representations. Advances in Neural Information Processing Systems, 33:12440 13:12449-12444
Xize Cheng. Linjun Li, Tao Jin, Rongjee Huang, Wang Lin, Zehan Wang, Huangdai L.u,, Ye Wang, Aox- iong Yin. and Zhou Zhao. 2023. Mixspeech: Cross- modality self-learning with audio-visual stream mixup for visual speech translation and recognition. arXiv preprint ar305:2303.05309
Joon Son Chung, Arsha Nagrani, Andrew Zisser- man, 2018. Voxceleh2: Deep speaker recognition.
arXiv preprint arXiv: 1806 05622.
Alex Graves, Santiago Fernández, Faustino Gomez, and Jürgen Schmidhuber. 2006. Connectionist temporal classification: labelling unsegmented sequence data with recurrent neural networks. In Proceedings of the 23rd international conference on Machine learning, pages 369-376.
Joanna Hong, Minsu Kim, Jeongsoo Choi, and Yong Man Ro. 2023. Watch or listen: Robust audio-visual speech recognition with visual coruu-- tion modeling and reliability scoring. arXiv preprint arX.v:2303.08336.
Minsu Kim, Jeong Hun Yeo, and Yong Man Ro. 2022. Distinguishing homophenes using multi-head visual- audio memory for lip reading. In Proceedings of the 36th AAAI Conference on Artificial Intelligence. Vancouver, BC, Canada, volume 22.
Suyoun Kim, Takaaki Hori, and Shinji Watanabe. 2017. Joint etc-attention based end-to-end speech recogni- tion using multi-task learning. 2017 IEEE inter- national conference on acoustics. speech and signal processing (ICASSP), pages 4839, IEEE.
Hao Li, Jinguo Zhu, Xiaohu Jiang, Xizhou Zhu, Hong- sheng Li. Chun Yuan. Xiaohua Wang, Yu Qiao. Xiaogang Wang, Wenhai Wang, et al. 2022. Uni- perceiver v2: A generalist model for large-scale vision and vision-language tasks. arXiv preprint arXiv:2211.09808


Reviews:
[{"review_id": "mN83MWpa1z", "reviewer": "Reviewer_djfB", "summary": "", "questions": "", "limitations": "", "rating": 2, "confidence": 2, "soundness": 2, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the appreciation of our paper and contribution. We hope this rebuttal helps to elucidate any confusing subjects such that reviewers would be open to improve their ratings. We will incorporate all feedback in our final manuscript.\n\n**(1) Performance comparison**\n\nWe would like to thank the reviewer’s valuable comments. We would like to point out that regarding the noisy environment, the proposed model achieved an average of 57.18 % WER excluding English, which shows a promising performance gain. This shows that the proposed model is also robust to the noisy environment, although it shows a small improvement in the clean environment. Regarding the reviewer’s concern, we would rewrite the claim of saying performance improvement and follow the fair comparison in the future manuscript to avoid the confusion.\n\n**(2) Comparison to multi-task baseline**\n\nWe appreciate the reviewer’s suggestion. We would like to kindly clarify that we design a singly modeled multilingual audio-visual speech recognition technique with the use of a language classification module and the novel prompt finetuning technique. To the best of our knowledge, prior approaches in multilingual speech recognition related to multi-task learning have typically involved incorporating the language class as an input parameter, so they have conducted slightly different approach direction from the proposed model. We will add performance results from the additional previous technique that relates both the multi-task learning and the proposed architecture for the future manuscript.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The paper describes experiments in the area of multi-lingual multimodal speech recognition and a publishes a dataset that can be used by others for that purpose.", "reasons_to_reject": "Generally, I don't agree with the conclusions that the authors derive from their experimental results. For instance, the authors claim an average 11% WER reduction due to the proposed method compared to their previous model. However, this compares results on two different test sets - the proposed model has been evaluated on English data, too, while the previous model has no result for English test data. If you exclude the English test data from the comparison (apple to apple) the previous model achieves an Avg WER of 39.51% on clean data and the proposed model achieves an Avg. WER of 39.45% which is not really an improvement. Furthermore, I believe the proposed approach misses a comparison to a multi-task training baseline which includes a language classification loss.", "questions_to_authors": "", "paper_topic_and_main_contributions": "The paper deals with multilingual audio-visual speech recognition. The authors describe a method that extracts a set of audio-visual features and augments them with  embeddings which have been trained to optimize a language classification loss. Experiments are evaluated on a dataset of multilingual video recordings of TED speeches.\n", "excitement": 2, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "kH1PQ9xgo2", "reviewer": "Reviewer_EVgV", "summary": "", "questions": "Please identify which results from Anwar et al. are cited in the paper.", "limitations": "", "rating": 2, "confidence": 4, "soundness": 2, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the appreciation of our paper and contribution. We hope this rebuttal helps to elucidate any confusing subjects such that reviewers would be open to improve their ratings. All feedback will be incorporated into our final manuscript. \n\n**(1) Fairness for performance comparison**\n\nWe would like to thank the reviewer for the valuable concern. We would like to clarify that for the fair comparison, we have evaluated the recognition performances of all the languages using the pre-trained model provided by Anwar et al. We will mention about the re-evaluation of the previous method from the pre-trained model for the fair comparison for the future manuscript. Furthermore, to avoid the confusion and for a fair comparison, we would rewrite the performance comparison part of the manuscript and rearrange Table 1.\n\n**(2) Understanding of prompt**\n\nWe design the novel language-representative prompt on top of the largely pre-trained audio-visual representation model in order to train the proposed architecture to not only determine the language of the input speech but also correctly predict what the input speech is delivering. The prompts are always trained in an end-to-end way along with the entire audio-visual multilingual speech recognition network architecture. Furthermore, we would like to clarify that the previous work (Anwar et al.,2023) has trained on the largely pre-trained audio-visual representation model (Shi et al., 2022), so we consider this as a baseline architecture that the reviewer has pointed out of excluding the language-representative prompt. \n\n**(3) Clarity of the manuscript**\n\nWe would greatly appreciate that the reviewer points out the parts that are less clear to understand. To avoid confusion, we will reflect all the confusion in the future manuscript.\n\n- __*line 013-014: “ … both label and nuance”*__ &nbsp; Through the proposed prompt-finetuning technique using the largely pre-trained model, the single proposed model can determine which language the input speech is delivering (label), and it can also predict how the speech is delivering (nuance) for the correct prediction.\n\n- __*line 014-016: “ … predict correct speech with correct label”*__ &nbsp; We would like to clarify that the proposed network architecture is trained to predict a sentence (a set of words) from the input speech. Hence, from the high-level point of view, we mention that the proposed model predicts the correct speech.  \n\n- __*line 054-057: “ … fine-tuning followed by pre-training”*__ &nbsp; We would like to thank the reviewer for the comment. We would make the clarification to “pre-training the multilingual speech recognition model or audio-visual speech representation model followed by fine-tuning the specific language”.\n\n- __*line 291: Eq (12)*__ &nbsp; We would like to clarify that the $\\gamma$ is the distribution ratio for each language, so the $\\gamma$ value changes every iteration regarding the distribution of the language in each mini-batch per iteration.\n\n- __*Caption of Table 1*__ &nbsp; We apologize for the misleading information in the caption of Table 1. We will change the caption of Table 1 to maintain the consistency in the future manuscript.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "* Formulation and empirical study of a single simple multi-lingual AVASR model\n", "reasons_to_reject": "* In the results presented in Table 1, the baseline numbers from Anwar et al don’t match those reported in Table 1.  Not sure which paper to look at for source of the reported Anwar et al. numbers.  Also, for a fair comparison of numbers in the Avg. column, especially to compare w/ Multilingual performance from Anwar et al, En results should not be included in Table 1.\n\n* The model includes a prompt, but the value of this component is not clear or empirically determined.  Are the prompts sometimes there and other times not?  What is the performance if the prompts are not included in the model training/test?\n\n* Paper lacks clarity / errors in places.  E.g.\n     + 013-014: “ … both label and nuance”. Not clear what this means\n     + 014-016: “ … predict correct speech with correct label” … is it predicting speech, or words?\n     + 054-057: “ … fine-tuning followed by pre-training” or the other way around?\n     + 219: Does Eq (12) need a sum over all samples?  Otherwise \\gamma has no impact on the optimization.\n     + Table 1 caption mentions boldfaced or underlined letters, but there are no such letters/numbers in the table.", "questions_to_authors": "Please identify which results from Anwar et al. are cited in the paper.", "paper_topic_and_main_contributions": "This paper proposes a single multilingual AVASR model for handling audio+visual input in multiple languages.  The model hypothesizes detected language in addition to transcript for spoken input.  It utilizes an encoder decoder architecture where the encoder is a pre-trained AV transformer model, and decoder is a multi-lingual transformer architecture.  A classifier is used for predicting the language of input speech.  Proposed model is trained/evaluated on the recently released MuAViC dataset.", "excitement": 2, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "E7iy0aj33A", "reviewer": "Reviewer_oy3M", "summary": "", "questions": "A. It is not clear in the methodology section how the trainable continuous embeddings (prompts) are generated and optimized during the fine-tuning process. Can such prompts be learned automatically, or do they require manual annotation? Please provide more details on this process to help readers better understand how these prompts contribute to the overall model.\n\nB. In the experiments, the proposed model is compared primarily with the work of Anwar et al. (2023). It would be beneficial to include comparisons with other relevant methods and show how the proposed approach performs in comparison to a broader range of existing methods in the field of multilingual AVSR. Are there other methods that could be included for comparison?\n\nC. The paper focuses on a dataset containing nine languages which may be considered heavily biased towards Indo-European languages or the languages with a relatively large amount of available data. How would the proposed method perform for languages with few data samples or for languages from very different linguistic families (e.g., tonal languages, agglutinative languages)? Are there plans to extend or validate the model on more diverse sets of languages in the future?\n\nD. The model has been tested in both clean and noisy environments. Although it shows improvement over the previous multilingual model, the performance seems to be lower in noisy environments. What are the possible reasons for the reduced performance, and could this be improved upon with further modifications to the model architecture or training strategy?", "limitations": "", "rating": 4, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We would like to express our gratitude for your valuable feedback. We hope this rebuttal helps to elucidate any confusing subjects such that reviewers would be open to improve their ratings. We will incorporate all feedback in our final manuscript.\n\n**(1) The lack of an in-depth ablation study**\n\nWe would like to thank the reviewer for delivering the concerns about the manuscript. We totally understand the concerns that the reviewer has raised. We would like to carefully mention that through different settings for prompt fine-tuning, we come up with the most comparable setting to the previous model. We will conduct more analysis on model performances using various choices of hyperparameters in the future manuscript.\n\n**(2) Unknown language setting**\n\nWe would greatly appreciate the reviewer’s suggestion. We would like to clarify that we mainly contribute to design the singly modeled multilingual audio-visual speech recognition technique with the use of a language classification module and the novel prompt finetuning technique for the first time. In regarding the aspect of real-world scenarios, we totally agree that it is crucial to have the model recognize speech from an unknown language, and this could be the next step to focus on researching the robust multilingual speech recognition model, even in unknown languages.\n\n**(3) Distribution of imbalanced dataset**\n\nTo address the concern of the reviewer, we kindly report the duration statistics of muavic dataset for audio-visual speech recognition in hours of the MuAViC dataset. We refer to the original manuscript of the MuAViC (Anwar et al.,2023).  We will add the additional information about the imbalanced distribution of the multilingual dataset in the future manuscript.\n\n|    | Train | Dev | Test |\n|----|-------|-----|------|\n| **En** |  436.0 | 1.1 | 0.8  |\n| **Ar** | 16.0  | 1.5 | 1.2 |\n| **De** | 10.0  | 1.6 | 1.5 |\n| **El** |  25.0 | 2.3 | 2.0 |\n| **Es** | 178.0 | 1.6 | 1.7  |\n| **Fr** |  176.0 | 1.8 | 1.5  |\n| **It** |101.0  | 1.9 | 1.9|\n| **Pt** |  153.0  | 1.5 | 1.8  |\n| **Ru** |  49.0 | 1.7 | 1.8|\n\n**(4) Investigating on other recent advancements**\n\nWe would appreciate the valuable suggestion. We clearly understand that comparing with the recent advancements like self-supervised techniques or architectures like transformers in multimodal learning is beneficial for the model analysis. We would like to clarify that the previous architecture and the largely pre-trained audio-visual representation model (Shi et al., 2022) deal with the self-supervised learning technique and utilize transformer.\n\n**(5) Experiments on other datasets**\n\nWe thank the reviewer’s concern focusing on the MuAViC dataset. We would like to kindly address that we primarily focus on the dataset because it is the largest open benchmark for multilingual AVSR [1]. The dataset consists of 1200 hours of transcribed AV speech in nine different languages from 8000+ speakers in TED and TEDx talks [2] which has been widely used for multilingual speech recognition. As the reviewer noted, incorporating a more diverse range of data would further enhance the proposed model’s robustness and capacity for generalization.  We will utilize different multilingual datasets for the generalization of the proposed method in the future manuscript. \n\n$\\newline$\n\n**Additional Questions**\n\n**A.** __*(Prompt learning)*__ &nbsp; The proposed language-representative prompts do not need any annotation during training. With the end-to-end training of the entire proposed architecture, the prompts are expected to learn the meaningful contents (i.e., which language the input speech contains and how the speech is delivering) for the correct prediction. We will clearly state the methodology section in the future manuscript to avoid any confusion.\n\n**B.** __*(Comparison to other methods)*__ &nbsp; We would like to thank the reviewer for the valuable suggestion. We make a comparison with the previous model that has conducted experiments on multilingual audio-visual speech recognition with a single model without any extra language class information.  There are a few multilingual speech recognition tasks like [3], but it has been conducted with a single input modality. We will add an additional performance comparison with the same setting that we have conducted in the experiment in the future manuscript. \n\n**C.** __*(Experiments on other languages)*__ &nbsp; We would like to kindly point out that to the best of our knowledge, it is the first time to focus on designing a single model for the multilingual audio-visual speech recognition task, so we start from designing the model with the most general languages. Our model has been able to improve the performance in low-resource languages in the training dataset and reduce the need for language-specific models, enabling more effective multilingual communication. Exploring robustness to languages from very different linguistic families would be the subsequent research direction to extend the generalization capacity of the proposed model. We have plans to build upon variances of linguistic families by incorporating AVSpeech [4] dataset which contains 4700 hours of audio-visual paired clips in 95 different languages. We believe that our work has potential benefits in reducing the need for language-specific models, improving the performance of speech recognition in low-resource languages, and enabling more effective multilingual communication.\n\n**D.** __*(Performances in noisy environments)*__ &nbsp; It has always been challenging to recognize the speech from the noisy input, as the baseline architecture (Anwar et al.,2023) has shown degraded performances in the noisy environment. The reduced performance is generally caused by the distraction of noises inserted from the clean audio waveform so that the network can hardly distinguish between the clean audio waveform and the noise. We can firstly enhance the noisy audio features by adapting the feature enhancement module like [5] to create the enhanced audio features and take them into the recognition model. \n\n$\\newline$\n\n[1] Ivanko, Denis, Dmitry Ryumin, and Alexey Karpov. \"A Review of Recent Advances on Deep Learning Methods for Audio-Visual Speech Recognition.\" Mathematics 11.12 (2023): 2665.\n\n[2] Salesky, Elizabeth, et al. \"The multilingual tedx corpus for speech recognition and translation.\" arXiv preprint arXiv:2102.01757 (2021).\n\n[3] Zinonos, Andreas, et al. \"Learning Cross-Lingual Visual Speech Representations.\" ICASSP 2023-2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2023.\n\n[4] Ephrat, Ariel, et al. \"Looking to listen at the cocktail party: A speaker-independent audio-visual model for speech separation.\" arXiv preprint arXiv:1804.03619 (2018).\n\n[5] Hong, Joanna, et al. \"Visual context-driven audio feature enhancement for robust end-to-end audio-visual speech recognition.\" arXiv preprint arXiv:2207.06020 (2022).", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. Novelty: The approach of fine-tuning a largely pre-trained audio-visual representation model with language prompts is new and innovative, enabling the model to effectively recognize and transcribe speech in multiple languages.\n\n2. Robustness: The proposed model demonstrates performance improvements in low-resource languages and exhibits strong results in both clean and noisy environments. This has significant practical implications in real-world applications where speech recognition systems often face challenges in noisy conditions.\n\n3. Addressing Language Imbalance: The paper proposes an objective function weight to balance the distribution of language data, mitigating potential biases towards dominant languages while improving the recognition and transcription of low-resource languages.\n\n4. Comprehensive Experiments: The experiments are conducted on a recent and challenging dataset (MuAViC), consisting of 1200 hours of audio-visual speech data in 9 languages, providing a comprehensive evaluation of the model.\n\n5. Potential Impact: The proposed single model approach can reduce the need for language-specific models and improve the performance of speech recognition in low-resource languages. This would benefit the NLP community in developing more efficient and robust multilingual speech recognition systems.", "reasons_to_reject": "1. One weakness is the lack of an in-depth ablation study on the proposed approach. Some questions remain unanswered, such as the effectiveness of prompt fine-tuning in various settings and the sensitivity of the model to the choice of hyperparameters.\n2. The paper does not provide results for a setting where the language is unknown during inference. Dealing with an unknown language is a crucial aspect in real-world scenarios.\n3. The authors mention the imbalance in the dataset, but the exact distribution is not provided. It is important to understand how each language is uniformly represented in experiments and the impact on the overall performance.\n4. Although the proposed approach performs better than existing models, it may still be interesting to investigate how this method can benefit from other recent advancements such as self-supervised learning or transformers in multimodal learning.\n5. The paper primarily focuses on the MuAViC dataset. It would be beneficial to consider other datasets to explore the generalization of the proposed method across diverse data sources.", "questions_to_authors": "A. It is not clear in the methodology section how the trainable continuous embeddings (prompts) are generated and optimized during the fine-tuning process. Can such prompts be learned automatically, or do they require manual annotation? Please provide more details on this process to help readers better understand how these prompts contribute to the overall model.\n\nB. In the experiments, the proposed model is compared primarily with the work of Anwar et al. (2023). It would be beneficial to include comparisons with other relevant methods and show how the proposed approach performs in comparison to a broader range of existing methods in the field of multilingual AVSR. Are there other methods that could be included for comparison?\n\nC. The paper focuses on a dataset containing nine languages which may be considered heavily biased towards Indo-European languages or the languages with a relatively large amount of available data. How would the proposed method perform for languages with few data samples or for languages from very different linguistic families (e.g., tonal languages, agglutinative languages)? Are there plans to extend or validate the model on more diverse sets of languages in the future?\n\nD. The model has been tested in both clean and noisy environments. Although it shows improvement over the previous multilingual model, the performance seems to be lower in noisy environments. What are the possible reasons for the reduced performance, and could this be improved upon with further modifications to the model architecture or training strategy?", "paper_topic_and_main_contributions": "This paper presents a novel approach for multilingual audio-visual speech recognition tasks by developing a single model on a multilingual dataset. The primary problem being addressed in the paper is the limited success of current audio-visual speech recognition models in multilingual datasets, as they are mostly focused on monolingual data, particularly English. The paper aims to design a single model capable of effectively distinguishing the language and properly recognizing the input speech in multiple languages.\n\nThe main contributions of the paper are as follows:\n\n1. Propose a single model framework for multilingual audio-visual speech recognition, inspired by the intuitive human language understanding process. The model is designed to determine the language of the input speech and recognize the speech correctly.\n\n2. Introduce an audio-visual guiding linguistic representation extractor by fine-tuning a largely pre-trained audio-visual representation model using prompts to provide language information. This allows the model to extract comprehensive linguistic information from the audio and video inputs.\n\n3. Address the issue of imbalanced language distribution in multilingual datasets by employing a weighted objective function that balances the distribution of language data while updating the loss during training. This aims to improve the performance of speech recognition in low-resource languages and prevent the model from being biased toward dominant languages.\n\n4. Validate the effectiveness of the proposed model by conducting experiments on MuAViC, a multilingual audio-visual corpus containing 9 languages. The results show that the proposed model outperforms the previous multilingual model with an average 11% WER reduction in a clean environment and an average 13% WER reduction in a noisy environment.", "excitement": 4, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper addresses multilingual audio-visual speech recognition and introduces a method for feature extraction and language classification loss optimization. Experiments are conducted on a multilingual dataset, MuAViC.

Pros:
- Innovative approach through fine-tuning with language prompts.
- Demonstrates robustness in low-resource languages and noisy environments.
- Addresses language imbalance with a weighted objective function.
- Comprehensive experiments on a challenging dataset.
- Potential impact on reducing language-specific models.

Cons:
- Issues with clarity, including unclear components and errors in the text.
- Inconsistent baseline numbers and unclear source for references.
- The value and impact of the prompt component are not adequately explained.
- Lack of an in-depth ablation study.
- No results for the scenario where the language is unknown during inference.
- Details about dataset language distribution are missing.
- Could explore the integration of recent advancements in multimodal learning.
- Focuses primarily on the MuAViC dataset; generalization across diverse data sources is not explored.


Overall, the paper's strengths lie in its innovative approach, robustness, and potential impact on multilingual speech recognition. However, there are areas of improvement related to clarity, experimental methodology, and further exploration of the proposed method's capabilities.
Decision: Accept-Findings