Efficient Training for Multilingual Visual Speech Recognition: Pre-training with Discretized Visual Speech Representation

Venue: acmmm_org_ACMMM_2024_Conference
Published: 2024-04-07 09:35:10

Abstract: This paper explores sentence-level multilingual Visual Speech Recognition (VSR) that can recognize different languages with a single trained model. As the massive multilingual modeling of visual data requires huge computational costs, we propose a novel training strategy, processing with visual speech units. Motivated by the recent success of the audio speech unit, we propose to use a visual speech unit that can be obtained by discretizing the visual speech features extracted from the self-supervised visual speech model. Through analysis, we verify that the visual speech units mainly contain viseme information while suppressing non-linguistic information. By using the visual speech units as the inputs of our system, we propose to pre-train a VSR model to predict corresponding text outputs on multilingual data constructed by merging several VSR databases. As both the inputs (i.e., visual speech units) and outputs (i.e., text) are discrete, we can greatly improve the training efficiency compared to the standard VSR training. Specifically, the input data size is reduced to 0.016% of the original video inputs. In order to complement the insufficient visual information in speech recognition, we apply curriculum learning where the inputs of the system begin with audio-visual speech units and gradually change to visual speech units. After pre-training, the model is finetuned on continuous features. We set new state-of-the-art multilingual VSR performances by achieving comparable performances to the previous language-specific VSR models, with a single trained model.

Main paper: 
 introduction: 
 These days, speech processing technologies have made great progress in diverse applications such as speech recognition [22, 27, 59, 65] , speech synthesis [10, 32, 50, 72, 76] , and speech translation [30, 31, 34, 42] . Now, it is easy to find a speech processing model that can proficiently handle approximately 100 languages [1, 62] . However, multilingualism has been mainly explored for audio-based speech processing systems [44, 68] , while visual-based speech processing systems are still tied to developing monolingual systems [37, 48, 75] . There are two reasons for the lagging development of visual-based speech processing systems: 1) The high dimensionality of visual data compared to audio puts a challenge in training a large-scale model with massive multilingual data. Compared to the same length of audio, visual data requires about six times larger bits [35] in a standard visual speech recognition process [47] . Moreover, the requirement of encoding spatial information using two-dimensional convolutions also increases the computation costs of visual speech processing compared to its counterpart. 2) The low quantity of labeled data in visual speech processing systems presents a formidable obstacle to technology development. In contrast to the tremendous amount of publicly available audio-text data [61] , a very limited number of video-text data are available, especially for non-English [37] .In this paper, we explore the multilingualism of visual speech processing, especially in speech recognition [5, 14, 46, 49, 54] . Hence, our objective is to devise a multilingual Visual Speech Recognition (VSR) method that can recognize different languages with a single trained model. In order to mitigate the challenges in visual speech processing, we propose a novel strategy, processing with visual speech units. The audio speech unit [41] is a discretized representation of an extracted speech feature from a self-supervised speech model [6, 28] . It contains phonemic content [66] while suppressing the other speech characteristics (e.g., speaker information) and can be employed as pseudo text. As it is the discretized signal of the original signal, the data size can be significantly reduced [8, 35] . Motivated by this, we propose to employ visual speech units, the quantized representation of the visual speech feature, in training multilingual VSR. As a result, one video frame having 61,952 bits (i.e., based on a grayscale image with 88 × 88 size) can be expressed with one visual speech unit which can be represented with just 10 bits. With the huge data size reduction, 0.016% compared to the original, we can boost the training more than 10 times faster than the standard VSR training. Through analysis, we validate that the visual speech unit contains viseme information, the visual counterpart of phoneme, while suppressing non-linguistic characteristics. Hence, enabling visual speech modeling even by using the visual speech units.Specifically, we employ AV-HuBERT [64] , a self-supervised visual speech model, to extract our visual speech units. We newly train the AV-HuBERT model on 5,512 hours of multilingual audiovisual data composed of nine languages, given that the original AV-HuBERT was trained solely on English audio-visual data. To differentiate between the English-only trained and multilingualtrained AV-HuBERT models, we denote the latter as mAV-HuBERT. With the mAV-HuBERT, we can correctly capture the multilingual viseme information into our visual speech unit. Then, we propose to pre-train an encoder-decoder model by setting its inputs with the visual speech units and the outputs with the corresponding text, forming a unit-to-unit translation framework (i.e., translation between discrete tokens) [34] . Moreover, inspired by the recent successes of VSR that leverage audio modal information to complement limited visual data [23, 36, 63, 64, 77] , we propose to use curriculum learning with a gradual increase in task difficulty using audio modality. Concretely, our unit-to-unit pre-training is initiated with audio-visual inputs and then gradually changed to visual inputs. With this curriculum learning, the model can find the optimization points stable and achieve higher performance with the complementary multi-modal information. To mitigate the comparatively small amount of public visual-text paired data, we utilize the recently proposed automatic labels of [45] and [75] where the text labels are obtained by their automatic labeling processes. Finally, the pre-trained model is finetuned with continuous input features to maximize the VSR performances.The major contributions can be summarized as follows:  
related work 2.1 visual speech recognition (vsr): 
 Visual Speech Recognition (VSR) aims to predict the spoken words from silent lip movements video. Early works [15, 53, 55, 67] focused on word-level VSR by using CNN [24] and the RNN [11, 26] . Large-scale lip-reading sentence datasets [3, 13] have boosted the development of sentence-level VSR. By employing Transformer [69] architecture, [2] proposed a powerful sentence-level end-to-end VSR model. Moreover, the integration of the hybrid CTC/Attention objective [56, 70] into VSR, greatly improved the recognition performances. Recent VSR technologies [7, 45, 47, 60] also employed transformer-variant architectures and improved the VSR performances. For advanced training strategies, many researchers try to reduce the gap between visual and audio modalities. They [4, 36, 38, 46, 63, 74, 77] studied how to effectively transfer audio knowledge into the VSR model by using knowledge distillation [25] and memory network [71] . However, these previous VSR approaches have mainly developed for high-resource languages, English and Mandarin [43] . VSR for different languages, especially low VSR resource languages, has only been addressed recently [37, 48, 75, 79] .In particular, a recent approach [75] proposed the labeled data for low VSR resource languages using automatic labeling processes. This paper is the first work exploring sentence-level multilingual VSR with a single model. To mitigate the huge computational costs in training the multilingual VSR model, we propose to pre-train the model with discrete inputs and outputs by using visual speech units. To complement the low amount of video-text data, we bring the automatic labels of [45, 75] and propose curriculum learning that utilizes audio modality to provide rich speech information.Audio speech unit [41] is the discretized speech representation of self-supervised speech models such as HuBERT [28] , Wav2Vec2.0 [6] , and WavLM [9] . It is possible to suppress non-linguistic features and mainly keep the linguistic contents by selecting proper layers to extract the speech features [41, 57] . By using the speech unit as pseudo text, Textless Natural Language Processing becomes possible [29, 34, 42, 51, 58] . Moreover, speech units have promising potential to be used in multi-modal processing as they greatly reduce the data size [8, 35, 52] .Motivated by this, we propose to employ a visual speech unit which is the quantized representation of visual speech features extracted from the self-supervised visual speech model [64] . We analyze the characteristics of visual speech units and show that the visual speech unit contains mainly viseme information while suppressing the other characteristics. 
method: 
 The objective of this paper is to develop a multilingual VSR model, so that multiple languages can be recognized by using a single trained model. To mitigate the large computational costs in developing visual speech processing systems, we propose visual speech units which are discretized representations of visual speech features encoded by a self-supervised speech model.Audio speech units [41] can be obtained by clustering the speech features of a self-supervised speech model such as HuBERT [28] . Analogous to audio speech units, we propose to employ visual speech units, which can be obtained by quantizing the visual speech features derived from a pre-trained visual speech model. In order to get visual speech units, we choose AV-HuBERT [64] for the self-supervised visual speech model, which is well-known for its discriminative visual speech features. However, AV-HuBERT is pretrained on English-only audio-visual data, which deviates from our primary objective of achieving multilingualism. Hence, we initially train a multilingual variant of AV-HuBERT (mAV-HuBERT) to ensure the accurate incorporation of multilingual viseme information into the final visual speech units. To this end, we train the model on 5,512 hours of multilingual dataset composed of 9 languages (En, Es, It, Fr, Pt, De, Ru, Ar, El) by merging LRS2 [13] , LRS3 [3] , VoxCeleb2 [12] , and AVSpeech [21] . As VoxCeleb2 and AVSpeech do not have language identities, we obtain the language identity of each utterance by using a pre-trained language identifier of Whisper [62] , to select the data. For the prediction target of the masked prediction of mAV-HuBERT, we use clusters of speech features Visual Speech Unit obtained from a pre-trained multilingual HuBERT [28, 42] . We use the target size of 1,000 and train the model for 350k steps with one iteration. Through analysis we confirm that mAV-HuBERT is more suitable for multilingual speech modeling than the English-only trained AV-HuBERT in Sec. 4.3.1.With the pre-trained mAV-HuBERT, we extract the visual speech unit by clustering (i.e., quantizing) the output visual speech features, as shown in Fig. 1(a) . Please note that AV-HuBERT can extract both audio-only features and visual-only features through its modality dropout. Hence, we only use the visual inputs to extract the visual speech units. For the token size of the visual speech unit, we use 1,000 so that each visual speech unit can be represented with just 10 bits. Please note that one video frame having grayscale and 88 × 88 size (i.e., the standard for visual speech recognition) requires 61,952 bits [47, 64] . Therefore, we can reduce the data size to 0.016% compared to the raw visual inputs, which enables us to greatly increase the training batch size and accelerate the training speed by removing the visual front-end (e.g., 2D CNNs). We analyze the efficiency of visual speech units by comparing them with the standard raw inputs in Sec. 4.3.2.By representing all training video data into visual speech units, we can significantly reduce the data size, enabling efficient training of a VSR model on large-scale multilingual data. Based on this key concept, we propose to pre-train our model to predict text by setting the inputs with visual speech units. Therefore, now the inputs and outputs are both discrete, which is illustrated in Fig. 1 As the visual speech units mainly contain linguistic information, we can pre-train the model to construct the knowledge of visual speech modeling even by using discrete inputs. We analyze the information contained in the visual speech units and validate how it can be worked, in Sec. 4.3.3. Nevertheless, translating visual speech units directly into output text from scratch is challenging for the model in identifying optimal solutions. Since visual information contains scarce speech information [36, 63, 77] compared to audio, training the model directly to perform visual-to-text conversion might be hard to find the solution.To mitigate this, we bring the motivation from the recent success of VSR, which utilizes auxiliary audio information during training [4, 38, 48, 63, 64, 74, 77] . Specifically, we initiate the pre-training with audio-visual speech units where both audio speech units and visual speech units are utilized as inputs, similar to [19, 64] . Then, we gradually masked out the audio speech units as the pre-training progressed, resulting in the final training stage exclusively utilizing visual speech units as inputs. Therefore, the model can easily find the optimization points through this curriculum learning, with the aid of complementary audio speech information. Concretely, the embeddings of the visual speech units 𝑓 𝑣 ∈ R 𝑇 ×𝐷 and audio speech units 𝑓 𝑎 ∈ R 𝑇 ×𝐷 are concatenated as, 𝑔 = M (𝑓 𝑎 ) ⊕ 𝑓 𝑣 , where 𝑇 is the sequence length, 𝐷 is the dimension of embedding, ⊕ represents concatenation operation in the embedding dimension, and 𝑔 ∈ R 𝑇 ×2𝐷 is the concatenated feature. M (•) is a masking function that randomly masks out 𝑝% of frames from the input sequence. We progressively increase the 𝑝 from 0 to 100 as the pre-training progresses so that the transition from audio-visual inputs to visual inputs can be made. The effectiveness of this curriculum learning using input transition from audio-visual to visual can be found in Sec. 4.3.4.Then, we reduce the embedding dimension of the concatenated feature 𝑔 using a linear layer. Here, we provide the language information by providing the language embedding which will be added to the feature, following [16] . Finally, through the Transformer encoder-decoder architecture, we translate the visual inputs into the output text in an auto-regressive manner. The objective function of our learning problem can be represented as follows,EQUATIONwhere 𝑦 𝑠 is the text annotation for current step 𝑠 and 𝑦 <𝑠 is the previous outputs, X is the input speech units, and 𝑆 is the length of the text. For multilingual training, we use five languages (En, Pt, Es, Fr, It) by merging LRS3 [3] , mTEDx [20] , automatic labels for En of [45] , and automatic labels for Pt, Es, Fr, and It of [75] , forming 4,545 hours of data. We summarize the dataset statistics in Table 1 .Even though we can directly perform multilingual VSR with the pre-trained model with discrete inputs, it is hard to outperform the model using continuous features. This is expected, as information is lost during the quantization process. However, as the pre-trained model has already learned how to model the multilingual visual pronunciation and generate languages, the finetuning with the continuous features is straightforward and more effective than direct training of the multilingual VSR model from scratch. For finetuning, we detach the unit embedding and linear layers, and attach the pre-trained mAV-HuBERT, as illustrated in Fig. 1(c ). The model is trained end-to-end with the same objective function and training data as the pre-training. In Sec. 4.3.5, we analyze the performances of multilingual VSR using the single proposed model and multiple previous monolingual models. 
experiment 4.1 dataset: 
 Lip Reading Sentences 2 (LRS2) [13] is one of the largest English datasets for VSR. The dataset consists of 223 hours of training data collected from British TV shows.Lip Reading Sentences 3 (LRS3) [3] is a popular English VSR database. It has about 430 hours of video, and each video clip is collected from TED and TEDx. We evaluate the English VSR performances on LRS3.Multilingual TEDx (mTEDx) [20] is a multilingual dataset originally proposed for speech recognition and translation. The dataset provides 8 languages collected from TEDx talks. As the dataset also provides the video links of original talks, we download the video online. We remove the unavailable videos for VSR by referring to [48] . We use four languages, Spanish (Es), Italian (It), French (Fr), and Portuguese (Pt) for training and evaluating the developed VSR model, following [37, 48, 75] .VoxCeleb2 [12] is a multilingual audio-visual dataset for speaker recognition [33] . This dataset has over 1 million utterances and contains 6,112 celebrities. For mAV-HuBERT training, we use the data corresponding to the following 9 languages, English (En), Spanish (Es), Italian (It), French (Fr), Portuguese (Pt), German (De), Russian (Ru), Arabic (Ar), and Greek (El), by identifying the language identity using Whisper [62] . For VSR training, as the dataset does not provide text annotations, we use the automatic labels of [45] for En, and [75] for Es, It, Fr, and Pt.Audio Visual Speech Dataset (AVSpeech) [21] is a largescale audio-visual speech dataset. AVSpeech contains roughly 290k YouTube videos, and the total duration of these videos is 4700 hours. Since the dataset does not provide text annotations, we use the same strategy with the VoxCeleb2, using 9 languages for mAV-HuBERT and automatic labels for VSR.Preprocessing. The video is resampled to 25 fps. We detect the facial landmarks using RetinaFace [17] , crop mouth regions using 96 × 96 sizes of bounding box, and convert them into grayscale. For data augmentation, we randomly crop the video into 88 × 88 and horizontally flip it during training. The audio is resampled to 16kHz. In order to obtain the audio speech unit, we feed the audio into a multilingual trained HuBERT [28, 42] and cluster the extracted features with 1,000 token size. Finally, the audio speech unit is resampled to 25 fps to align with the sampling rate of the visual inputs. For the text, we construct one multilingual dictionary with 1,000 subword units by using SentencePiece tokenizer [40] .Architecture. The mAV-HuBERT has the same architecture as the AV-HuBERT [64] large configuration and only differs in pretraining data. For visual speech unit-to-text translation, the model is composed of two unit embedding layers, one linear layer, one language embedding layer, 6 transformer encoders, and 6 transformer decoders. Each unit embedding layer embeds 1,000 tokens Training. For training mAV-HuBERT, we follow the original AV-HuBERT [64] and train it with the masked prediction task. For the prediction target, we use 1,000 clusters extracted from multilingual trained HuBERT [28, 42] . We train the model for 350k steps using 64 3090 RTX GPUs. For pre-training the proposed model (i.e., visual speech unit-to-text translation), we train the model for 11 epochs with a tri-stage learning rate scheduler and 32 GPUs. For the progressive masking M (•), we set 𝑝 as 0 for the first 10% of training. From 10% to 70% of training, we linearly increase the masking ratio 𝑝 from 0 to 100. After 70% of training, 𝑝 is set to 100 so that only visual speech units are used. We finetune the model with the continuous features for 8 epochs using 32 GPUs. For all experiments, the Adam optimizer [39] is used. For beam search decoding, we use a beam width chosen from {20, 25, 30, 35} and a length penalty of 0. The detailed training configuration can be found in the supplementary.Before extracting the visual speech units by using SSL visual speech models, we need to confirm which model is best suitable for our multilingual VSR purposes. Therefore, we compare the performances of English-trained AV-HuBERT and multilingual-trained AV-HuBERT (mAV-HuBERT). To this end, we finetune the pre-trained mAV-HuBERT and AV-HuBERT on 5 languages (i.e., En, Es, It, Fr, Pt) by merging LRS3 and mTEDx databases. The multilingual VSR performances of the two models are shown in Table 2 . The results show that when performing multilingual VSR with a single model, the mAV-HuBERT is much more effective than the English-only trained AV-HuBERT. Specifically, for Es, It, Fr, and Pt, the mAV-HuBERT outperforms the AV-HuBERT with large margins of over 10% WERs. For English, English-only trained AV-HuBERT achieves better performance, and this tendency is also observed in Radford et al. [62] ; Increasing language diversity without scaling the model size, the performances for major languages can be lower than the monolingual model. As the mAV-HuBERT shows better performances in modeling multilingual visual speech, we extract the visual speech unit by using the mAV-HuBERT.Inputs. To confirm the efficiency of using visual speech units instead of raw videos as inputs, we compare the batch size, training iteration time, and total training time between the standard VSR method that uses raw video as inputs and the proposed method. 1) phoneme mapping visualization [66] and 2) speaker recognition. Firstly, following [66] , we visualize the phonetic information of the audio speech unit and visual speech unit which are obtained from mAV-HuBERT. Therefore, only audio inputs are used to extract audio speech units and video inputs for visual speech units. We set the cluster centroid as each row of the weight of the final layer of mAV-HuBERT (i.e., classifier for 1,000 units), so that the same boundary can be obtained for different modalities. Fig. 3 displays the visualization results of the 200 units out of 1,000 that appeared most frequently. We can confirm that each unit in both the audio speech units and visual speech units contains distinctive linguistic information (i.e., phoneme or viseme). By comparing the audio speech unit and visual speech unit, we can find that the homophenes which refer to the different pronunciations having the same lip movement (e.g., name, tame, dame) are confusingly represented in visual speech units [38] . For example, some units representing 'n' in audio speech units are changed to 'd' or 't' in visual speech units. Moreover, we can find that compared to audio speech units, more visual speech units are pointing to the vowel (i.e., blue-colored area), which shows the ambiguity of lip movements compared to the audio modality. These are the natural results reflecting the characteristics of different speech modalities, and should not be taken as an indication of the inadequacy of the visual modality in speech modeling. In addition, we visualize some video frames corresponding to each visual speech unit in Fig. 4 , illustrating that similar lip movements are consistently mapped to the same index regardless of pose variations and speakers. For example, the 648-th visual speech unit represents lip frames related to the viseme 'a', and the 912-th visual speech unit corresponds to the viseme 'o'. Through the visualization, we can confirm that our visual speech units contain the linguistic information, the viseme, which enables us to pre-train the model to build the knowledge of visual speech modeling.Secondly, we analyze how much degree the speech units contain non-linguistic information through speaker recognition. To this end, we use raw audio speech, audio/visual speech features extracted from mAV-HuBERT, and audio/visual speech units as inputs to perform the speaker recognition. For the model, we utilize a pretrained speaker recognition model of [18] and train the model with different inputs. To match the input size with the model, we use one additional embedding layer for both audio speech units and visual speech units. When we use the speech features from mAV-HuBERT, we utilize one additional linear layer for both audio speech features and visual speech features. For training, we utilize data less than 20 seconds in VoxCeleb2 [12] dev set. The speaker verification is performed on the test data of VoxCeleb2 having less than 20 seconds, where each test sample is assigned one positive sample and one negative sample. Therefore, 27,816 positive pairs and negative pairs are utilized respectively, thus random prediction yields 50% EER.The speaker verification results (Equal Error Rate; EER) are shown in Table 4 . When we use raw audio as input, the system can almost perfectly distinguish the input speakers with 2.38% EER. When we use the features of the pre-trained mAV-HuBERT, the verification performance is dropped to 14.96% and 19.42% on audio and visual modalities, respectively. This shows that the masked prediction of AV-HuBERT [64] forces the model to learn linguistic information while somewhat discarding the speaker information. Finally, when we discretize the speech features of mAV-HuBERT and obtain the speech units, we can greatly suppress the speaker information. The performance is dropped to 28.84% by using the audio speech unit and 32.74% by using the visual speech unit. Through the above two experiments, we can confirm that the quantization process averages out the speaker effects (i.e., non-linguistic information) in the speech representations while maintaining the content. This enables us to build a speech representation model by using speech units as input. 5 . The performance of '-Unit Pretraining' is obtained by directly finetuning the mAV-HuBERT on the multilingual videotext without using visual speech units. In this case, the overall VSR performances are dropped especially for non-English languages. Moreover, we require more training times compared to the proposed method as shown in Table 3 . When we do not utilize curriculum learning (i.e., '-Curriculum'), the performance dramatically decreases and even shows worse performances than the without pre-training method in some languages. This shows that directly performing the visual speech unit to text translation from scratch is challenging to the model in finding the optimal points. Therefore, the proposed curriculum learning is crucial when learning from the visual speech units. These results also coincide with the previous methods utilizing multi-modal complementary in VSR training [4, 38, 63, 64, 73, 77] . Finally, when we directly perform multilingual VSR with the pre-trained model (i.e., with visual speech units), overall performances are decreased as fine information is lost during the quantization process. Therefore, finetuning with the continuous features for a few epochs should be performed to maximize the pre-trained knowledge.Model. We validate the effectiveness of the proposed multilingual VSR method by comparing it with 1) the multilingual VSR model and 2) the monolingual VSR model. Since there is no prior work exploring multilingual VSR with a single model, we train AV-HuBERT [64] to perform multilingual VSR and set it as our baseline. Moreover, since the previous non-English VSR methods are languagespecific, we compare the performance of our model with multiple monolingual models.Comparison with multilingual VSR method. Table 6 shows the performance comparison results of multilingual VSR methods. Both the AV-HuBERT and the proposed method are finetuned on 4,545 hours of multilingual video-text paired data. The proposed method outperforms the AV-HuBERT for all languages except English. In particular, the proposed method demonstrates significantly improved performance for Es, It, Fr, and Pt, with gains of more than 4% WERs. For the high-resource language En, the proposed method achieves similar performance with AV-HuBERT but slightly falls behind. Considering multilingualism, the results confirm that Comparison with monolingual VSR method. Here, we compare the proposed multilingual VSR performances with the previous state-of-the-art monolingual VSR methods. Please note that the proposed method utilizes a single trained model across the languages, while different methods utilize multiple language-specific VSR models. The results are shown in Table 7 . By comparing with the recent state-of-the-art method [75] , we outperform it in 3 languages Spanish (Es), Italian (It), and French (Fr), by 3.0%, 2.1%, and 3.1% WER, respectively. In the English VSR, our method achieves 24.4% WER while the current previous method [45] achieves 20.5% WER. As discussed in Radford et al. [62] , the performance of high-resource languages can be further improved by scaling the model size to accommodate the diversity of languages. As a result, the proposed multilingual VSR model achieves the best score in Es, It, and Fr, and the second-best score in En and Pt with a single-trained model. Through the comparisons, we can confirm not only the computation efficiency but also the effectiveness of the proposed method in multilingual VSR by outperforming and achieving comparable results with the previous language-specific VSR methods. 
conclusion: 
 In this paper, we proposed an efficient multilingual VSR method using a single model. Specifically, we proposed to use visual speech units for pre-training a VSR model to mitigate the huge computational loads in building massive multilingual visual speech modeling. With the proposed strategy, we can greatly reduce the data size and effectively pre-train the VSR model on large-scale multilingual VSR databases. By analyzing the visual speech unit, we validated it contains linguistic information and enables visual speech modeling using discrete inputs. To complement visual speech information with audio, we proposed curriculum learning by gradually increasing the task difficulty. Finally, by finetuning the model on continuous features, we set new state-of-the-art multilingual VSR performances by achieving comparable VSR performances with the previous language-specific VSR models. To the best of our knowledge, this is the first work exploring the multilingual VSR and employing visual speech units as inputs of VSR systems.  
 
 Appendix:  


Decision: The paper proposes a new sentence-level multilingual process to learn units for visual speech recognition, based on AV-Hubert. This model utilizes multilingual data to pre-train mAV-Hubert in order to extract discrete visual speech units, followed by fine-tuning using continuous representations. Although the training consists of two phases, it can achieve the results of traditional VSR models in a shorter training time, while multilingual visual recognition can be accomplished using only one model. The work demonstrates that resource sharing can also be useful for visual speech recognition, and improves performance over several baselines. The work is novel and fits ACM MM themes well.

Reasons to accept:
- The paper proposes that a multilingual visual model can be built and it achieves the best performance over multiple languages compared to baselines.
- The paper uses curriculum learning, which transitions from audio-visual speech units to visual speech units, helping the model achieve stable optimization and better performance.
- The proposed approach increases efficiency compared to baselines.

Reasons to reject:
- Reviewers asked several clarification questions, to which authors responded well. However there is no guarantee that an updated version of the paper will not still meet readers open questions.

Reviews: review: paper_summary: paper_summary: The paper proposes a new sentence-level multilingual visual recognition scheme, based on AV-Hubert, which utilizes multilingual data to train mAV-Hubert in order to extract visual speech units. Pre-training is performed using discrete units, followed by fine-tuning using continuous representations. Although the training consists of two phases, it can achieve the results of traditional VSR models in a shorter training time, while multilingual visual recognition can be accomplished using only one model.
strength_weakness: strength_weakness: - The paper proposes that multilingual visual recognition can be accomplished using only a single model and it achieves the best performance over multiple languages compared to baselines.
- The paper proposes to use the visual speech unit as the training intermediate medium, which significantly improves the training efficiency, and experimentally verifies that the visual speech unit effectively eliminates irrelevant information.
- The paper validates the usefulness of mAV-Hubert and the rationality of the training scheme.
limitations: limitations: - I am confused about the difference between Table 2 and Table 6. Is there only a dataset size inconsistency between the two experiments (i.e., Table 2 uses only LRS3 and mTEDx, Table 6 uses all 4,545 hours of data)?
- Is the mAV-HuBERT frozen or trainable during the fine-tuning phase? And does the paper try the effect of merging pre-training and fine-tuning together (i.e., directly using continuous representations as inputs to enable the system to accomplish end-to-end learning)?
- Placing the training data (hours) for each language in the supplemental material into the paper (e.g., Table 7) would make it easier to compare with the baselines.
paper_summary: paper_summary: This paper proposes an efficient training strategy for multilingual visual speech recognition using discretized visual speech representation. By using these visual speech units as inputs and discrete text outputs, the training data size is significantly reduced, improving training efficiency by approximately 10 times.
strength_weakness: strength_weakness: 1. This is the first work exploring sentence-level multilingual VSR with a single-trained model. 
2.  The use of curriculum learning, which gradually transitions from audio-visual speech units to visual speech units, helps the model achieve stable optimization and better performance.
limitations: limitations: 1. Solely using 1000 clusters for pre-training lacks sufficient support. It is suggested that adds the ablation study about the number of clusters. 
2. The authors demonstrate that using clustered features for tasks like speaker identification leads to degraded performance, indicating a reduction in non-linguistic information, they do not effectively prove that linguistic information has been preserved with minimal loss, which is crucial for recognition tasks. It is suggested that the authors provide evidence that the reduction in non-linguistic information is achieved without significant loss of linguistic information, ensuring that the model remains effective for recognition tasks.
3. It is suggested that a longitudinal comparison is needed to determine the effectiveness of different fine-tuning strategies. Specifically, the authors should consider the comparison of the following settings:
1)Fine-tuning with Audio-Visual Speech Units Followed by Visual Speech Units: First fine-tune the model using both audio and visual speech units, and then fine-tune it further using only visual speech units.
2)Curriculum Learning Approach: Gradually increase the task difficulty by progressively transitioning from audio-visual speech units to visual speech units.
paper_summary: paper_summary: This paper proposes an efficient training approach for multilingual Visual Speech Recognition (VSR) using discretized visual speech representation. The proposed method involves the extraction of visual speech units by quantizing visual speech features obtained from a self-supervised model, mAV-HuBERT, trained on multilingual audio-visual data.
strength_weakness: strength_weakness: The training efficiency is highly improved, which is a key to the VSR task.
limitations: limitations: The paper requires additional experiments to demonstrate whether different pre-training clustering quantities affect the results for different languages. 
Figure 3 separates Audio Speech Units and Visual Speech Units into two distinct plots. This approach lacks a direct comparison of the clustering characteristics between audio and visual data. 
The details regarding the curriculum learning approach are insufficient. 
The paper requires a detailed analysis of the mixed dataset used for training the multilingual VSR model.
Only the AvHuBERT in Tab.6 is regarded as the multilingual baseline, more state-of-the-art approaches are required to validate the proposed approach.