MIR-GAN: Refining Frame-Level Modality-Invariant Representations with Adversarial Network for Audio-Visual Speech Recognition
Year: 2023

Abstract:
Audio-visual speech recognition (AVSR) attracts a surge of research interest recently by leveraging multimodal signals to understand human speech. Mainstream approaches addressing this task have developed sophisticated architectures and techniques for multi-modality fusion and representation learning. However, the natural heterogeneity of different modalities causes distribution gap between their representations, making it challenging to fuse them. In this paper, we aim to learn the shared representations across modalities to bridge their gap. Different from existing similar methods on other multimodal tasks like sentiment analysis, we focus on the temporal contextual dependencies considering the sequence-to-sequence task setting of AVSR. In particular, we propose an adversarial network to refine framelevel modality-invariant representations (MIRGAN), which captures the commonality across modalities to ease the subsequent multimodal fusion process. Extensive experiments on public benchmarks LRS3 and LRS2 show that our approach outperforms the state-of-the-arts1.

heading: 1 Introduction
text: Human perception of the world intrinsically comprises multiple modalities, including vision, audio, text, etc. (McGurk and MacDonald, 1976; Baltrušaitis et al., 2018). Audio-visual speech recognition (AVSR) leverages both audio and visual modalities to understand human speech, improving the noise-robustness of audio-only speech recognition with noise-invariant lip movement information (Sumby and Pollack, 1954). Thanks to recent advances of deep learning techniques, AVSR research has gained a remarkable progress (Afouras et al., 2018a; Ma et al., 2021; Shi et al., 2022b).
Currently, the mainstream AVSR approaches are centered around developing sophisticated architectures and techniques for multi-modality fusion,
1Code is available at https://github.com/YUCHE N005/MIR-GAN.
including simple feature concatenation (Makino et al., 2019; Ma et al., 2021; Pan et al., 2022), recurrent neural network (Petridis et al., 2018; Xu et al., 2020) and cross-modal attention module (Afouras et al., 2018a; Lee et al., 2020). Despite the advances, these approaches are often challenged by the representation gap persisting between naturally heterogeneous modalities (Hazarika et al., 2020).
Recently in some other multimodal tasks like sentiment analysis (Hazarika et al., 2020; Yu et al., 2021; Yao and Mihalcea, 2022) and cross-modal retrieval (Xiong et al., 2020), there have been research works proposing to learn two distinct representations to benefit multimodal learning. The first representation is modality-invariant, where multiple modalities of a same utterance are mapped to a shared space, indicating the homogeneous semantic meaning from the speaker. In addition, they also learn modality-specific representations that are private to each modality. Given an utterance, each modality contains some unique features with respect to speaker-sensitive information (Tsiros, 2013). Combing these two representations provides a holistic view of multimodal data for downstream tasks (Yang et al., 2022). However, these methods focus on utterance-level representations that could be easily mapped to either shared or individual modality space using similarity cost functions, which does not apply to AVSR task that requires
11610
sequence-to-sequence mapping with temporal contextual dependencies (Petridis et al., 2018).
Motivated by above observations, we propose an adversarial network to refine frame-level modalityinvariant representations (MIR-GAN) for capturing the commonality across modalities, which bridges their heterogeneous gap to ease the subsequent multimodal fusion. In particular, we first design a MIR generator to learn modality-invariant representations over the shared audio-visual modality space. Meanwhile, a modality discriminator is proposed to strengthen its modality agnosticism via adversarial learning. Moreover, to further enrich its contextual semantic information, we propose a mutual information maximization strategy to align the refined representations to both audio and visual modality sequences. Finally, both modality-invariant and -specific representations are fused for downstream speech recognition. Empirical results demonstrate the effectiveness of our approach. In summary, our main contributions are:
• We present MIR-GAN, an AVSR approach to refine frame-level modality-invariant representations, which captures the commonality across modalities and thus bridges their heterogeneous gap to ease multimodal fusion.
• We first learn modality-invariant representations with a MIR generator, followed by another modality discriminator to strengthen its modality agnosticism via adversarial learning. Furthermore, we propose a mutual information maximization strategy to enrich its contextual semantic information. Finally, both modality-invariant and -specific representations are fused for downstream recognition.
• Our proposed MIR-GAN outperforms the state-of-the-arts on LRS3 and LRS2 benchmarks. Extensive experiments also show its superiority on ASR and VSR tasks.

heading: 2 Related Work
text: Audio-Visual Speech Recognition. Current mainstream AVSR methods focus on sophisticated architectures and techniques for audio-visual modality fusion. Prior methods like RNN-T (Makino et al., 2019), Hyb-Conformer (Ma et al., 2021) and MoCo+wav2vec (Pan et al., 2022) employ simple feature concatenation for multimodal fusion, other works including Hyb-RNN (Petridis et al., 2018)
and EG-seq2seq (Xu et al., 2020) leverage recurrent neural network for audio-visual fusion. In addition, cross-modal attention has also become popular recently for multimodal interaction and fusion in AVSR tasks, such as TM-seq2seq (Afouras et al., 2018a), DCM (Lee et al., 2020) and MMST (Song et al., 2022). Despite the effectiveness, these fusion techniques are often challenged by the representation gap between naturally heterogeneous modalities. Recently, multimodal self-supervised learning has been popular for capturing unified crossmodal representations, like AV-HuBERT (Shi et al., 2022a) and u-HuBERT (Hsu and Shi, 2022), which achieve the state-of-the-art but require abundant unlabeled data and computing resources. In this work, we propose a supervised learning scheme to efficiently refine modality-invariant representations for bridging the heterogeneous modality gap.
Modality-Invariant and -Specific Representations. Recent studies in many multimodal tasks suggest that the model benefits from both shared and individual modality representations, including multimodal sentiment analysis (Hazarika et al., 2020; Yu et al., 2021; Yang et al., 2022), person re-identification (Wei et al., 2021; Huang et al., 2022), cross-modal retrival (Zeng et al., 2022) and image-sentence matching (Liu et al., 2019), etc. MISA (Hazarika et al., 2020) maps the multimodal features into two spaces as modality-invariant and -specific representations, and then fuses them for downstream classification. MCLNet (Hao et al., 2021) learns modality-invariant representations by minimizing inter-modal discrepancy and maximizing cross-modal similarity. VI-REID (Feng et al., 2019) builds an individual network for each modality, with a shared identity loss to learn modalityinvariant representations. However, these methods map utterance-level representations to modalityinvariant or -specific spaces via similarity cost functions, while AVSR is sequence-to-sequence task that requires contextual semantic information. To this end, we propose an adversarial network with mutual information maximization to refine framelevel modality-invariant representations that subjects to temporal contextual dependencies.
Adversarial Network. The concept of adversarial network starts from GAN (Goodfellow et al., 2014), which has attracted a surge of research interests due to its strong ability of generating high-quality novel samples according to existing data. The bestknown applications include image-to-image trans-
va . (d) Visual and
audio encoders that learn modality-specific representations fspev , f spe a . “MIM” is mutual information maximization.
lation (Isola et al., 2017) and image synthesis (Denton et al., 2015; Radford et al., 2015). Recently, GAN is further applied to multimodal tasks such as text-to-image synthesis (Reed et al., 2016; Tan et al., 2020), video captioning (Yang et al., 2018; Bai et al., 2021) and cross-modal retrieval (Qian et al., 2021). In this work, we leverage the strong distinguishing ability of adversarial network to strengthen the modality agnosticism of the learned modality-invariant representations.

heading: 3 Methodology

heading: 3.1 Overview
text: The overall architecture of our proposed MIR-GAN is illustrated in Fig. 2. First, we have two front-end modules2 to process the input streams, which generate two modality sequences, i.e., fv, fa ∈ RT×D, where T is number of frames and D is embedding size. These two sequences are then fed by visual and audio encoders respectively to generate modality-specific representations, i.e., fspev , f spe a ∈ RT×D. Based on that, we propose a MIR generator to learn modality-invariant representations by extracting the shared information of two modalities, i.e., f invva ∈ RT×D. Meanwhile, we design a modality discriminator to strengthen its modality agnosticism via adversarial learning. In addition, to further
2Details are presented in Appendix A.3.
enrich its contextual semantic information, we propose a mutual information maximization (MIM) strategy to align the refined representations to both audio and visual modality sequences. Finally, both modality-invariant and -specific representations are fused for downstream speech recognition.

heading: 3.2 Visual & Audio Encoders
text: As illustrated in Fig. 2 (d), we introduce a pair of visual and audio encoders to learn modality-specific representations. Following Transformer (Vaswani et al., 2017) architecture, they first employ selfattention modules to capture the contextual dependencies within each modality, followed by crossattention modules for interaction between two modalities, which can initially narrow their gap to benefit the subsequent modality-invariant representation learning. Finally, there are feed-forward networks to generate the modality-specific outputs.

heading: 3.3 MIR-GAN
text: With learned modality-specific representations, we propose MIR-GAN to refine frame-level modalityinvariant representations. First, we design a MIR generator to extract the shared information of two modalities, which generates a modality-invariant representation f invva ∈ RT×D. Meanwhile, we design a modality discriminator to strengthen its modality agnosticism via adversarial learning.

heading: 3.3.1 MIR Generator
text: Fig. 2 (b) details the architecture of proposed MIR generator G, where we design a hybrid-modal attention (HMA) module to extract out the part of information in each modality-specific representation that is related to both modalities:
sm = HMA(f spe m , fva), m ∈ {v, a}, (1)
where the subscript m denotes modality. The resulted features are then added to input sequence fva to form the final modality-invariant representation:
f invva = Norm(fva + ∑
m∈{v,a} Conv(sm)), (2)
where the “Norm” denotes layer normalization (Ba et al., 2016), “Conv” denotes 1×1 convolution followed by PReLU activation (He et al., 2015).
Hybrid-Modal Attention (HMA) first involves a cross-attention sub-module to extract the information in each modality-specific representation that is related to both modalities, with the query input fva comprising both visual and audio sequence information, as shown in Fig. 3:
f sharem = Cross-Attention(fva, f spe m , f spe m ), (3)
To further make the extracted feature invariant to modalities, we design a parallel convolutional network to learn a mask for filtering out the modalityspecific information:
sm = f share m ⊗ σ(Conv(f spem ∥fva)), (4)
where “Conv” denotes 1×1 convolutional layer, ∥ denotes feature concatenation, σ denotes Sigmoid activation, ⊗ denotes element-wise multiplication.
As a result, the output representation sm from HMA involves information regarding both visual and audio modalities, making the final output f invva (in Eq. 2) invariant to modalities.

heading: 3.3.2 Modality Discriminator
text: With the generated modality-invariant representation, we further design a modality discriminator D to strengthen its modality agnosticism via adversarial learning. As shown in Fig. 2 (c), the discriminator consists of two linear layers followed by Sigmoid activation to predict a scalar between 0 and 1 for each frame, indicating which modality it belongs to (i.e., 0 for visual and 1 for audio):
D(f) ∈ RT×1, f ∈ {fspev , fspea , f invva }, (5) Therefore, for frames in modality-specific representations f spev and f spe a , we hope the discriminator can correctly classify the modality type, i.e., 0 or 1. In contrast, in order to strengthen the modality agnosticism of refined representation f invva , we hope it can confuse the discriminator with the output around 0.5, i.e., a medium between two modalities.
With above designs of generator and discriminator, the adversarial training objective of MIR-GAN can be mathematically formulated as:
LGAN = LD + LG = Ef [logD(fspea ) + log(1−D(fspev ))] + Ef [−logD(f invva )− log(1−D(f invva ))],
(6)
where f invva = G(f spe v , f spe a , fva), E denotes the expectation over all the temporal frames in current data batch. Details of the corresponding optimization strategy are illustrated in Alg. 1.

heading: 3.4 Mutual Information Maximization
text: The MIR-GAN successfully refines the modalityinvariant representation by focusing on the modality commonality and agnosticism, while the original semantic information may not be preserved. To this end, we further design a mutual information maximization (MIM) strategy via contrastive learning to enrich the contextual semantic information in refined modality-invariant representation.
In particular, we formulate a contrastive loss function to maximize the mutual information between modality-invariant representation f invva and the modality-specific representations fspev , f spe a :
LMIM =− T∑
i=1 log exp(⟨f invva_i, fspev_i ⟩/τ)∑T j=1 exp(⟨f invva_i, f spe v_j ⟩/τ)
− T∑
i=1 log exp(⟨f invva_i, fspea_i ⟩/τ)∑T j=1 exp(⟨f invva_i, f spe a_j ⟩/τ) ,
(7)
where ⟨ ·, · ⟩ denotes cosine similarity, τ is temperature parameter. The subscripts i and j denote frame index, where f invva /f spe v /f spe a ∈ RT×D.
The constructed positive and negative samples are distinguished by frame index. As same frame of different representations express similar semantic meanings, we assign them as positive samples to strengthen consistency, while the mismatched frames are pulled apart from each other. As a result, the MIM strategy can enrich the semantic information in final modality-invariant representation.

heading: 3.5 Optimization
text: The optimization strategy of MIR-GAN is detailed in Alg. 1. After the forward-propagation process, we calculate LGAN and LMIM according to Eq. 6 and Eq. 7. Meanwhile, the downstream speech recognition loss Lrec is calculated as the crossentropy between recognized text and the groundtruth transcription. The final training objective of MIR-GAN can therefore be written as: L = Lrec + λGAN · LGAN + λMIM · LMIM ,
(8)
where λGAN , λMIM are weighting parameters to balance different training objectives.
Inspired by GAN training strategy (Goodfellow et al., 2014), we split the back-propagation process into two steps. First, we maximize LGAN to update the discriminator, where the generator is detached from optimization. According to Eq. 6, maximizing the first term of LGAN (i.e., LD) trains the discriminator to correctly classify the two modalities, while increasing the second term amounts to informing discriminator that f invva is modality-specific and can either be visual or audio3 (this is opposite to what we desire as modality-invariant). Second, we freeze discriminator and update the rest network, where minimizing LG pushes the discrimination output of f invva to 0.5,
3 which is a medium between visual and audio modalities, i.e., modality-agnostic. In addition, Lrec optimizes the downstream speech recognition model and LMIM implements the MIM strategy. The entire system is trained in an end-toend manner with well-tuned weighting parameters.

heading: 4 Experiments

heading: 4.1 Experimental Setup
text: Datasets. We conduct experiments on two largescale public benchmarks, LRS3 (Afouras et al.,
3Function log(x) + log(1− x) reaches maximum at x = 0.5, and the minimum is obtained around x = 0 and x = 1.
Algorithm 1 MIR-GAN Optimization. Require: Training data D that contains visual-audio pairs
(xv, xa) and the text transcription y. The MIR-GAN network θ that consists of front-ends θvf and θaf , encoders θvae, MIR generator θG, modality discriminator θD and downstream speech recognition model θrec. Hyper-parameter weights λGAN , λMIM . 1: Randomly initialize the entire system θ. 2: if select self-supervised setting then 3: Load the pre-trained AV-HuBERT for speech recogni-
tion model θrec and front-ends θvf , θaf 4: end if 5: while not converged do 6: for (xv, xa) ∈ D do 7: FORWARD-PROPAGATION: 8: fv = θvf (xv), fa = θaf (xa) ▷ front-ends 9: fspev , fspea = θvae(fv, fa) ▷ encoders 10: fva = fv∥fa 11: f invva = θG(fspev , fspea , fva) ▷ Generator 12: ŷ = θrec(fspev ∥fspea ∥f invva ) ▷ recognition 13: TRAINING OBJECTIVES: 14: LGAN (LD and LG) in Eq. 6 ▷ Discriminator 15: LMIM in Eq. 7 ▷ MI maximization 16: Lrec = CrossEntropy(ŷ, y) 17: BACK-PROPAGATION: ▷ adversarial training 18: UPDATE DISCRIMINATOR: ▷ unfreeze θD 19: argmax
θD LGAN 20: UPDATE THE REST NETWORK: ▷ freeze θD 21: argmin
θ\θD Lrec+λGAN ·LG+λMIM ·LMIM
22: end for 23: end while
2018b) and LRS2 (Chung et al., 2017). LRS3 dataset collects 433 hours of transcribed English videos in TED and TEDx talks from over 5000 speakers, which is the largest publicly available labeled audio-visual speech recognition dataset. LRS2 dataset contains 224 hours of video speech, with a total of 144K clips from BBC programs. Model Configurations. We first build a base model with only front-ends and downstream speech recognition module, which follows Transformer architecture with 24 encoder layers and 9 decoder layers. Based on that, we build the MIR-GAN with NE = 3 visual & audio encoder layers and NG = 3 MIR generator layers. To maintain similar model size, we only use 12 encoder layers and 9 decoder layers in the recognition model. The number of parameters in our base model and MIR-GAN are 476M and 469M respectively. We also use Conformer (Gulati et al., 2020) as our backbone. In addition, we implement a self-supervised setting by loading pre-trained AV-HuBERT4. Following prior work (Shi et al., 2022b), we employ data augmentation and noisy test set based on MUSAN
4https://github.com/facebookresearch/ av_hubert
Method Backbone Criterion Unlabeled Labeled DataAug LM WER(%)
noise (Snyder et al., 2015). More detailed settings are presented in Appendix A.3 - A.5. Baselines. To evaluate our proposed MIR-GAN, we select some popular AVSR methods for comparison, which can be roughly divided into two groups. The first is supervised learning method, including TM-seq2seq/CTC (Afouras et al., 2018a), RNN-T (Makino et al., 2019), EG-seq2seq (Xu et al., 2020) and Hyb-Conformer (Ma et al., 2021). Another one is the recently popular self-supervised learning method such as MoCo+wav2vec (Pan et al., 2022), AV-HuBERT (Shi et al., 2022b) and u-HuBERT (Hsu and Shi, 2022).

heading: 4.2 Main Results
text: We conduct experiments on two public datasets under supervised and self-supervised settings, depending on whether use the AV-HuBERT pre-trained model. Results show that our proposed MIR-GAN achieves the state-of-the-art under both settings. LRS3 Benchmark. Table 1 presents the AVSR performance of our proposed MIR-GAN and prior methods on LRS3 benchmark. Under supervised setting, our MIR-GAN achieves significant improvement over the base model in both clean and noisy testing conditions, and the best performance achieves new state-of-the-art (2.1% vs. 2.3%) while without using the language model rescoring. In addition, the Conformer backbone consistently outperforms Transformer (2.1% vs. 2.8%, 8.5%

heading: Supervised

heading: Proposed (Supervised)
text: vs. 11.7%). Under self-supervised setting, MIRGAN also improves the performance of base model, which surpasses or matches previous state-of-theart (1.2% vs. 1.2%, 5.6% vs. 5.8%) while using less unlabeled data for pre-training. LRS2 Benchmark. Table 2 compares the AVSR results of MIR-GAN and baselines on LRS2 benchmark. We can observe that the proposed MIR-GAN outperforms previous state-of-the-art by a large
margin under both supervised and self-supervised settings (3.2% vs. 3.7%, 2.2% vs. 2.6%). In addition, we also observe promising gains of performance in noisy testing conditions.
As a result, our proposed MIR-GAN achieves new state-of-the-art under both supervised and self-supervised settings on two public benchmarks, which demonstrates its superiority on AVSR task.

heading: 4.3 Ablation Study
text: Table 3 presents the ablation study of each component in MIR-GAN. There are three parts of ablation that are independent with each other, i.e., each study is conducted where other two components are kept same as the full MIR-GAN. Importance of Representations. We first investigate the importance of modality-invariant and - specific representations by discarding each of them. When removing the refined modality-invariant representations from multi-modality fusion, the downstream speech recognition performance degrades a lot under all configurations, which verifies its significance of bridging the modality gap. Similarly, we observe that the modality-specific representations also plays an important role in AVSR. Importance of Modules. In this part, we study the role of each module in the proposed MIR-GAN. The visual and audio encoders are designed to extract deep modality-specific representations, which contributes to performance gains of MIR-GAN. Then we replace the core module - MIR generator with simple feature concatenation in refining modality-invariant representations, which results in significant performance degradation. Another key module - modality discriminator also contributes
a lot in MIR-GAN by strengthening the modality agnosticism of refined representations from MIR generator. In this sense, we conclude that all the modules in proposed MIR-GAN contribute positively to the multimodal representation learning.
Importance of Strategies. With the adversarial training strategy illustrated in Alg. 1, the proposed modality discriminator effectively strengthens the modality agnosticism of the refined representations from generator. To verify its effectiveness, we remove the adversarial training strategy from MIRGAN, which results in similar performance degradation to the previous case without modality discriminator. Therefore, it demonstrates the key role of this strategy in learning modality-invariant representations, where further visualization is shown in Fig. 5. Meanwhile, we design a MIM strategy to enrich the contextual semantic information in the refined modality-invariant representations, and similar performance drops can be observed in absence of such strategy. Furthermore, we visualize the attention maps in Fig. 4 to show its effectiveness. The clear diagonals in (c) and (d) indicate the strong ability of MIM strategy to align modality-invariant and -specific representations, which enriches the contextual semantic information in the former.
Visualizations of Modality-Invariant and - Specific Representations. Fig. 5 presents the t-SNE visualization of modality-invariant and - specific representations to illustrate the principle of MIR-GAN. First, we observe from (a) base model that the two modality-specific representations are distantly separated, indicating the heterogeneous gap between different modalities (Hazarika et al., 2020). With the proposed MIR-GAN (no modality discriminator), the two modalities are pushed closer by the interaction between encoders, and the refined modality-invariant representations serve as a medium between them. However, these refined representations are still entangled with audio modality-specific representations5, making them less modality-invariant. Finally, the proposed discriminator effectively strengthens their modality agnosticism via adversarial learning, which are dispersed between two modalities to capture their commonality and thus bridge the heterogeneous modality gap. As a result, the subsequent multi-modality fusion process would be eased and generate better features for downstream recognition.

heading: Comparison with Utterance-Level Approaches
text: As illustrated in §2, prior works have investigated utterance-level modality-invariant and -specific representations with similarity cost functions, includ-
5Audio modality plays the dominant role in AVSR task.

heading: Supervised

heading: Proposed (Supervised)
text: ing MISA (Hazarika et al., 2020), MCLNet (Hao et al., 2021) and VI-REID (Feng et al., 2019). We implement them in our framework as comparison to our proposed MIR-GAN, where we employ their designed similarity cost functions on frame-level representations. As illustrated in Table 4, these utterance-level approaches can also improve AVSR results but still underperforms our proposed approach by a large margin. Performance on Single-Modality Inputs. Furthermore, Table 5 presents the performance of our MIRGAN on single-modality inputs. First, we observe that in all models using both modalities performs better than single modality, and the audio-only case achieves much better results than visual-only case, which shows the dominance of audio modality in AVSR task. Under two single-modality cases, our
proposed MIR-GAN both achieves significant improvement over the base model, and the best performance outperforms or matches previous stateof-the-arts in both supervised and self-supervised settings (2.3% vs. 2.3%, 34.2% vs. 33.6%; 1.3% vs. 1.4%, 26.6% vs. 26.9%). Therefore, even with missing modality, our MIR-GAN can still refine effective modality-invariant representations to benefit the downstream speech recognition, which further verifies the generality of our approach.

heading: 5 Conclusion
text: In this paper, we propose MIR-GAN, an adversarial network to refine frame-level modality-invariant representations for AVSR, which captures the commonality across modalities to ease the multimodal fusion process. MIR-GAN first learns modalityinvariant representation with MIR generator, followed by a modality discriminator to strengthen its modality agnosticism via adversarial learning. Furthermore, we propose a mutual information maximization strategy to enrich its contextual semantic information. Finally, both modality-invariant and -specific representations are fused to provide a holistic view of multimodal data for downstream task. Experiments on public benchmarks show that our MIR-GAN achieves the state-of-the-art.

heading: Limitations
text: The main novelty of our proposed MIR-GAN is refining frame-level modality-invariant representations via adversarial learning. It is promising to combine this approach with the popular selfsupervised pre-training to learn unified multimodal representations. In this work, we only load pretrained AV-HuBERT for the front-ends and speech recognition model, while the proposed modules (i.e., encoders, generator, discriminator) are still trained from scratch. In future, we may include the entire MIR-GAN into self-supervised learning scheme, together with the adversarial learning to refine better multimodal representations.

heading: Ethics Statement
text: All the data used in this paper are publicly available and are used under the following five licenses: the Creative Commons BY-NC-ND 4.0 License and Creative Commons Attribution 4.0 International License, the TED Terms of Use, the YouTube’s Terms of Service, and the BBC’s Terms of Use. The data is collected from TED and BBC and contain
thousands of speakers from a wide range of races. To protect the anonymity, only the mouth area of a speaker is visualized wherever used in the paper.

heading: Acknowledgements
text: The computational work for this article was partially performed on resources of the National Supercomputing Centre, Singapore (https://ww w.nscc.sg).

heading: A Experimental Details

heading: A.1 Datasets
text: LRS36 (Afouras et al., 2018b) is currently the largest public sentence-level lip reading dataset, which contains over 400 hours of English video extracted from TED and TEDx talks on YouTube. The training data is divided into two parts: pretrain (403 hours) and trainval (30 hours), and both of them are transcribed at sentence level. The pretrain
6https://www.robots.ox.ac.uk/~vgg/dat a/lip_reading/lrs3.html
Method Backbone Criterion Unlabeled Labeled DataAug LM WER(%)
part differs from trainval in that the duration of its video clips are at a much wider range. Since there is no official development set provided, we randomly select 1,200 samples from trainval as validation set (∼ 1 hour) for early stopping and hyper-parameter tuning. In addition, it provides a standard test set (0.9 hours) for evaluation. LRS27 (Chung et al., 2017) is a large-scale publicly available labeled audio-visual (A-V) datasets, which consists of 224 hours of video clips from BBC programs. The training data is divided into three parts: pretrain (195 hours), train (28 hours) and val (0.6 hours), which are all transcribed at sentence level. An official test set (0.5 hours) is provided for evaluation use.

heading: A.2 Data Preprocessing
text: The data preprocessing for above two datasets follows the LRS3 preprocessing steps in prior work (Shi et al., 2022a). For the audio stream, we extract the 26-dimensional log filter-bank feature at a stride of 10 ms from input raw waveform. For the video clips, we detect the 68 facial keypoints using dlib toolkit (King, 2009) and align the image frame to a reference face frame via affine transformation. Then, we convert the image frame to gray-scale and crop a 96×96 region-of-interest (ROI) centered on the detected mouth. During train-
7https://www.robots.ox.ac.uk/~vgg/dat a/lip_reading/lrs2.html
ing, we randomly crop a 88×88 region from the whole ROI and flip it horizontally with a probability of 0.5. At inference time, the 88×88 ROI is center cropped without horizontal flipping. To synchronize these two modalities, we stack each 4 neighboring acoustic frames to match the image frames that are sampled at 25Hz.

heading: A.3 Model Settings
text: Front-ends. We introduce the modified ResNet-18 from prior work (Shi et al., 2022a) as visual frontend, where the first convolutional layer is replaced by a 3D convolutional layer with kernel size of 5×7×7. The visual feature is flattened into an 1D vector by spatial average pooling in the end. For audio front-end, we use one linear projection layer followed by layer normalization (Ba et al., 2016). MIR-GAN. We build the MIR-GAN framework based on Transformer, where the embedding dimension/feed-forward dimension/attention heads in each Transformer layer are set to 1024/4096/16 respectively. In addition, we also employ Conformer as backbone, where the depth-wise convolution kernel size is set to 31. We use a dropout of p = 0.1 after the self-attention block within each Transformer layer, and each Transformer layer is dropped (Fan et al., 2019) at a rate of 0.1.

heading: A.4 Data Augmentation
text: Following prior work (Shi et al., 2022b), we use many noise categories for data augmentation. We
select the noise categories of “babble”, “music” and “natural” from MUSAN noise dataset (Snyder et al., 2015), and extract some “speech” noise samples from LRS3 dataset. All categories are divided into training, validation and test partitions.
During training process, we randomly select one noise category and sample a noise clip from its training partition. Then, we randomly mix the sampled noise with input clean audio, at signal-to-noise ratio (SNR) of 0dB with a probability of 0.25.
At inference time, we evaluate our model on clean and noisy test sets respectively. Specifically, the system performance on each noise type is evaluated separately, where the testing noise clips are added at five different SNR levels: {−10,−5, 0, 5, 10}dB. At last, the testing results on different noise types and SNR levels will be averaged to obtain the final noisy WER result.

heading: A.5 Training Details
text: Training. We follow the sequence-to-sequence finetuning configurations of AV-HuBERT (Shi et al., 2022b) to train our systems. We use Transformer decoder to decode the encoded features into unigram-based subword units (Kudo, 2018), where the vocabulary size is set to 1000. The temperature τ in Eq. 7 is set to 0.1, and the weighting parameters λGAN/λMIM in Eq. 8 are set to 0.01/0.005 respectively. The entire system is trained for 60K steps using Adam optimizer (Kingma and Ba, 2014), where the learning rate is warmed up to a peak of 0.001 for the first 20K updates and then linearly decayed. The finetuning process takes ∼ 1.4 days on 4 NVIDIA-V100-32GB GPUs. Inference. No language model is used during inference. We employ beam search for decoding, where the beam width and length penalty are set to 50 and 1 respectively. All the hyper-parameters in our systems are tuned on validation set. Since our experimental results are quite stable, a single run is performed for each reported result.

heading: A.6 Baselines
text: In this section, we describe the baselines for comparison.
• TM-seq2seq (Afouras et al., 2018a): TMseq2seq proposes a Transformer-based AVSR system to model the A-V features separately and then attentively fuse them for decoding, and uses sequence-to-sequence loss (Watanabe et al., 2017) as training criterion.
• TM-CTC (Afouras et al., 2018a): TM-CTC shares the same architecture with TM-seq2seq, but uses CTC loss (Graves et al., 2006) as training criterion.
• Hyb-RNN (Petridis et al., 2018): Hyb-RNN proposes a RNN-based AVSR model with hybrid seq2seq/CTC loss (Watanabe et al., 2017), where the A-V features are encoded separately and then concatenated for decoding.
• RNN-T (Makino et al., 2019): RNN-T adopts the popular recurrent neural network transducer (Graves, 2012) for AVSR task, where the audio and visual features are concatenated before fed into the encoder.
• EG-seq2seq (Xu et al., 2020): EG-seq2seq builds a joint audio enhancement and multimodal speech recognition system based on the element-wise attention gated recurrent unit (Zhang et al., 2019), where the A-V features are concatenated before decoding.
• LF-MMI TDNN (Yu et al., 2020): LF-MMI TDNN proposes a joint audio-visual speech separation and recognition system based on time-delay neural network (TDNN), where the A-V features are concatenated before fed into the recognition network.
• Hyb-Conformer (Ma et al., 2021): HybConformer proposes a Conformer-based (Gulati et al., 2020) AVSR system with hybrid seq2seq/CTC loss, where the A-V input streams are first encoded separately and then concatenated for decoding.
• MoCo+wav2vec (Pan et al., 2022): MoCo+wav2vec employs self-supervised pre-trained audio and visual front-ends, i.e., wav2vec 2.0 (Baevski et al., 2020) and MoCo v2 (Chen et al., 2020), to generate better audio-visual features for fusion and decoding.
• AV-HuBERT (Shi et al., 2022a,b): AVHuBERT employs self-supervised learning to capture deep A-V contextual information, where the A-V features are masked and concatenated before fed into Transformer encoder to calculate masked-prediction loss for pretraining, and cross-entropy based sequence-tosequence loss is used for finetuning.
• u-HuBERT (Hsu and Shi, 2022): u-HuBERT extends AV-HuBERT to a unified framework of audio-visual and audio-only pre-training.
• Distill-Pretrain (Ma et al., 2022): DistillPretrain proposes a Conformer-based VSR framework with additional distillation from pre-trained ASR and VSR models.
ACL 2023 Responsible NLP Checklist

heading: A For every submission:
text: 3 A1. Did you describe the limitations of your work?
Limitations is after conclusion without section number
A2. Did you discuss any potential risks of your work? Not applicable. Left blank.
3 A3. Do the abstract and introduction summarize the paper’s main claims? Section 1
7 A4. Have you used AI writing assistants when working on this paper? Left blank.
B 7 Did you use or create scientific artifacts? Left blank.
B1. Did you cite the creators of artifacts you used? No response.
B2. Did you discuss the license or terms for use and / or distribution of any artifacts? No response.
B3. Did you discuss if your use of existing artifact(s) was consistent with their intended use, provided that it was specified? For the artifacts you create, do you specify intended use and whether that is compatible with the original access conditions (in particular, derivatives of data accessed for research purposes should not be used outside of research contexts)? No response.
B4. Did you discuss the steps taken to check whether the data that was collected / used contains any information that names or uniquely identifies individual people or offensive content, and the steps taken to protect / anonymize it? No response.
B5. Did you provide documentation of the artifacts, e.g., coverage of domains, languages, and linguistic phenomena, demographic groups represented, etc.? No response.
B6. Did you report relevant statistics like the number of examples, details of train / test / dev splits, etc. for the data that you used / created? Even for commonly-used benchmark datasets, include the number of examples in train / validation / test splits, as these provide necessary context for a reader to understand experimental results. For example, small differences in accuracy on large test sets may be significant, while on small test sets they may not be. No response.
C 3 Did you run computational experiments? Section 4
3 C1. Did you report the number of parameters in the models used, the total computational budget (e.g., GPU hours), and computing infrastructure used? Section 4 and Appendix A
The Responsible NLP Checklist used at ACL 2023 is adopted from NAACL 2022, with the addition of a question on AI writing assistance.
3 C2. Did you discuss the experimental setup, including hyperparameter search and best-found hyperparameter values? Section 4 and Appendix A
C3. Did you report descriptive statistics about your results (e.g., error bars around results, summary statistics from sets of experiments), and is it transparent whether you are reporting the max, mean, etc. or just a single run? Not applicable. Left blank.
3 C4. If you used existing packages (e.g., for preprocessing, for normalization, or for evaluation), did you report the implementation, model, and parameter settings used (e.g., NLTK, Spacy, ROUGE, etc.)? Appendix A
D 7 Did you use human annotators (e.g., crowdworkers) or research with human participants? Left blank.
D1. Did you report the full text of instructions given to participants, including e.g., screenshots, disclaimers of any risks to participants or annotators, etc.? No response.
D2. Did you report information about how you recruited (e.g., crowdsourcing platform, students) and paid participants, and discuss if such payment is adequate given the participants’ demographic (e.g., country of residence)? No response.
D3. Did you discuss whether and how consent was obtained from people whose data you’re using/curating? For example, if you collected data via crowdsourcing, did your instructions to crowdworkers explain how the data would be used? No response.
D4. Was the data collection protocol approved (or determined exempt) by an ethics review board? No response.
D5. Did you report the basic demographic and geographic characteristics of the annotator population that is the source of the data? No response.

heading: Future Work
text: In future, we may include the entire MIR-GAN into self-supervised learning scheme, together with the adversarial learning to refine better multimodal representations