On Robustness to Missing Video for Audiovisual Speech Recognition
Authors: Oscar Chang
Otavio Braga
Hank Liao
Dmitriy Serdyuk
Olivier Siohan
Venue: Accepted by TMLR ()

Abstract: It has been shown that learning audiovisual features can lead to improved speech recognition performance over audio-only features, especially for noisy speech. However, in many common applications, the visual features are partially or entirely missing, e.g. the speaker might move off screen. Multi-modal models need to be robust: missing video frames should not degrade the performance of an audiovisual model to be worse than that of a single-modality audio-only model. While there have been many attempts at building robust models, there is little consensus on how robustness should be evaluated. To address this, we introduce a framework that allows claims about robustness to be evaluated in a precise and testable way. We also conduct a systematic empirical study of the robustness of common audiovisual speech recognition architectures on a range of acoustic noise conditions and test suites. Finally, we show that an architecture-agnostic solution based on cascades can consistently achieve robustness to missing video, even in settings where existing techniques for robustness like dropout fall short.

Published in Transactions on Machine Learning Research (08/2022)
On Robustness to Missing Video for Audiovisual Speech Recognition
Oscar Chang Google LLC, USA
oscarchang@ggogge.com
Otavio Braga Google LLC, USA
obragaa@google.coo com
Hank Liao Google LLC, USA
hankliao @atiiiia@@goggeeccooo com
Dmitriy Serdyuk Google LLC, USA
dserdgub@@google.com
Olivier Siohan Google LLC, USA
siohan wwwooalgg......ommmmmmm com
Reviewed on OpenReview: https: // opt.r/veeerreeeeee.em net/ forum? id- fXorezbDu0
Abstract
has been shown that learning audiovisual features can lead to improved speech recognition performance over audio-only features, ospecially for noisy speech. However, in many common applications, the visual features are partially entirely missing, e.g. the speaker might move off screen. Multi-modal models need to be robust: missing video frames should not degrade the performance of an audiovisual model to be worse than that of a single-modality audio- only model. While there have been many attempts at building robust models, there is little consensus on how robustness should evaluated. To address this, we introduce a framework that allows claims about robustness be evaluated a procise and testable way. We also conduct & systematic empirical study of the robustness of common audiovisual speech recognition architectures
range of acoustic noise conditions and test suites. Finally, we show that an architecture-ggnosie solution based cascades can consistently achieve robustness to missing video, even in settings where existing techniques for robustness like dropout fall short.
1 Introduction
Learning from multiple modalities using large-scale datasets has increasingly been shown to produce stronger representations over those learned from a single modality. Such approaches have led to state-of-tho-at performance on numerous tasks in computer vision, natural language processing, and speech recognition (Radford et al., 2021 Ramesh et al. 2021 Yuan et al. 2021 Shi et al. 2022p. As multi-modal lcarning becomes more popular, is paramount that should be developed and deployed in trustworthy manner. This means that multi-moda systems have to be architected in ways that not just leverage features from additional modalities when they are present, but are also robust to missing features from these modalities when they are absent.
In this paper, we study the problem of building audiovisual automatic speech recognition (ASR) models that are robust to missing vidoo. This problem is decidedly asymmetric: only robustness to missing video, and not audio, is desired. This because practical ASR applications, which makes only the audio, and not video, indispensable (Serdyuk et al. 2021
lip-reading models still are not performant enough for many

---

Published in Transactions on Machine Learning Research (08/2022)
Table 1: Examples of prior work in AV ASR. The second column is a model trained and tested on AV, the third column trained on AV and tested on AO, and the fourth column trained on AO and tested on AO. A robust model should show ascending numbers from left to right. s_ means that this information was not provided by the prior work.
PRIOR WORK
Metric(Mav, Testav)
Metric[Mav Testao) Metric(Mao, Testao)
ROBUST
CHUNG ET AL.
22017
13.9 WER
17.7 WER
ZHOU ET AL. 2019)
9.07 CER
10.33 CER
SHI ET AL. [2022]
1.3 WER
1.5 WER
MAKINO ET AL.
(2019)
20.5 WER
24.0 WER
21.5 WER
ZHANG AL.
2019
26.2 PER
71.8 PER
35.8 PER
OuR WORK
26.12 WER
31.08 WER
33.54 WER
Audiovisual (AV) models have consistently achieved ASR performance superior to audio-only (AO) ones (Afouras et al.) 2018 Petridis et al., 2018: Ma et al.] 2021b). with especially dramatic gains for noisy or overlapping speech (Chung et al.) 2017 Abdelaziz et al.) 2017 Rose et al., 2021). But it is common for the video of the speaker to be partially entirely missing in typical ASR applications like providing closed captions for online inelings. For example, the speaker might move off screen, the camera can be turned off, the speaker will occasionally be occluded by other on-screen objects or changes in lighting conditions, etc.
If missing modalities can degrade the performance of a multi-modal model to be worse than that of a single- modality model, then the whole raison d'être for multi-modal learning in the first. place becomes questionable. This motivates our goal to build a robust audiovisual model that accords with the following intuition: Intuition 1. A model robust to missing video if additional video information at either training or test time can only help, and not hurrt, its performance.
The missing modality problem has received significant attention in the multi-modal learning community. Specifically, in the domain of audiovisual learning, recent years have seen determined efforts to build robust models for tasks including but not limited to: speech recognition (Makino ct al. 2019 Zhhagg et al. 2019
Zhou et al. 2019). expression recognition (Parthasarathy & Sundaram 2020), event localization (Xuan et al.
2020 voice activity detection [Tao) 2018 Hou et al.] (201)), video classification (Nagrani et al. 2021 ), and
speech enhancement (Gogate et al. 2021 ). Other studies of robustness in multi-modal learning include
Morgado et al. (2021) and Han et al. (2022).
Despite the intense interest in the missing modality problem in the audiovisual and ASR literature, there has surprisingly been no consensus how Intuition [ ought to be translated into concrete, testable claims. For example. Chung et al. (2017); Parthasarathy & Sundaram (2020); Xuan et al. (2020) show that when trained on AV data, their models do better when tested on AV compared to AO data. Afouras et all (2018); Zhou ct al. (2019); Xu et al. (2020); Ma et al. (2021b) show instcad that their proposed mothods yield better performance when trained and tested on AV data compared to when trained and tested on AO data. And Ngiam et al.] (2011): ;: Shii al. (2022) show that training on AV instead of AO data yields better test performance on AO. All these different tests capture disjoint aspects of model robustness that are neither equivalent to nor superset of one another. By testing only one of these aspects, existing work fails to ascertain if their models are truly robust. In fact, on the rare occasion that multiple aspects of robustness are tested, they are found to be robust with respect to one test, but not another (cf. rows 4 and 5 of Table Dh. Morcover, it is unclear that even the combination of the abovo-mentioned tests suffices to cnsure that a model is robust enough to be deployed in a real-world production environment.
1.1 Our Contributions
In response to this uncertainty, we make the following three salient contributions.
a) Robustness Framework Existing robustness criteria in the literature are inadequate because they try reduce robustness to single numerical comparison. In practice however, there are distinct aspects of robustness that individual ad-hoc comparisons fail to capture. To address this, we propose a mathematical
2

---

Published in Transactions on Machine Learning Research (08/2022)
Joiner
Joiner
Audiovisual Model
Language Model
Audiovisual Model
Language Model
Audio Model
Video Frames
Labels
Audio Frames
Video Frames
Labels
Audio Frames
a)
Audiovisual RNN-T
b)
Cascaded Audiovisual RNN-T
Figure 1: a) An audiovisual model trained with the RNN--T loss. b) A cascaded audiovisual model. The dashed lines indicate different routes the imput can take: the AO path, which goes through the audio model, and the AV path, which goes through the audiovisual model.
framework based on order theory that defines what it means for model to be robust to missing video under wide variety of scttings including missing vidco at training, testing, or even partially missing video. Our framework is motivated by the key observation that seltings with varying amounts of video information can be put in a partial order. A robust model is one that respects this order: as the amount of video information increases, the performance of a robust model should increase as well. Even though there may be an exponential number of test conditions (since each video frame can either be present or absent). we provide useful simplifications that allow for practical simulations of distinct scenarios of missing video. This is critical because for the same quantity of missing video frames, the manner in which the frames are dropped can significantly affect the degradation caused by the missing video. For example, in the case of AV ASR, dropping every second video frame causes only a slight degradation compared to dropping half the video frames in a contiguous segment. By showing how claims about robustness can be made precise and explicitly testable, our framework is a contribution not just to the AV ASR literature, but to the audiovisual and multi-modal learning ones as well.
b) Empirical Results for Existing Robustness Techniques on Different Architectures We con- duct a comprchensive empirical study of the robustness of common audiovisual speech rocognition architec- tures in the literature. Existing work recognizes that the solution to achieving robustness to missing video at test time is to expose the model to a similar condition at training time (to close the distribution gap between training and test), and advocates for a dropout-based approach (Chung et al. 2017; Makino et al. 2019 Zhang et al., 2019). Randomly dropping the vidco or an individual vidco frame at training time can be under- stood as an implicit. ensemble method that allows a single model to sample from 2 or 2number of frames possible
missing video settings, and learn from all of them. However, the previous works of [Makino et al. (2019) and Zhang et al. (2019) showed that training with dropout caused big degradation and feedforward sequential memory networks respectively (cf. rows 4 and 5 of Table 1. We replicate Makino (et al. (2019)'s 's findings for LSTMs on noisy 0db AO test set: training an 8 layer LSTM on AO resulted in 46.38 WER, but on the same test set, training the same model on AV actually resulted in 51.53 WER, which is am 11% increase over training on AO. Using video dropout only partially rectifies the problem, yielding 48.13 WER, which still a 3.8% increase over training on AO. These results violate Intuition I. since the addi- tion of vidco information at training time, whether dropout is used or not, actually resulted in worse test per- formance on AO. Surprisingly, we found that unlike prior architectures in the literature, conformers can atlain robustness with the use of dropout. This is fortuitous and notable empirical finding, because state-of-the- art architectures in both AO and AV ASR are based on conformers (Gulati et al. 2020 Ma et al. 2021a
an AO test set for LSTMs
c) Cascaded Audiovisual Models robustness is needed. It may not always be practical to use conformers due to their exorbitant memory requirements. Besides the choice of architecture, the success of dropout in producing a robust model also seems to hinge on ad-hoc empirical factors like noise conditions or the quality of features in the training and test datasets.
In general, a principled and architecture-agnosti
approach to
3

---

Published in Transactions on Machine Learning Research (08/2022)
Fortunately, the asymmetry of our problem (only robustness to missing video, and not missing audio, is desired) presents a unique opportunity. When the video is absent, a simple way to guarantee the robustness of an AV model is to have it produce the same predictions as an AO model. But when the video is present, the model can improve its predictions by fusing the acoustic representations with the visual oncs. This contral insight motivates our proposal of a cascaded audiovisual model: stack an AV model on top of an AO model, and route inputs via the AO path or the AV path (cf. Figure [D). There is a wealth of litcrature that points to the architecture-agnottr nature of cascades: LSTMs and Conformers for ASR in Narayanan et al. (2021) LSTMs and Transformers for ASR in Shi et al. (2020), and ViT, EfficientNet, ResNet, MobileNet and X3D for computer vision in Wang et al. (2020). We confirm that this finding also applies to the robustness problem by conducting comprehensive experiments spanning the vast majority of architectures used in AV ASR: using LSTMs and Conformers (with Transformers omilted because they are roughly convolution-free conformers) for the encoder, as well as both concatenation- proposed cascaded model was found be robust under a wide variety of architoctural combinations (for example when the base encoder and cascaded encoder were of different architectures) on range of test sets and noise conditions, even when existing techniques like dropout fail to achieve robustness. While cascades are not new, the proposal to use them our work, variants of dropout have been the only proposed technique for robustness, even when it had already been noted in Makino et al. (2019) for example that dropout did not achieve robustness in some set tings.
and attention- based audiovisual fusion methods. The
solution for robustness is novel and significant given that prior to
Besides being conceptually simple and architecture-inee innprterrrnideeeedeetttt cascaded models also enjoy numerous other advantages: they can be trained in one pass, are applicable streamable ASR models, and provide interpretable representations for the different modalities. The simplicity of cascades belies the complexity of the representation learning they do. We observed that jointly training both the AO and AV parts of the cascaded model yielded superior performance on both AO and AV test sets compared to first training the AO part, freezing its weights, and then training the cascaded AV part (even though the latter takes twice as much training time). Interestingly, we also saw that robustness was consistently achieved by cascading the entire video, but not cascading a partial collection of individual video frames, even though the test suites mostly contained partially, not entirely, missing video. Given the relative scarcity of high quality AV training data compared AO, cascading an AV model over a frozen AO model is an embarrassingly simple recipe for turning an arbitrary pre-trained AO model into robust AV model, thus opening up the wide gamut of existing AO ASR literature to the promise of audiovisual learning.
1.2 Organization for the Rest of the Paper
The remainder of the paper is organized as follows: Section provides an overview of audiovisual speech recognition. Section 3 introduces dropout and cascades. Section 4 reviews prior claims to robustness in the literature, and defines a new mathematical framework for reasoning about robustness. Scction 3 presents our experimental setup, and Section 0 presents the results of those experiments. Section acknowledges the limitations of our work. Finally, Section S concludes the paper.
2 Audiovisual Speech Recognition Methodology
Audiovisual speech recognition is the task of transcribing an audiovisual clip of speech, also known as an utterance, into text. Formally, we define an andiovisual speech recognition model M: A, V + Y as a function that consumes an audio input of waveforms or spectrograms and video input of mouth or face tracks, and produces a natural language transcript of what was said. The audio input a is a soquence of na acoustic frames of dimension d& that is represented by a real-valued tensor in Ru..d., The video input sequence of n. visual frames of dimension dy that is represented by real-valued tensor in Re..dy, The text output f E Y is a sequence of ny onc-hot vectors of dimension dy (vocabulary size) that is represented by a real-valued tensor in dy.,, Our formulation is general, because all existing AO ASR models can be considered as special cases of AV ASR models that do not use the video input.
V is a
For given data point (a, v), we can assume R2 = nyy bocause synchronizing the audio and vidco features is a necessary pro-psscss step for AV ASR (Chung et al., 2017 Afouras et al. 2018 Makino et al.) 2019h. The two modalities and are typically combined on frame level to form input sequence xr,
4

---

Published in Transactions on Machine Learning Research (08/2022)
i.. E [u] 2, Fusela, U2), with concatenation (CAT) being the most common method of fusion in the literature and cross-modal attention (CM) being the alternative (Wei et al. 2020)
2.1 Training: RNN-T
The RNN-T loss has become popular approach for training deep ASR models, because it is streamable unlike soquence-o--equeene losses, and allows the model to produce output tokens conditionally dependent on the history of provious tokens unlike CTC or the cross-entropy loss. can be written as follows.
T+U II P( 2t, yo.. Bu--1); 2=1
CRNN-T
E
(1)
JEAANN-T (x.y)
where the alignments ARNN-T(2, y) refer to the set of all possible sequences of T blanks and U labels.
An audiovisual speech model is generally factored into two separate components: language model (also called the decoder) and an audiovisual model (also called the encoder). The RNN-T loss is then computed from these two components using a joiner (cf. Figure 1.). Given input sequence x, the probability of the transcript can be calculated as follows.
P(ylx) (oiner(LM(y)) Encoder(a, v)),
(2)
Encoder(a, v) AVM AVM(Puse(a,z)).
We refer the interested reader to Graves 2012 He et al. 2019) for more information on the RNN-T loss.
For our study on robustness, we assume that we have a parallel corpus AV data, i.c. both the audio and video are present every training data point. Tf the model M given full access to the training data, we refer to the trained model as MAv. If M is given access to only the audio portion of the data, we refer the trained model Mao, and call its test performance the AO Bascline of the model.
2.2 Testing: Word Error Rate
The most common metric for evaluating ASR performance is the word error rate. For given labeled data point (a, v, and model output M(a,v), v), the word error rate of ÿ is the ratio of substitutions, deletions, and insertions in ÿ to the number of words in WER(Ø, 3) == S+D+1 The word error rate of model M W
over given test distribution Test is WER(M, Test) == T) WER(M(a, v),
3 Techniques for Robustness
The encoder Equation 2 can be modified to expose the model to missing vidco during training time, so to prepare to handle missing video at test time.
3.1 Existing Work: Dropout
Makino al. (0119) proposed to randomly drop the entire video utterance (Dropout Utt), while Zhang et al. (2019) proposed to randomly drop each video frame (Dropout Frame). Dropout can be implemented by sampling Bernoulli random variables and replacing v in Equation 2 with v' at training time. For Dropout Ctt, Bernoulli is sampled per utterance, while for Dropout Frame, a Bernoulli is sampled per frame.
Utt: of == 20;;,
2 ~ Bernoullipp).
(3)
Frame: v/ = 2,14,
2, Bernoullípp).
(4)
There are also proposals apply dropout simultaneously both the audio and video features at the utterance level (AV Dropout Utt) (Chung el. al. 2017 Shi et al. 2022
(a', of) = 12-0 v.) + (a; 0) + (0,
Multinomiaa(p.g.7))
(5)
5

---

Published in Transactions on Machine Learning Research (08/2022)
While dropout make model robust, we observed in the literature and on experiments that it can be sensitive to the choice of architecture or the specific test sets used. This is because training a mixture of AV and AO inputs does not guarantee that the architecture is able to automatically disentangle the AV and AO representations.
3.2 Proposed Method: Cascades
Instead of putting this burden on the model's architecture, we introduce an architecture-agnostic method that can benefit from the vidco information when it there, and gracefully degrade to the audio-only case when it is not. The basic idea is split the model explicitly into an acoustic model (AM) and an audiovisual model (AVM), and cascade the AVM on top of the AM (c.f. Figure [p). For each frame of the input, the model routes it to the AV path if the video frame is present, and to the AO path if not.
AM((;) Cascade(a, v), = AVM(Fusse (.sa(( v.))
0/2 = 0, hewwiis
(6)
Suppose that the cascaded model is trained in two passes (Two-Pass). On the first pass, no video information is made available, i.c. all inputs are routed to the AO path. On the second pass, the entire model except for the AVM is frozen, and all the video information is made available, i.c. inputs are routed to the AV path. Because the parts of the model that activate when it sees AO data are frozen, the cascaded model is guaranteed to never perform worse than its AO baseline regardless of its specific architecture. In fact, as we will show in our experiments, two passes are not neoded, and we can train robust model one pass by stochastically routing the inputs between the AO and AV paths. Like dropout, the stochastic selection of routes done by randomly dropping either the video (Cascade Utt) the video frames (Cascade Frame). Unlike dropout, cascades use the AVM if and only if the video is present, thus explicitly disentangling the AV from the AO representations.
4 Robustness Framework
We propose the use of order theory (Davey & Priestley 2002 as suitable mathematical language for evaluating robustness in rigorous way. By abstracting away absolute numbers, order theory allows claims about robustness to be formalized as statements about the relative ranking of given model's performance under different conditions.
4.1 Technical Preliminaries
To begin, we state the definition of a poset, and show how we can make comparisons between pairs of real vectors, vector-valued random variables, and test distributions.
Definition 4.1. A poset is set P equipped with binary relation & that is i) reflexive: Vp E P p.
ii) anti-symmetric: VpI,P2 E P.Pi < P2 and P2 I = pr P2, and iii) transitive: Vpi, P2. E P. P2 and P2 S P3 = P1 S P3-
Definition 4.2. For real vectors and b, if all the elements in a less than or equals to all the elements in b, i.e. Vi E [1,m],a a; < b, Definition 4.3. For veclor-valued random variables and B, A B i[ S E[B).
To model missing video frames, we can encode them using a (randomly distributed) binary mask p. The comparison between two test distributions can then be reduced to comparing the missing video probabilities for each frame.
Definition 4.4. For test distribution Test V,Y, and pair of real vectors (or vector-valued random variables) Pr. P2, (A, Pr V.Y) < (A,p V,Y) if p1 & P2.
Proposition 4.5. poset of test distributions can be constructed from an underlying poset P as follows: C(P) = [(A,p- v,Y))pEP.

---

Published in Transactions on Machine Learning Research (08/2022)
4.2 Prior Work
After careful survey of the AV (ASR) literature, we found that there are at least three distinct claims made about AV models, as exemplified by the first three rows of Table [ We explain why they are not sufficient claims to robustness.
For test distribution (A, V,Y), let Testav == Y) denote testing in the presence of the entire video, and Testao == (A,0, Y) denote testing in its absence. The first claim is that Metric(Mavs Testav) S Melric(Mav. TesL e.g. Chung et al. (2017); Parthasarathy & Sundaram (2020); Xuan et al. (2020). Notice that this can be true if only for the trivial reason that testing MMA on out-of-distributii data Testao leads to worse generalization. The second claim is that Metric(MAv. Testav) S Metric(Mao. Testao),
e.g. Afouras et al. 20188; ; Zhou et al. (2019) Xu et al. (2020): Ma et al. (2021b). This is not a sufficient claim to robustness, because it can be true even if the model suffers a catastrophic degradation in performance when trained on AV but tested on AO, as the case for example in rows 4 and of Table [ The third claim is that Metric(Mav. TestAo)= I Metric(Maa. Testao), c-g. Nggam et al. (2011); [Shi et al. (2022). This claim does not articulate the possibility of improved performance if the video modality was present. at. test time.
One way to formalize robustness might be to combine the above three claims into Metrie(Mav. ) S Mevric[Mav. Testao) s Metric(Mao. Testao). But this is also not an adequate definition that captures real-world settings where the visual modality is only partly missing, i.e. some video frames are dropped, but not all of them. In what follows, we present a comprehensive framework for evaluating robustness, and show that the combination of the three claims is a special case of robustness to Tutt.
4.3 Definition of Robustness
Intuition 1 requires that an audiovisual model not do worse than an audio-only model, given additional visual information at training or test time. This motivates the following definition of robustness.
Definition 4.6. A model M is robust to poset of test distributions T if it has both
1) Train-Time Robustness: VT E T, WER(MAv,T) S WER(MAO. T), and
2) Test-Time Robustness: VT,, T3 E T, T; WER(MAv T.) < WE).
Because each of the two properties is defined with respect to some set of testing conditions, it follows that robustness is not a universal property of the model, but rather statement about its performance a given test suite. This key feature of our definition allows it to be reified into concrete, empirically testable claims on specific test sets.
An important detail our definition is that there is an asymmetry: train-time robustness involves a coarse comparison between just Mav and Mao, while test-time robustness allows for very fine-grained comparisons depending on the size of the poset. This reflects both theoretical and practical requirements. At training time, methods like dropout intentionally discard some video information for the purpose of simulating test time conditions, but at test time, it is imperative that all the available video information is used. Thus, it is not meaningful to distinguish between training methods that use different amounts of video information (e.g. dropout with p = 0.5 instead of p = 0.25), but absolutely essential to distinguish between the performance of a model tested with half of the vidco frames missing versus a quarter of the frames missing. Furthermore, training a model is significantly more computationally demanding than running inference, so is necessary that the number of comparisons needed evaluate train-time robustness be lot smaller than what needed to evaluate test-time robustness.
Our definition also implies the following corollary, which shows why the two properties are not merely necessary, but also jointly sufficient by spanning the entire space of training and test conditions.
Corollary 4.7. Let M be a model robust to T. Then, VT, E T,T. S TJ = WER(MAV, T.) < WER(MAo: Tz)-
7

---

Published in Transactions on Machine Learning Research (08/2022)
4.4 Test Suites of Missing Video
We show how our proposed definition can be used to ascertain model robustness under distinct setlings of missing video. Let n be the number of video frames in a speech utterance. The naive approach is to test all combinations of the presence or absence of every vidco frame.
Test Suite 4.8. Trrme :- 1("),
But this is expensive to run, due to the exponential number of test distributions. Luckily, we can dramatically reduce the number of tests by focusing on pragmatic scenarios common to AV ASR like randomly dropped video, video dropped in contiguous way, and video dropped at a constant rate. We use a running example of transcribing an online meeting to motivate and illustrate the following test suites.
The most common cause of missing video in online meetings is when the user turns off their camera.
Test Suite 4.9. Tast == C({0", 1*)).
We can generalize Tute to UBt to capture the scenario where the user turns their camera on with probability r. Test Suite 4.10. Let BerUtt(r) a vector-valued random variable parameterized by a sealar r E [0, 1].
P(BerUl(r)- = -1") = P(BerUtt(r) = = 1
(7)
Therti((r)= == C((BerUU(())rER). In our experiments, R = 10, 2.
Another cause of missing video in online meetings is an unreliable internet connection. The more unreliable it is, the more likely that internet packets for individual video frames will be dropped. We can simulate this condition using i.idd. Bernoulli random variables for each frame.
Test Suite 4.11. Let BerFrame(s) be a vector-valued random variable parameterized by a scalar s E [0, 1].
Vi E nI, P(BerFramel s); = 1) = 8, P P(BerFrame(s) ()); = 0) = 8.
TherFrame C((BBrrrame(s))ess) In our experiments, S = (0. 2:
3
Vidco frames can also be dropped in contiguous segment from the start, in the middle, or at the end, depending on when the user decides to move off screen and when they re-enter. We can simulate different. amounts of missing video in each case with a deterministic binary mask.
Test Suite 4.12. Let Oasb denote a binary vector where the ith element is iff i 1 n]. (oon(A, (A, = ((aas)((a (ab)))(A))) In our experiments, Tstert == Teon (1 0), (0, 4), (0, (), (0, (0. (0, 1))), Tend == Teaal ((0, 1), (1, 1), 1), 1), (1, Triid = Tea (e( 2 (3. 3), G () (0,
If the hardware has competing demands for its compute, it might decode the video at a smaller frame rate, which leads to vidco frames being dropped at a constant rate.
Test Suite 4.13. Let 3r denote binary vector where the ith element is 0 iff is a multiple of x. Traso(K) = C((BAREE)). In our experiments, K = {0, T23 32
1}.
Finally, we can unify all these test suites.
Test Suite 4.14. Tast := TberUts U TherProme U Tatart U Tmid U Tend U Trute-
This list of test suites (visualized in Figure 2 is not exhaustive, but serves a proof of concept for how exlensible our framework is - by changing the poset of test. distributions, we change the kind of robustness that we are testing. Our proposed test suites are not only qualitatively different, they also cover both easy and hard cases for ASR. For example, dropping every second frame makes for an casy test suite, while dropping a contiguous segment of half the frames makes for a challenging one. This is intuitive because for given word, having partial visual context. for it makes the ASR task significantly easier than having visual context.
8

---

Published in Transactions on Machine Learning Research (08/2022)
U1
U2
U3
U4
TberUu TheFFraee
Tstant
Tmid
Tend
Trate
Figure 2: Dropping 50% of the frames under different test suites.
5
Experimental Setup
Audiovisual Features The acoustic features are computed as log-scaled mel spectrograms sampled 16kHz using a Hann window of size 25ms and step 10ms, with every three consecutive spectrograms stacked into one audio frame with dimension 240. At training time, we apply multi-style training for data augmentation (Cui et al. 2015). The visual features are computed by extracting mouth tracks from the video, re-sampling them at the frequency of the acoustic features (to ensure that both modalities have the same sequence length), cropping them to 32x32, and encoding them using a VGG into video frames of dimension 256. Because the visual features are extracted using an independent face tracker like in Afouras et al. (018); Makino et al. (2019), the AV ASR model knows whether the vidco frame is missing or not. The VGG not pre-trained frozen, and its parameters are learned during the training process. Our models use a sequence length of up to = 512, which corresponds to approximately 15 seconds of audiovisual speech.
Training Data We closcly adhere to the process outlined by Makino et al. (009)) to create largo-scale dataset containing 100, 000 hours of AV data from public YouTube videas. This is done by mining segments of videos where the force-aligned user uploaded transcript matches a production quality ASR system with high confidence. Then, SyncNet (Chung & Zisserman 2016) is applied as a filtering step to ensure that the video track also matches the audio track with high confidence. Unlike Makino et al. (2019), we synchronize the frame rate of the video to match the audio, instead of vice versa, to ensure that do not affect the AO bascline. As explained in Section
2.1
train models under two settings: MAo and MAV.
Test Data As discussed in Section 4.3 robustness should be treated not as universal property of model, but as an empirical claim about its performance on particular test set. We benchmark our models on a separate test set of YouTube videos that contains 550 hours of professionally transcribed audiovisual clips (27, 353 utterances and 342, 507 words), under varying amounts artificially added babble noise (clean, 20db, 10db, Odb) from the NoiseX corpus (Varga & Steeneken 1993). As explained in Section 4.4 we evaluate our models on six test suites: TherUtt TherFrame: Tstorts Tmidd Tend: Trate- Both the training and test data were collected by following the Google AI principles (goo).
Statistical Methodology Our study of robustness focuses on the relative ranking between the WER results of a given model under different test conditions, rather than the absolute WER numbers. That said, in practice, comparisons between any two results have to take into account statistical variance and uncertainty. To this end, we closely follow the prior AV ASR methodology of Makino et al. (2019) assembling a large- scale audiovisual dataset for training and calculating confidence intervals for the test WER. If either of two results falls within the other's 95% confidence interval, we consider them equivalent, otherwise, we consider one result better than the other.
Methods for Robustness We compare between the following methods: Audio Baseline, Vanilla, Vanilla (25I.), Cascade Utt, Dropout Ut., Cascade Frame, Dropout Frame, AV Dropout Utt, and Two-Pass. While Section g contains a self-contained description for most of these methods, we re-iterate them here for the reader's convenience.
Audio Baseline: The model is trained audio-only inputs.
9

---

Published in Transactions on Machine Learning Research (08/2022)
Vanilla:
The model is trained on audio-visual imputs.
Vanilla (25L): Same as the Vamilla model, but with 25 layers.
Cascade Utt: The model randomly drops the entire video. Audio-only inputs are routed to the audio encoder, while audio-visual inputs are routed to the audio-visual encoder.
Dropout Utt: The model randomly drops the entire video. Inputs with missing video are consumed by the model directly.
Cascade Frame: The model randomly drops each video frame. Audio-only inputs are routed to the audio encoder, while audio-visual inputs are routed to the audio-visual encoder.
Dropout Frame: The model randomly drops each video frame. Inputs with missing video are consumed by the model directly.
The model randomly drops the entire video. The model also randomly drops the entire audio. Inputs with missing audio or video are consumed by the model directly.
AV Dropout Ut::
Two-Pass: Same architecture as the Cascade Utt and Cascade Frame models, but the model is trained in two passes. On the first pass, audio-only imputs are consumed, and all model weights are trainable. On the second pass, audio-visual inputs are consumed, but only the andio-visual encoder is trainable.
Architectural Configurations All models use a two-layer bidirectional LSTM with hidden dimension 2048 for the decoder, and a one-layer MLP with hidden dimension 640 for the joiner. The decoder uses an English character-based vocabulary, with embedding dimension of 128 and a beam width of size 8. The architectural differences between the different models reside in the encoder.
The encoder is a conformer that uses concatenation to fuse the andiovisual features. For cascaded models, both the base and the cascaded encoder are conformers.
Conformer CAT:
Conformer CM: The encoder is a conformer that uses cross-modal attention, specifically the one used in Afouras et al. (2018), to fuse the audiovisual features. For cascaded models, both the base and the cascaded encoder are conformers.
Con-LSTM CAT: This is only used in the context of the cascaded model, with a conformer as the base encoder and an LSTM as the cascaded encoder. Concatenation is used to fuse the audiovisual features.
LSTM CAT: The encoder is an LSTM that uses concatenation to fuse the andiovisual features. For cascaded models, both the base and the cascaded encoder are LSTMss
LSTM-CON CAT: This only used in the context of the cascaded model, with an LSTM as the base encoder and conformer as the cascaded encoder. Concatenation is used to fuse the andiovisual features.
The conformer encoders are configured with 17 layers, full context attention, model dimension 512, 8 attention heads, convolutional kernel size 32, no dropout (this refers to regular neural network dropout instead of vidco dropout), and group normalization with 32 groups in the place of layer normalization. The Cascade Uu, Cascade Frame, and Two-Pass models use 17 conformer layers for the base AM and 8 conformer layers for the cascaded AVM. The Vanilla (25L.) model uses 25 conformer layers instcad of 17.
The LSTM encoders are configured with 8 bidirectional layers, model dimension 512 (for each direction), and weight normalization. The Cascade Utt model uses 8 LSTM layers for the base AM and 4 LSTM layers for the cascaded AVM.
Optimization All our models are trained in exactly the same way: Adam with Bn = 0.9, 32 = 0.97, batch size 4096, and learning rate 0.001 for a total of 500k steps with linear warmup in the first 10k steps and an exponential decay to the smaller learning rate of 0.0001 from steps 300k to 400k. Two-Pass uses the same optimization setup for both training passes.
10

---

Published in Transactions on Machine Learning Research (08/2022)
6
Results
To begin, walk the reader through examples in Table 2 to show how we establish the robustness of model. Cascade Utt is robust because the WER increases monotonically as more frames are dropped (Test- Time Robustness), while always staying below its AO baseline of 33.54 Sraen-Timee Robustness). On the other hand, the vanilla model is not robust, because when tested on the setting where all the frames are dropped, its WER of 35.51 exceeds its AO bascline, thus violating Train-Time Robustness. Dropout Frame is not robust, because it oblains lower WERs of 27.11 and 26.51, compared to 27.58, by dropping more frames at test time, which violates Test-Time Robustness.
Vanilla AV ASR Models are Not Robust The vanilla conformer and LSTM models were found to not be robust under most settings tested (cf. Table Figure 3 shows the WER of a vanilla model increasing as the number of missing video frames at test time increases, eventually doing worse than its AO baseline. This underscores why the study of robustness is necessary: we would like our AV models to always do at least as well as the AO models regardless of video availability at test time.
Type of Missing Video Affects Performance For the same quantity of missing video, the manner in which the frames are dropped can significantly affect how quickly the performance of a model degrades. In Figure 3 observe relationship between WER and amount of dropped frames that is linear for TherUa, Tstart, Tiid: 'Tend and convex for TherFrame: Trate- In other words, the model is fairly tolerant of frames being randomly dropped or dropped at a constant rate, but not when dropped in a contiguous segment. This accords with the intuition from our discussion in Section 44
For the utterance methods, we ran a hyper- ppper-parameter scarch over p E f0.25, 0.5, 0.75]. In every setting tested, we found that p = 0.25 was optimal for Cascade Utt, and p p= 0.5 was optimal for Dropout Utt, i.e. it was never the case that setting an alternative p led to robustness if this did not. This finding supports the motivation for the asymmetry in our definition of robustness in Section 4.3 it is not pertinent that p = 0.25, p = 0.5 were trained on less vidco information than p = 0.75, but noleworthy that they were better able to withstand the loss of video information at test time. For the frame methods, we scarched p E (0.1, 0.2), and found = 0.1 to be optimal for both Cascade Frame and Dropout Frame. For AV Dropout Utt, we found p = = = f.r = 4. which is used by Shi et al. (0022) to be optimal over p 3. q g.r r 1 which is used by Chung et al. (2017).
Tuning Sampling Hypor-Parametory
Cascade Utt is Consistently Robust Like Makino ct al.
(201 we saw that Dropout Utt can degrade the performance of an LSTM CAT to be worse than its AO baseline. This degradation is especially pronounced for Conformer CM (cf. Figure -. Across all the architectural configurations, four noise conditions, and six test suites, Cascade Utt was the only technique found to be consistently robust (cf. Table 3). This finding held even when the base and cascaded encoder are of different architoctures, like Con-LSTM CAT and LSTM- Con Cat, which points to the architecture-agnootic nature of cascaded models.
Dropout Utt Performs Well on Conformer CAT The main caveat to Cascade Utt is that while always robust, it does not always produce the lowest absolute WER. In Figure 4 see that when tested on full AV in the Odb setting, Cascade Utt uctually performs worse than Dropout Uit and the vanilla model for the concatenation models. This finding emphasizes that the evaluation of robustness, which consists of relative comparisons for a fixed model, is independent from the evaluation of absolute WER. It is thus fortuitous that Dropout Utt is both robust and has lower WER than the vanilla conformer CAT, which is a state-of-the-art architecture for AV ASR (a et al., 2021a
Frame Methods Underperform Utterance Methods In general, the frame methods are not as robust the utterance methods. This can perhaps be expected because training augmentations often do not generalize to unseen test corruptions (Mintun et al. 2021). For example, in Table 2 Cascade Frame underperforms its AO baseline. This might be because even if the probability of dropping each of 512 frames is high (90%), the model is nevertheless rarely exposed to an emply video at training time (0.9512 = 10-21).
Cascade Utt Achieves its Model Capacity Cascade Utt was found to have the same or strictly lower WER than Two-Pass for all test suites on all quantities of missing video tested. This means that independent of the capacity of the cascaded model to achieve robustness, it is also important to consider how we optimize
11

---

Published in Transactions on Machine Learning Research (08/2022)
it. Not only is it not necessary to achieve robustness by training in two passes, doing it in only pass (i.e. using half the resources) can actually result. in better absolute WER performance.
We train the cascaded model with just AO and AV data respectively to measure the capacity of the AM and AVM sub-models. In Figure D we note that Cascade Utt is able to achieve the capacity of both sub- models, and segue between them as frames are dropped during test time. For conformer CAT, a layer AM cascaded with an layer AVM performs equivalently on fuull AV data test time vanilla 25 layer AVM at all noise conditions except Odb. This implies that unlike adversarial robustness, which is gained at necessary cost model size performance clean data (Bubeck & Sellke 2021), robustness to missing video can be achieved via cascades without using any extra resources or trading off test performance on AV.
Table 2: Robustness to Trute for Odb. Columns three to seven represent the amount of dropped video frames. The WER results in bold cause the model to be considered non-robust.
ARCHITECTURE
METHOD
0
32
ROBUST
CONFORMER CAT
AUDIO BASELINE
33.51 + 0.43
33.51 + 0.43
33.54
0.43
33.54
0.43
33.51 + 0.43
CONFORMER CAT
VANILLA
24.96
1 0.35
24.95
1: 0.35
25.08 1 0.35
25.90 1 0.35
35.51 1 0.43
x
CONPORMER CAT
CASCADE UrT
26.12
0.36
26.25
1 0.36
26.72
0.37
28.73
0.39
31.08
0.40
CONFORMER CAT
DROPOUT UTT
24.33 1 0.34
24.36
1: 0.34
24.49
0.34
0.36 26.66
31.67 1 0.40
1:
34.17
CONFORMER CAT
DROPOUT FRAME
27.58
- 0.37
27.11 = 0.36
26.51 + 0.36
27.56 + 0.38
32.54 = 0.41
CONFORMER CAT
DROPOUT UrT
23.61
0.33
23. + 0.33
23.80 0.33
25.36
0.35
33.55 = 0.11
CONFORMER CAT
Two--Aas
30.06 0.42
30.11 + 0.42
30.47
0.43
31.91
+ 0.43
33.54 + 0.43
berUtt
T berFrame
Tstart
Trate
20
20
20
WER
19
19
18
18
18
0.5
1.0
0.0
0.5
0.5
1.0
0.0
0.5
Dropped Frames
Dropped
Dropped Frames
Dropped Frames
Figure The X-marked lines show the WER of vanilla 17 layer conccalenation--asel conformer model trained AV, and tested on 10db with different test suites. The dashed line denote the performance of the same model trained on AO, i.e. its AO baseline. The vanilla AV model performs worse as more video frames are dropped at test time, eventually doing worse than its AO baseline.
Conformer CAT
LSTM CAT
Conformer CM
55.0
35
WER
30
42.5
25
30
30.0
0.0
0.5
1.0
0.0
0.5
1.0
0.0
0.5
1.0
Dropped Fromess
Drop pod Rommss
Dropped Frames
Figure 4: The X-marked, square-marked, diamond-marked, and dashed lines respectively show the WERs for the vanilla model, Dropout Utt, Cascade Utt, and the AO baseline tested Odb using TherUtt across three architectures.
12

---

Published in Transactions on Machine Learning Research (08/2022)
Table 3: Robustness Tall across different experimental settings. Cascade Utt is the only method found be consistently robust across all settings tested.
ARCHITECTURE, METHOD
CLEAN
200B
10DB
ODB
CONFORMER CAT. VANILLA
x
X
X
CONPORMER CAT, Cascade Utt
CONFORMER CAT, DROPOUT UTT
CONFORMER CAT, CASCADE FRAME
X
CONPORMER CAT, DROPOUT
CONFORMER CAT, AV DROPOUT UTT
J
CONPORMER CAT, Two-PASS
CoN-LSTM CAT, Cascade Utt
V
V
LSTM CAT, VANILLA
X
LSTM CAT, Cascade Utt
LSTM CAT, DROPOUT UTT
LSTM-CON CAT, Cascade
CONFORMER CM. VANILA
X
CONPORMER CM. Cascade
CONPORMER CM, DROPOUT UTT
X
T berUtt
T berFrame
T Ttart
Trae
20
20
WER
19
18
18
18
18
0.0
0.5
1.0
0.0
0.5
1.0
0.0
0.5
1.0
0.0
0.5
1.0
Dropped Frames
Dropped
Dropped Frames
Dropped
Figure 5: The X-marked and diamond-marked lines respectively show the WER of vanilla 25 layer conformer AVM and 25 layer cascaded conformer model (17 AM AVM layers) tested on 10db. Both models use concatenation for audiovisual fusion. The top and bottom dashed lines respectively denote the performance of the cascaded model when all its inputs are forcefully routed to the AO and the AV paths during both training and testing.
7 Limitations of Our Work
There are several limitations of our work that should improved and addressed in future work. First, the proposed robustness framework only makes comparisons between the relative performance of model under different test settings. This ignores the absolute WER performance of given model, which is another important, although orthogonal, evaluation metric. Second, the framework makes binary evaluations: model is either robust not robust. A more general framework might be able to allow for more fine-grained evaluations that makes it easier to quantify and compare the robustness of a set of given models. A possible solution is to count the number test suites under which a given model is robust. Third, the current work only studies robustness lo missing video and not other common video corruptions like jitter and blur. Fourth, the cascade methods require knowledge that the video frame missing in order to route the inputs, while the dropout methods do not. Because the mouth tracks are extracted with an independent face tracker (as explained in Section 5). this information is made known to the AV ASR model. A possible solution is to integrate the face tracker into an end-to-end system that the model has to now also make decisions about whether the video frame is missing or not. Please see Appendix D for an extended discussion of this issue.
8 Conclusion
this paper, we developed the first principled framework for evaluating robustness to missing video for audiovisual speech recognition. By providing rigorous definition for robustness, our work fills a longstanding gap in the literature by making it easy for claims about robustness to be specified in an empirically testable way. Our extensive empirical experiments also show that cascaded models are reliable method to achieve
13

---

Published in Transactions on Machine Learning Research (08/2022)
robustness under wide variety of architectural configurations and test suites, including in settings where existing techniques like dropout fall short. We hope that our work will spur the development of more robust. models and techniques in the AV ASR and multi-modal learning communities.
14

---

Published in Transactions on Machine Learning Research (08/2022)
References
Artificial intelligence at google: Our principles.
Accessed: 2022-05-26.
Ahmed Hussen Abdelaziz et al. Nted-timit: A new database and baseline for noise-robust audio-visual speech recognition. In INTERSPEECH, pp. 3752-3756, 2017.
Triantafyllos Afouras, Joon Son Chung, Andrew Senior, Oriol Vinyals, and Andrew Zisserman. Deep audio- visual speech recognition. JEEE transactions pattern analysis and machine intelligence, 2018.
Sébastien Bubeck and Mark Sellke. A universal law of robustness via isoperimetry. ar preprinl arXiv:2105.128000 2021.
Joon Son Chung and Andrew Zisserman. Out of time: automated lip syne in the wild. In Asian conference on computer vision, pp. 251-2622 Springer, 2016.
Joon Son Chung, Andrew Senior, Oriol Vinyals, and Andrew Zisserman. Lip reading sentences in the wild. In 2017 IEEE Conference Computer Vision and Pattern Recognition (CVPR), pp.
TEEE, 2017.
Xiaodong Cui, Vaibhava Goel, and Brian Kingsbury. Data augmentation for deep neural network acoustic modeling. IEEE/ACM Transactions on Audio, Speech, and Language Processing, 23()):1469-1477 2015.
Brian A Davey and Hilary A Priestley. Intvoduction to lollices and order. Cambridge university press, 2002.
Mandar Gogate, Kia Dashtipour, and Amir Hussain. Towards robust real-time andio-visual speech enhance- ment. arXin preprint arXiv:21 12. 190600 2021.
Alex Graves. Sequence transduction with recurrent neural networks. arXiv preprint arXiv:1211.8711 2012.
Anmol Gulati, James Qin, Chung-Cheng Chiu, Niki Parmar, Yu Zhang, Jiahui Yu, Wei Han, Shibo Wang, Zhengdong Zhang, Yonghui Wu, et al. Conformer: Convolution-augnentee transformer for speech recognition. arXiv preprint 200.2000500100, 2020.
Haochen Han, Qinghua Zheng, Minnan Luo, Kaiyao Miao, Feng Tian, and Yan Chen. Noise-tolerant learning for audio-visual action recognition. orXiv preprint wxXiv:225.00611 2022.
Yanzhang He, Tara N Sainath, Rohit Prabhavalkar, Ian McGraw, Raziel Alvarez, Ding Zhao, David Rybach, Anjuli Kannan, Yonghui Wu, Ruoming Pang, et al. Streaming end-to-end speech recognition for mobile devices. In ICASSP (ICASSP), pp. 6381-6385. IEEE, 2019.
IEEE International Conference on Acoustics, Speech and Signal Processing
Yuanbo Hou, Zhesong Yu, Xia Liang, Xingjian Du, Bilci Zhu, Zejun Ma, and Dick Botteldooren. Attention- based cross-modal fusion for audio-visual voice activity detection in musical video streams. arXiv preprinl 0rXiv:2106.11411 2021.
Pingchuan Ma, Rodrigo Mira, Stavros Petridis, Björn W Schuller, and Maja Pantic. Lira: Learning visual speech representations from audio through superviiin.. arXie preprint Xrx:2:1106. 09171, 2021a.
Pingchuan Ma, Stavros Petridis, and Maja Pantic. End-to-end audio-visual speech recognition with conformers. In TCA (CASSP 2021-2021 TEFE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 7613 7617. IEEE, 2021b.
Takaki Makino, Hank Liao, Yannis Assael, Brendan Shillingford, Basilio Garcia, Otavio Braga, and Olivier Siohan. Recurrent neural network transducer for audio-visual speoch recognition. In 2019 IEEE automatic speech recognition and understonding workshop (ASRU), pp. 905 912. IEEE, 2019.
Eric Mintun, Alexander Kirillov, and Saining Xic. On interaction between augmentations and corruptions natural corruption robustness. arXiv preprint arXiv:2102. 11273. 2021.
Pedro Morgado, Ishan Misra, and Nuno Vasconcelos. Robust audio-visual instance discrimination. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 12934-12945, 2021.
15

---

Published in Transactions on Machine Learning Research (08/2022)
Arsha Nagrani, Shan Yang, Amurag Arnab, Aren Jansen, Cordelia Schmid, and Chen Sun. Attention bottlenecks for multimodal fusion. Advances in Neural Information Processing Systems, 34, 2021.
Arun Narayanan, Tara N Sainath, Ruoming Pang, Jiahui Yu, Chung-Cheng Chiu, Rohit Prabhavalkar, Ehsan Variani, and Trevor Strohman. Cascaded encoders for unifying streaming and non-streaming ast. In ICASSP 2021-2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 5629-562 5633. IEEE, 2021.
Jiquan Ngiam, Aditya Khosla, Mingyu Kim, Juhan Nam, Honglak Lee, and Andrew Y Ng. Multimodal deep learning. In ICML, 2011.
Srinivas Parthasarathy and Shiva Sundaram. Training strategies to handle missing modalities for audio-visual expression recognition. In Companion Publication of the 2020 International Conference on Multimodal Interaction, pp. 400 404, 2020.
Stavros Petridis, Themos Stafylakis, Pingehuan Ma, Feipeng Cai, Georgios Teimiropoulos, and Maja Pantic. Fad-to-end audiovisual speech recognition. Tn 2018 TEEE international conference acousties, speech and signal processing (ICASSP), pp. 6548 6552. IEEE, 2018
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. arXiv preprint AXx::2103.000222 2021.
Aditya Ramesh, Mikhail Pavlov, Gabriel Goh, Scott Gray, Chelsea Voss, Alec Radford, Mark Chen, and llya Sutskever. Zero-shot lext-lo-image generation. arXiv preprint XrXie:2102. 12092, 2021.
Richard Rose, Olivier Siohan, Anshuman Tripathi, and Olavio Braga. End-to-end audio-visual speech recognition for overlapping speech. Proc. Interspeech 2021, pp. 3016 3020,, 2021.
Dmitriy Serdyuk, Otavio Braga, and Olivier Siohan. Audio-visual speech recognition is worth 32x32x8 voxels. arXiv preprint arXiv:2109.009 09.09536, 2021.
Bowen Shi, Wei-Ning Hsu, Kushal Lakhotia, and Abdelrahman Mohamed. Learning audio-visual speech representation by masked multimodal cluster prediction. arXiv preprint 1rXim:2201.021:0 2022.
Jing Shi, Xuankai Chang, Pengcheng Guo, Shinji Watanabe, Yusuke Fujita, Jiaming Xu, Bo Xu, and Lei
Xie. Sequence to multi-sequence learning via conditional chain mapping for mixture signals. Advonces in Neural Information Processing Systems, 33:3735-3744 2020.
Fei Tao. Advances in Audiovisual Speech Processing for Robust Voice Activity Detection and Automatic Speech Recognition. PhD thesis, 2018
Andrew Varga and Herman JM Steeneken. Assessment for automatic speech recognition: Ii. noisex A database and an experiment to study the effect of additive noise on speech recognition systems. Speech communécation, 12(3)2247-251, 1993.
Xiaofang Wang, Dan Kondratyuk, Eric Christiansen, Kris M Kitani, Yair Alon, and Elad Eban. Wisdom of committees: An overlooked approach to faster and more accurate models, arXiv preprint w.Xiiv2012.01990. 2020.
Liangfa Wei, Jie Zhang, Junfong Hou, and Lirong Dai. Attentive fusion onhanced audio-visual encoding for transformer based robust speech recognition. In 2020 Asia-Paccfh Signal and Information Processing Association Annual Summit. and Conference (APSIPA ASC), pp. 638 638-643. IEFF, 2020.
Bo Xu, Cheng Lu, Yandong Guo, and Jacob Wang. Discriminative multi-modality speech recognition. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pottern Recognition, pp. -4442, 2020.
16

---

Published in Transactions on Machine Learning Research (08/2022)
Hanyu Xuan, Zhenyu Zhang, Shuo Chen, Jian Yang, and Yan Yan. Cross-modal attention network for temporal inconsistent audio-visual event localization. In Proceedings of the AAAT Conference on Arlificial Intelligence, volume 34, pp. 279 286, 2020.
Lu Yuan, Dongdong Chen, Yi-Ling Chen, Noel Codella, Xiyang Dai, Jianfeng Gao, Houdong Hu, Xuedong Huang, Boxin Li, Chunyuan Li, et al. Florence: A new foundation model for computer vision. arXiv preprinl ar Xiv:21111. 11432, 2021.
Shiliang Zhang, Ming Lei, Bin Ma, and Lei Xic. Robust audio-visual speech recognition using bimodal dfsmn with multi-condition training and dropout regularization. In ICASSP 2019-2011 IEEE International Conference Acoustics, Speech and Signal Processing (TCASSP), pp. 6570-6574. TEEE, 2019.
Pan Zhou, Wenwen Yang, Wei Chen, Yanfeng Wang, and Jia Jia. Modality attention for end-to-ent audio- visual speoch rocognition. In ICASSP 2019-2019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 6565 6569. IEEE, 2019.
17

---

Published in Transactions on Machine Learning Research (08/2022)
Appendix
A Overview
The appendix is organized as follows: Appendix B is broader impact statement. Appendix C states the source of the numbers in Table | Appendix D provides an in-depth discussion regarding the assumption in our work that the AV ASR model possesses knowledge that the video frames are missing. Appendix E documents our results in the main paper in a more comprehensive manner. In particular, we plot the graphs of our main experiments shown in part in Figures g to in their expanded form, and provided further analysis. We also document. the absolute WER numbers and their confidence intervals across our extensive experimental settings, in similar format to that of Table 2 in the main paper. Appendix F shows the results of additional experiments on the TED LRS3 datasct. In the Supplementary matcrials, we have attached an example script writlen in lensorflow code lo show how our robust ness test suites can be generated.
B Broader Impact Statement
By the nature of the dataset collection process, the performance of the model with under- representede groups is not being mcasured and that optimizing for the specific benchmark metrics proposed here may have unknown effects (either positive or negative) on the performance models within such groups.
C About Table 1
The first row comes from Table in Chung et al. (0017 ) The second row comes from Table in Zhou et al. (2019). The third row comes from Table in [Shi et al. (2022). The fourth row comes from Tables and in Makino et al. (2019). The fifth row comes from Tables and in Zhang al. (2019)
Knowledge of Missing Video Frames
In our paper, we assume that the AV ASR model knows whether a given video frame missing. This begs the question: since we know that there missing video, why not just trivially swap to an audio-only model in that case? A naive ensemble containing an audio-only model and an audio-visual model would be robust to missing video by construction. However, there are several issues with this approach. First, the naive ensemble would be significantly more computationally demanding than having cascaded model in terms of compute, memory, and disk. Second, the naive ensemble would not perform as well in the situation when there is partial vidco information available, since the audio model operates independently of the video model. It not feasible to build an ensemble of 2 models where n. the number frames to account for every missing video scenario. Third, a naive ensemble that does not share decoder state between the audio and audio-visual model would not be compatible with the streaming scenario. The proposed cascaded model can be used in the streaming scenario, because it shares decoder state via the stochastic routing of inputs.
18

---

Published in Transactions on Machine Learning Research (08/2022)
E
Extended Results
TerUtt
berframe
sstrrt
mid
Tend
rrate
Clean WER
17.5
17.5
17.5
17.5
17.5
17.5
17.0
17.0
17.0
17.0
17.0
17.0
0.0
0.5
1.0
0.0
0.5
1.0
0.0
0.5
1.0
0.5
1.0
0.0
0.5
1.0
0,0
0.5
1,0
Dropped Fromes
Drosped Frames
Dropoed Frames
Dropped Frames
Drosped
Dropped Frames
TberUtt
TberFrame
start
T mid
T Tend
T Trate
20db WER
18.0
18.0
18.0
18.0
18,0
17.5
17.5
17.5
17.5
17.5
175
17.0
17.0
17.0
17.0
17.0
17.0
1.0
0.5
0.5
Dropped Framess
Drosped Prames
Dropped Frames
Dropped Frames
Dropped Frames
Drapped Frames
TberUtt
Tberframe
Ttarrt
mid
T Tend
Trate
10db WER
20
20
20
20
20
20
18
18
18
18
18
18
1.0
1.0
0.5
0.5
0.5
Dropped
Drosped Frames
Dropped
Dropped
Drosped
Dropped
TberUtt
TberFrame
start
mid
Tend
Trate
0dl WER
35
35
35
35
35
30
30
30
30
30
0.0
0.5
1.0
0.0
0.5
1.0
0.0
0.5
1.0
0.0
0.5
1.0
0.0
0.5
1.0
0.0
0.5
1.0
Dropped Fromes
Drooped Fromes
Dropoed Frames
Dropped
Drogped
Dropped
Figure 6: Conformer CAT shown across four acoustic noise conditions and six different test suites. The X- marked, square-marked, diamond-marked, and dashed lines respectively show the WER for the vanilla model, Dropout Utt, Cascade Utt, and the AO baseline. The shaded area around each line denotes the model's 95% confidence interval.
Advantage of Vanilla Conformer CAT AV Models Varies Across Acoustic Noise Conditions that when all the frames are dropped, the vanilla model consistently underperforms its how noisy the audio is. For
In Figure f AO bascline. However, the point at which this happens varies depending example, for TherUtt, we observe the overwhelming advantage of using the visual modality in the 0db and 10db setting, such that even with half the video frames available, the vanilla AV model would outperform its AO baseline. By contrast, in the clean or 20db setting, the visual modality brings such a small advantage, if any, that even with all the video frames available, the AV and AO models perform similarly within the bounds of the confidence interval.
As in the results from the main paper, we can that Cascade Utt is consistently robust, but Dropout Utt. is to be preferred for the Conformer CAT model since it has better WER performance.
19

---

Published in Transactions on Machine Learning Research (08/2022)
TerUtt
beerrrame
start
mid
Tend
Trate
Clean WER
20.5
20.5
19.5
19.5
19.5
19.5
19.5
19.5
0.0
0.5
1.0
0.0
0.5
1.0
0.0
0.5
1.0
0.0
0.5
1.0
0.0
0.5
1.0
0.0
0.5
1.0
Dropped Frames
Drosped Frames
Dropped Frames
Dropped Frames
Dropped Frames
Dropped Frames
TeeUUt
TeerFrame
starr
mid
Tend
Trat
20db WER
0.5
1.0
0.5
1.0
0.5
1.0
0.5
1.0
0.5
1.0
0.5
1.0
Dropped Fromess
Dropped Frames
Dropped Frames
Dropped Frames
Dropped Frames
Dropped Frames
TerUtt
Tberframe
start
mid
Tend
T Trate
10db WER
25.0
25.0
25.0
22.5
22.5
22.5
22.5
22.5
22.5
20.0
20.0
20.0
20.0
1.0
Dropped Fremes
Dropped Frames
Dropped Frames
Dropped Frames
Dropped Frames
Drapped Frames
TberUtt
TberFrame
Tstart
mid
Tend
Trate
Odb WER
Dropped
Dropped
Dropped
Dropped
Dropped
Dropped
Figure 7: LSTM CAT shown across four acoustic noise conditions and six different test suites. The X- marked, square-marked, diamond-marked, and dashed lines respectively show the WER for the vanilla model, Dropout Utt, Cascade Utt, and the AO baseline. The shaded area around each line denotes the model's 95% confidence interval.
LSTM CAT AV Models Have a Bigger Advantage over their AO Baseline In Figure i we see that unlike the Conformer CAT models, the vanilla LSTM AV models outperform their AO baseline all noise conditions. This suggests that the advantage from the visual modality is larger when the base architecture weaker. In other words, if the architecture not capable at learning good representations from the acoustics alone, it is more likely that the visual modality will bring bigger improvements.
We see that for the LSTM CAT architccture, Cascade Utt is still consistently robust, but now also outperforms Dropout Utl in terms of WER performance.
20

---

Published in Transactions on Machine Learning Research (08/2022)
TberUtt
TeerFrame
start
mid
Tend
Trate
Clean WER
100
50
50
50
50
50
50
0.0
0.5
1.0
0.0
0.5
1.0
0.0
0.5
1.0
0.0
0.5
1.0
0.0
0.5
1.0
0.0
0.5
1.0
Dropped Frames
Dropped Frames
Dropped Frames
Dropp od Frames
Dropped Fromes
Dropped Fromes
TberUtt
DerFrram
T starrt
mid
Tend
Trate
20db WER
50
50
19
19
19
19
19
19
0.0
1.0
1.0
0.0
1.0
0.0
1.0
0.0
0.5
1.0
0.0
0.5
Dropped Frames
Dropped Fromes
Dropped Frames
Dropp od Fromes
Dropped Frames
Dropped Fromes
TberUtt
Tberrrame
staart
T mid
T Tend
T Trtte
10db WER
100
100
100
100
100
100
50
50
50
20
20
20
20
20
20
1.0
Dropped Frames
Dropped Prames
Dropped Fremms
Dropped Frimes
Dropped Promes
Dropped Frames
T ber
TeerFrame
start
T
Tend
Trate
Odb WER
100
100
100
100
100
100
50
50
50
50
50
50
30
30
30
30
30
30
0.0
Drapped
Dropped
Dropped
Dropped
Dropped
Figure 8: Conformer CM shown across four acoustic noise conditions and six different test suites. The X-marked, square-marked, diamond-marked, and dashed lines respectively show the WER for the vanilla model, Dropout Utt, Cascade Utt, and the AO baseline. The shaded area around each line denotes the model's 95% confidence interval. For Tetart, Tmid, Tend, FoerPrame, Trates Dropout Utt does not have its WER monotonically increase as more frames are dropped, indicating that it lacks Test-Time Robustness.
Dropout Utt is Catastrophic on Conformer CM In Figure we notice that Dropout Utt performs catastrophically in response to missing video. Perhaps, this is because the attention mechanism used does not play nicely with the anomalous behavior introduced by blank video frames. The WER degradation also does not scale monotonically with the number of dropped video frames, since actually does better with all the frames dropped compared to 50% or 75% of the frames dropped.
that when the model
For the Conformer CM architecture, Cascade Utt remains consistently robust across the different test suites and acoustic noise conditions tested.
21

---

Published in Transactions on Machine Learning Research (08/2022)
In the pages that follow, present comprehensive tables of WER numbers across the various different experimental settings so that the reader can follow up the details if interested.
Table 4: Robustness to TerUUt for clean. Columns three to seven represent the amount dropped video frames.
ARCHITECTURE
METHOD
0.0
0.5
ROBUST
CONFORMER CAT
AUDIO BASELINE
17.27 0.26
17.27 + 0.26
17.27
0.26
17.27 = 0.26
17.
0.26
CONFORMER CAT
VANILLA
17.37 0 0.26
17.44
4.22 0.26
17.51
0.26
17.63 0.26
17.72
+ 0.26
CONFORMER CAT
VANILLA (25L)
17.33 0.22
17.38 1 0.26
17.42
1 0.26
17.45
0.26
17.51
0.26
CONFORMER CAT
CASCADE Urr
0.26
1..2
17.38
0.26
0.26 17.37
17.37
0.26
CONFORSIER CAT
DROPOUT Urt
0.26
17.21
0.26 17.29
1..3
17:36 = 0.26
CAT
CASCADE FRAME
0.26
0.26 17.47
CONFORMER CAT
DROPOUT FRAME
17.43 0.27
17.45
0.227
17.44
0.27
0.27
17.48 1 0.27
CONFORMER CAT
AV DROPOUT UTT
1.23
17.11
17.12 0.26
17.3 0.26
17.5 0.26
CONFORMER CAT
TWO-PASS
17.28 0.22
17.27 0.22
17.29 0.26
0.26 17.28
17:27 0,26
CON-LSTM CAT
CaSCADe UTT
0:226
17.28 0.22
17.28 0.26
0.26 17.26
17.3 0.26
LSTM CAT
AuDIa BASKLANE
0.29
20.56 0.229
20.56 0.29
20.56 0.29
20.56 0.29
LSTM CAT
VANILLA
0.27
19.2 0.28
19.51 + 0.28
19.98 0.29
20.42 0.29
LSTM CAT
CASCADE
0.29
LSTM CAT
DROPOUT UrT
19.55 0.22
19.81 0.228
20.07 1 0.28
20.32 0.29
20.61 1 0.29
LSTM-CON CAT
CASCADE UTT
18.22 0.27
18.84
19.48 0.28
20.08 0.28
20.64 0.29
CONFORMER CM
AUDIO BASELINE
0.234
17.34
17.34 0.26
17.34 0.26
17.34
0.26
CONFORMER CM
VANILLA
19.15
38.44
59.35
0.86
78.51 0.74
98.33
1 0.05
CONFORMER CM
CasCaDE Urr
0.26
1.244
17.11
17
17.37 0.26
CONFORMER CM
DROPOUT Urr
19.25 21:25:100 0.28
19.47 + 0.28
19.67 + 0.29
19.94 0.29
20.14 0.29
Table 5: Robustness to Frame for clean. Columns three to seven represent the amount of dropped video frames.
ARCHIECCTRRE
METHOD
0.0
0.5
ROBUST
CONFORMER CAT
AUDIa BASKLINE
17.27
17:27 +
17.27
17.27 = 0.26
17:27
CONFORMER CAT
VANILLA
17.37 0.26
17.39 0.26
17.41
0.26
17.55 0.26
17.72
0.26
CONFORMER CAT
VANILLA (25L)
17.33
17.29 - 0.26
0.26 17.34
17.41 = 0.26
17.51 + 0.26
CONFORMER CAT
CaSCADE Urr
0.26
17.3
17.37
CONFORMER CAT
DROPOUT UTT
1.226
17.19
17.21 1: 0.26
17.31 = 0.26
17.36 1 0.26
0.21
17.15
0.26
CONFORMKI CAT
DROPOUT FRAME
0.24
17.39 0.22
17.39 0.27
17.43 0.27
17.48 0.27
CONFORMER CAT
AV DROPOUT UrT
0.26
17.37 0.22
17.12 0.26
17.49 0.26
17.5
0.26
CONFORMER CAT
TwO-PASS
17.28
17.22 0.26
17.25
0.26
17.26
U.26
17:27 U.26
CoN-LSTM CAT
CaSCADE Urt
0.226
17.26 0.26
17.29 0.26
17.29 0.26
17.3 0.26
LSTM CAT
AUDIO BASELINE
0.29
0.29
LSTM CAT
VANILLA
18.73
18.91 1 0.27
19.28 1: 0.28
19.84 0.28
20.42 1 0.29
CAT
0.27
19.62
20.05
0.29
0.29
LSTM CAT
DROPOUT Urr
19.55
19.84
20.2 = 0.29
20.46 0.29
20.61 + 0.29
LSTM-CON CAT
CASCADE UrT
18.22
18.58 0.27
19.22 0.27
19.9 0.28
20.64 0.29
CONFORMER CM
AUDIO BASELINE
17.34 0.26
17.34
17.34 0.26
17.34 0.26
17.34 0.26
CONFORMER CM
VANILLA
19.15 0.28
19.77 + 0.28
20.65 0.3
0.33 23.56
98.33 + 0.05
CASCADE
CONFORMER CM
DROPOUT Urr
19.25
0.28
23.2 1. 0.33
28.66 1: 0.39
0.52 47.17
20.14 1 0.29
22

---

Published in Transactions on Machine Learning Research (08/2022)
Table 6: Robustness to Tstart for clean. Columns three to seven represent the amount of dropped video frames.
ARCHITECTURE
METROD
0.0
0.25
0.5
0.75
1.0
ROBUST
CONFORMEI CAT
AUDIO BASKLINK
17.27 + 0.26
17.27 + 0.26
17.27 + 0.26
17.27 = 0.26
17:27
+ 0.26
CONFORMER CAT
VANILLA
17.37
17.41 0.26
17.51
0.26
17.05 = 0.26
17.72
0.26
CONFORMER CAT
VANILLA (25L)
17.33
17.32
0.226
17.44 0.26
0.26 17.43
17.51 + 0.26
CONFORMER CAT
CASCADE UTT
17.34 0.22
17.35 0.22
17.39 1: 0.26
17.4.10.22
17.37 1 0.26
CONFORMRI CAT
DROPOUT UrT
0.26
17.26
17.26 0.26
17.28 0.26
17.36 0.26
CONFORSIER CAT
CASCADE FRAME
0.26
17.16
17.2
0.26 17.33
17.25 0.26
CAT
DROPOUT FRAME
0.24
1.22
17.43
0.27
CONFORMER CAT
AV DROPOUT UrT
17.35
17.43
17,46 0.26
17.48 0.26
17.5 0.26
CONFORMER CAT
TWO-PASS
17.28
17.32
0.26
17.31 0.26
17.3 0.26
17.27 0.26
CON- LSTM CAT
CASCADE UTT
17.25
17.25 0.26
17.25 0.26
0.26 20.56
17.27
17.3 0.26
LSTM CAT
AUDIO BASELINE
0.229
20.56 0.29
20.56 0.29
0.29
20.56 1: 0.29
LSTM CAT
VANILLA
0.27
19.08 +
19.58 0.28
19.96 0.29
20.42 0.29
LSTM CAT
CasCADE UrT
0.21
19.65 + 0.28
20.0 = 0.28
20.37 0.28
20.6 0.29
LSTM CAT
DROPOUT UTT
0.26
20.15
0.29
20.61
LSTM-CON CAT
CaSCADE Urr
0..2
18.93
19.58 0.28
20.22 0.29
20.61 0.29
CONFORMER CM
AUDIO BASELINE
0.26
17.34
17.34 0.26
17.34 0.26
17.34 0.26
CONFORMER CM
VANILLA
0.215
19.64
20.5 0.29
24.1 0.34
98.33 0.05
CONFORMER CM
Cascade UTT
17.45
17.47 1 0.26
17.45 4: 0.26
17.4 0.226
17.37 1 0.26
CONFORMER CM
DROPOUT Urr
0.22
22.7
0.32
30.19
48.8
20.14 0.29
Table 7: Robustness to Tmid for clean. Columns three to seven represent the amount of dropped video frames.
ARCHITECTURE
METHOD
0.0
0.5
ROBUST
CONFORMER CAT
AUDIO BASELINE
17.27
0.26
17.27
+ 0.26
17.27
0.26
17.27 = 0.26
17.27 + 0.26
CONFORMER CAT
VANILLA
17.37
0.26
17.43 1: 0.26
17.52
0.26 17.45
17.61
0.26
17.72 1 0.26
CONFORMER CAT
VANILLA (25L)
0.226
17.35
0.22
0.26
17.48 = 0.26
17.51
1: 0.26
CONFORMER CAT
CASCADE Urr
0.26
17.37
0.26
17.39 0.26
0.26 17.37
17.37 0.26
CONFORMER CAT
DROPOUT UTT
1..19
17.22
0.26 17.28
17.32
0.26
17.36 1: 0.26
CONFORMER CAT
CASCADE FRAME
07.1
17.2
17.24
0.26 17.33
17:25
CONFORMER CAT
DROPOUT FRAME
0.24
17.43
17.45 0.27
0.27 17.48
17.48
0.27
CONFORMER CAT
AV DROPOUT UTT
17.35
17.42 0.26
17.52 0.26
17.51 D.26
17.3 0.26
CONFORMER CAT
TWSS-PASS
17.24 0.226
17.28 0.26
17.27 0.26
17.27 0.26
CON-LSTM CAT
CaSCADE UTT
17.25
17.25 4 0.26
17.27 0.26
17.31 = 0.26
17.3 0.26
LSTM CAT
AUDIO BASELINE
0.29
20.56 0.29
20.56
20.56
20.56
LSTM CAT
VANILLA
0.22
19.13 0.28
19.57 + 0.28
19.96 0.28
20.42 + 0.29
LSTM CAT
CASCADE Urt
0.21
19.6 0.28
20.0 0.28
20.34 0.28
20.6 0.29
LSTM CAT
DROPOUT UrT
0..2
1
20.17 0.28
20.45 0.29
20.61 = 0.29
LSTM-CON CAT
CASCADE UrT
18.22 1: 0.27
18.76
19.45 0.28
20.11 0.28
20.64 1 0.29
CONFORMER CM
AUDIO BASKLINR
17.34
1..34
17.34 0.26
17.34 0.26
17.34 0.26
CONFORMER CM
VANILLA
19.15
19.7 0.29
20.65
1005-00 0.3
24.9 - 0.35
98.33 0.05
CONFORMER CM
CasCADE Urr
17.45
1.21
17.4
CONFORMER CM
DROPOUT Urr
19.25 0.255:00 0.28
23.45 0.33
32.0 = 0.41
50.91 0.53
20. 14 - 0.29
X
Table 8: Robustness to Tend for clean. Columns three to seven represent the amount of dropped video frames.
ARCHTBECTURE
METROD
0.0
0.25
0.5
0.75
1.0
ROBUST
CONFORMER CAT
AUDIO BASELINE
17.27
0.26
17.27
1 0.26
17.27
0.26
17.27
0.26
17.27
1: 0.26
CONFORMER CAT
VANILLA
17.37 4: 0.26
17.46
+ 0.26
0.26 17.56
17.63 = 0.26
17.72
+ 0.26
CONFORMER CAT
VANILLA (25L)
17.33
17.34
17.36
17.46
0.26
17.51
CONFORMER CAT
CASCADE Urr
17.34
0.22 17.38
17.35
0.26
17.36
0.26
17.37
+ 0.26
CONFORMER CAT
DROPOUT Urr
0.26
17.25
17.32
0.26
0.26 17.33
17.36
0.26
CONFORMER CAT
CASCADE FRAME
17.18
0.266
17.29
0.26
17.35
0.26
17.25
0.26
CONFORMER CAT
DROPOCT FRAME
0.22
17.4 0.27
17.45
0.27
17.49
0.27
17.48 1 0.27
CONFORMER CAT
AV DROPOUT UTT
17.35
17.42
17.46
0.26
0.26 17.53
17.5 1 0.26
CONFORMER CAT
Two-PASS
17.28 0.22
17.24 1 0.26
17.25 1: 0.26
0.26 17.25
17.27 1: 0.26
CON-LSTM CXT
CASCADE
0.22
17.29
0.26
0.26
LSTM CAT
AuDia BASKLANK
20.56
20.56 + 0.29
20.56 0.29
20.56 0.29
20.56 + 0.29
LSTM CAT
VANILLA
0.27
19.16 0.28
19.59 0.28
19.94 0.28
20 0.29
LSTM CAT
CaSCADE UrT
1..19
19.54 0.28
19.88
0.28
20.31 0.28
20.6 + 0.29
LSTM CAT
DROPOUT UrT
0.228
19.82 0.28
20.14 1 0.28
20.42 0.29
20.61 1 0.29
LSTM-CCO CAT
CASCADE Urr
0.277
0.27
19.3 0.27
19.96 0.28
20.64 0.29
CONFORSER CM
AuDIO BASELINE
0.26
17.31
17.34 0.26
17.31 26
17:31 0.26
CONFORMER
VanILlA
19.15
19.64
20.38
23.79 0.34
98.33
CONFORMER CM
CaSCADE Urr
17.45
17.44+0.20
17.4 0.26
17.36 0.26
17.37 0.26
CONFORMER CM
DROPOUT Urr
19.25 0.28
22.75 + 0.32
29.96 0.39
48.67 0.51
20.14 0.29
X
23

---

Published in Transactions on Machine Learning Research (08/2022)
Table 9: Robustness to Trate for clean. Columns three to eight represent the amount of dropped video frames.
AncHracctoa
Marooo
Rooest
ComFoRseR
BASELIME
17.27
0:28
0.26
17:27
ConForMER
0:20
0.20
ContorMen
VANELA (250)
CommoNeER CAT
Cancane PRase
17.18
17 0:26
17.2
17:19
17.25
Commworr CAT
Diorver Frasee
17.43
0:27
0:27
17:37
0.26
17.42
0.26
17.48
Cossocas CAT
AV DRooEy Uvr
17.35
36 0.26
0.28
0.26
17.43
1.0.0
17.5
COSTORSEE CAT
PAss
17.28
0.26
17.20
0.26
17.27
0.20
17.20
0.26
17.23
0.26
17.27
0.26
Cos- LSTM CAT
CAsCABE UTT
17.20
17.25
0.26
17.26
0:26
17.29
0.26
17.31
0.26
17.3 0.36
LSTM CAT
Anuoo BASELINK
20.36
20.36
0:29
0.29 20.36
20.58
0.29
20.38
0.29
20.56 1: 0.29
LSTM CAT
VASILLA
18.73
0.22
18.71
0.27
0.27 18.74
0.27 18:77
19.19
0.28
20.42 0.29
LSTM CAT
CAscARE Upr
19.19
0.27
19.2
0.27
19:22
0.27 18.6
0.27 19.4
19.38
0.25
20.8
0.20
DRarocT
19:55
0:28
20.12
20.61
LSTMS-COs
Cast
18.22
0.27
9:27
0.27
1
20.64
BASELLME
17:34
0:26
20.
Table 10: Robustness to TherUre for 20db. Columns three to seven represent the amount of dropped video
frames.
ARCHITETTRRE
METROD
0.0
0.25
0.5
0.75
1.0
ROBUST
CAT
AUDIO BASELINE
CONFORMER CAT
VANILLA
17.55=0..26
17.64 + 0.26
17.74 0.26
17.83 = 0.27
17.9
0.27
0.27
X
CONFORMER CAT
VANILLA (25L)
1.24
17.54
17.63 0.26
17.72 = 0.26
17.79
0.26
CONFORMER CAT
CASCADE UrT
17.48
17.49 0.22
17.54
0.26
17.57 0.26
17.59
0.26
CONFORMER CAT
DROPOUT Uru
0.26
17.4 0.26
17.47 0.26
17.51 =1.1100.26
0.26 17.57
CONFORMER CAT
FRAME
0.26
CONFORMER CAT
DROPOUT FRAME
17.62:::03 0.27
17.63 1. 0.27
0.27 17.65
17.67 0.27
17.68
0.27
CONFORMER
DROPOUT UTT
3.222.0..2
0.26
17.78
CONFORMER CAT
Two-PASS
17.5
17.51 0.22
0.27 17.51
17.51
17.48 0.26
17:47
0.26
CoN-LSTM CAT
CASCADE UrT
0.24
17.51
0.26
17.53 0.26
17.54
0.26
LSTM CAT
AuDIo BASELINE
2..04:1..29
21.04 + 0.29
21.04 0.29
21.04 0.29
21.04
0.29
LSTM CAT
VANILLA
0.29
19.51 + 0.28
20.01 0.29
20.52 0.29
21.01 + 0.3
LSTM CAT
CaSCADE
0.29
LSTM CAT
DROPOUT UrT
19.85.1..22 0.28
20.2 0.29
20.54 1 0.29
20.81 0.29
21.14 1 0.29
LSTM-CON
CASCADR
0.27
19.13
19.81
0.29 17.6
CONFORMER CM
AUDIO BASKLINE
1.26
17.6
17.6 0.26
17.6 0.26
CONFORMER CM
VANILLA
19.48
39.15
38.97 0.86
79.16 0.73
98.3 0.05
CONFORMER CM
CASCADE UTT
17.67 0.26
17.64 0.26
17.63
0.26
17.63 0.26
17.59 0.26
CONFORMER CM
DROPOUT Urr
19.43 0.28
19.65 + 0.28
19.9 = 0.29
20.13 0.29
20.39 + 0.3
Table 11: Robustness to TherFrome for 20db. Columns three to represent the amount of dropped video frames.
ARCHIEECTRRE
METHOD
0.0
0.25
0.5
0.75
1.0
ROBUST
CONFORMER CAT
Aubia BASKLANK
17.47
0.26
17.47 + 0.26
17.47
0.26
17.47
0.26
17.47
0.26
CONFORMER CAT
VANILLA
17.35 17.35+0.3 0.26
17.54 1..1+0.22
17.63
0.26
17.78 = 0.26
17.9
0.27
X
CONFORMER CAT
VANILLA (25L)
0.26 17.55
CONFORMER CAT
CASCADE Urr
17.48 11.88..0..20
17.49 0.22
17.53 1: 0.26
0.26
17.59
1 0.26
CONFORMER CAT
DROPOUT Urr
17.33
17.32 0.226
17.39
0.26
17.48
0.26
17.57
0.26
CONFORMER CAT
CASCADE PRAME
17.35 0.26
17.33 0.26
17.36
0.26 17.62
17.38
0.26
17.45
0.26
CONFORMER CAT
DRAPOCT FRAME
0.227
17.59 0.27
0.27
17.66 = 0.27
17.68
4: 0.27
CONFORMIE CAT
AV DROPOUT UrT
0.26
17.53
17.58
17.72
17.78 0.26
CONFORMER CAT
Two-PASS
17.5
17.47 0.26
17. 0.26
17.3 + 0.26
17.47 + 0.26
LSTM CAT
CASCADE
0.26
LSTM CAT
AUDIO HASELINE
21.04
21.04 1 0.29
21.04 1: 0.29
21.04 0.29
21.04 1 0.29
LSTM CAT
VANILLA
18.99 0.299-0..27
19.21 1 0.28
19.69 0.28
20.34
0.29
21.01 0.3
LSTM CAT
CASCADE Urr
0.226
19.92 1 0.28
20.45
0.29
0.29 20.77
21.09
0.29
LSTM CAT
DROPOUT Urt
19.85
20.22 0.28
20.58 1 0.29
20.9
21.14 0.29
LSTM-CON CAT
CaSCADE Urr
0.24
18.93
0.29
CONFORMER CM
AUDIO BASELINE
17.6
17.6 0.266
17.6 0.26
17.6 1:66100 0.26
17.6 4 0.26
CONFORMER
20.27
21.06
0.05
CONFORMKR CM
Cascade Urr
17.67
17.62 4 0.26
17.62 + 0.26
17.6 0.26
17.59 4 0.26
CONFORMER CM
DROPOUT Urr
19.43 1. 0.28
23.65 1 0.33
29.21
0.39
48.13 - 0.53
20.39 1: 0.3
X
24

---

Published in Transactions on Machine Learning Research (08/2022)
Table 12: Robustness to Tstart for 20db. Columns three to seven represent the amount of dropped video frames.
ARCHITECTURE
METROD
0.0
0.25
0.5
0.75
1.0
ROBUST
CONFORMII CAT
AUDIO BASKLINK
17.47 0.26
17.47 + 0.26
17.47 + 0.26
17.47
0.26
17:47 + 0.26
CONFORMER CAT
VANILLA
1..55=00.26
17.62 0.26
17.76 0.26
17.87 = 0.26
17.9 0.27
CONFORMER CAT
VANILLA (25L)
17.48 77.48:00.28
17.5 0.226
17.64
0.26
17.69
0.26
17.79 + 0.26
CONFORMER CAT
CASCADE UTT
0.22
17.5 1.2.
17.57
1. 0.26
17.59 = 0.26
17.59
1 0.26
CONFORMRI CAT
DROPOUT UrT
0.26
1..4
17.45 0.26
17.49
0.26
17.57
0.26
CONFORSIER CAT
CASCADE FRAME
17.35
17.3 0.26
17.37 0.26
17.52 = 0.26
17.45
0.26
CONFORMER CAT
DROPOUT FRAME
0.27
1.26
17.62
17.68 0.27
CONFORMER CAT
AV DROPOUT UrT
0:26
17.58 0.26
17.67 + 0.26
17.7
17.78 + 0.26
CONFORMER CAT
TWO-PASS
17.5
17.5 0.26
17.52 0.26
17.52 = 0.26
17.47 + 0.26
CoN- LSTM CAT
CASCADE UTT
0..4
17.48
17.51 0.26
17.52 0.26
17.54 1 0.26
LSTM CAT
AUDIO BASELINE
21.04
2..04
21.04 0.29
21.04 0.29
21.04 1 0.29
LSTM CAT
VANILLA
0.27
0.24
19.98 0.29
20.44 0.29
21.01 +
LSTM CAT
CasCADE UrT
0.24
20.0 0.28
20.12 0.29
20.85 = 0.29
21.09 = 0.29
LSTM CAT
DROPOUT UTT
20.22
20.9
LSTM-CON CAT
CaSCADE Urr
08.4
0.
19.93 0.28
20.62 02::0.2 0.29
21.11 0.29
CONFORMER CM
AUDIO BASELINE
17.6
17.6 0
17.6 = U.26
0.26
17.6 0.26
CONFORMER CM
VANILLA
19.48.1.0.26
20.1 0.29
21.0 0.3
24.85 0.35
98.3 0.00
CONFORMER CM
Cascade UTT
17.67 0:266
17.68 4: 0.26
17.64 0.26
17.63 = 0.26
17.59 + 0.26
CONFORMER CM
DROPOUT Urr
0.28
23.07 0.32
30.48
49.47 0.55
20.39 0.3
Table 13: Robustness to Tiidd for 20db. Columns three to seven represent the amount of dropped video frames.
ARCHITECTURE
METHOD
0.0
0.5
ROBUST
CONFORMER CAT
AUDIO BASELINE
17.47
0.26
17.47
+ 0.26
17.47
0.26
17.47 = 0.26
17.47
+ 0.26
CONFORMER CAT
VANILLA
17.35
0.26
17.63 1 0.26
17.71
0.26
17.84 0.26
17.9 4 0.27
CONFORMER CAT
VANILLA (25L)
17.48
0.26
17.53
0.26
17.82
1: 0.26
17.71 = 0.26
17.79
1 0.26
CONFORMER CAT
CASCADE Urr
17.48
17.52
17.56
17.04 0.26
17.59 0.26
CONFORMER CAT
DROPOUT UTT
1..33
17.39 0.226
17.47 1: 0.26
17.53 0.26
17.57 1 0.26
CONFORMER CAT
CASCADE FRAME
0:26
17.36
17.41
17.53 0.26
17.45
CONFORMER CAT
DROPOUT FRAME
0..62
17.62 0.22
17.63 0.27
17.69 0.27
17.68
0.27
CONFORMER CAT
AV DROPOUT UTT
17.32
17.59 + 0.26
17.7 + 0.26
17.76 D.26
17.78 + 0.26
CONFORMER CAT
TWSS-PASS
17.5
17.48.1...
17.51 0.26
17.51
0.26
17.47 0.26
CON-LSTM CAT
CaSCADE UTT
0:24
17.49 0.226
17.49 0.26
17.52 = 0.26
17.54 1 0.26
LSTM CAT
AUDIO BASELANE
21.04 0.29
21.01
0.29
LSTM CAT
VANILLA
0.29
19.5 1 0.28
20.01 + 0.29
20.49 0.29
21.01 +: 0.3
LSTM CAT
CASCADE Uri
19.97 0.28
20.4 0.29
20.83 0.29
21.09 + 0.29
LSTM CAT
DROPOUT UrT
19.83
20.21 0.28
20.61 +01+00 0.29
20.95 0.29
21.14 = 0.29
LSTM-CON CAT
CASCADE UrT
18.48 1.248.00.27
19.11 0.27
19.83 0.28
20.53 0.29
21.11 1 0.29
CONFORMER CM
AUDIO HASELINR
0.26
17.6 0.26
17.6 0.26
17.6 0.26
17.6 0.26
CONFORMER CM
VANILLA
19.48
20.11
21.22 + 0.3
25.62 = 0.36
98.3 4.05
CONFORMER CM
CasCADE Urr
17.67
17.62
17.59
0.26
CONFORMER CM
DROPOUT Urr
19.43
23.81 + 0.33
32.54 2 0.41
51.38 0.53
20.39 + 0.3
X
Table 14: Robustness Tend for 20db. Columns three to seven represent the amount of dropped video frames.
ARCHTBETTURE
METROD
0.0
0.25
0.5
0.75
1.0
ROBUST
CONFORMER CAT
AUDIO BASELINE
17.47
0.26
17.47
1 0.26
17.47
0.26
17.47
0.26
17.47
1: 0.26
CONFORMER CAT
VANILLA
17.35
0.26
17.65
+ 0.26
17.75
0.26 17.65
17.91 = 0.26
17.9 + 0.27
X
CONFORMER CAT
VANILLA (25L)
17.48
17.53
0.26
17.79
CONFORMER CAT
CASCADE Urr
0.26
17.52
17.51 01511000 0.26
17.54
0.26
17.59
+ 0.26
CONFORMER CAT
DROPOUT Urr
1.23
17.38 1
17.45 0.26
0.26 17.56
17.53
17.57
0.26
CONFORMER CAT
CASCADE FRAME
17.33
17.38 4 0.26
17.17 0.26
0.26
17.45
0.26
CONFORMER CAT
DROPOCT FRAME
17.62 0.22
17.6 0.27
17.68
0.27
17.72
0.27
17.68
1 0.27
CONFORMER CAT
AV DROPOUT UTT
0.26
17.61 0.28
0.26 17.69
0.26 17.76
17.78
0.26
CONFORMER CAT
Two-PASS
17.5::0.27
17.45 1 0.26
0.26 17.45
17.51
0.26
17.47
1 0.26
CON-LSTM CAT
CASCADE
1.226
17.52 1
17.55 0.26
LSTM CAT
AuDIa BASKLANK
0.29
21.04 + 0.29
21.04 0.29
21.04 0.29
21.04 + 0.29
LSTM CAT
VANILLA
0.29
19.52 - 0.28
20.05 0.28
20.49 = 0.29
21.01 0.3
LSTM CAT
CASCADE UrT
19.47 09.47=0.22
19.85
1 0.28
20.27 0.28
20.75 0.29
21.09 1 U.29
LSTM CAT
DROPOUT UrT
0.225
20.22 0.228
20.55
0.29 0.29
20.87 - 0.29
21.14 0.29
LSTM--CON CAT
CASCADE Urr
0.24
0.277
19.66 0.28
20.44 0.28
21.11 0.29
CONFORSER CM
AUDIO BASELINE
17.6
17.6 0.26
17.6
17.6
17.600.20
CONFORMER
VanILlA
0.248....25.
20.05 0.22
20.93 0.3
24.49 0.34
98.3 0.05
CONFORMER CM
CaSCAdE Urr
0.266
17.67 0.26
17.62
0.26
17.59 + 0.26
CONFORMER CM
DROPOUT Urr
19.43 0.28
23.1 0.32
30.47 0.39
49.24 0.51
20.39 0.3
X
25

---

Published in Transactions on Machine Learning Research (08/2022)
Table 15: Robustness to Trate for 20db. Columns three to eight represent the amount of dropped video frames.
AncHraactoRa
Maraoo
Rooest
ComFoRsER
BASELIME
0:28
17.47
ConForMER
0:20
0.20
17.0
ContorMen
VANELA (25L)
0:26
CommoNeER CAT
Cancane PRase
17.35
17.35 0:26
0:20
0.26
0.36
17.45
Commworr CAT
Diorver Frasee
17.62
17.62
17.6
0.27
17:57
0.27
17.62
0.27
08.68
Cossocas CAT
AV Diooey Uvr
17.52
17.5
0.28
17:51
0.266
17.81
0.26
COSTORSEE CAT
Two PAss
17.0 0.277
17.40 1740-4 0.27
17.40
0.27
17.40
0.226
17.44
0.26
17.47
0.26
Cos- LSTM CAT
CAsCABE UTT
17.45 0.25
17:48 4848-1 0.26
17.48
0:26
17.3 0.26
17.34
0.26
17:04
0.25
LSTM CAT
Anuoo BASELINK
21.04
21.04 0:29
0.29 21.04
21.04 0.29
21.04 0.29
21.04
0.29
LSTM CAT
VASILLA
18.90 0.27
18.00 0:27
0:25 19.02
0.28 19.09
19.38 0.25
21.01
0.3
LSTM CAT
CascARE Upr
19.47
0.28
19.45 0.28
19-53
0.25
19.7
0.25
20.4 0.29
21.09
0.29
DRarocT
19.85 1:
0:28
0:24 18.80
20.0
0.28
20.36 1.0.99
21.14
LSMII-COs
Cast
18.48
0.27
0:27
0.27
19.4 10-4:..222
21.11
BASELLME
17.6
0.26 31.27
17.6
Table 16: Robustness to TherUre for 10db. Columns three to seven represent the amount of dropped video
frames.
ARCHITETTRR
METROD
0.0
0.25
0.5
0.75
1.0
ROBUST
CAT
AUDIO BASELINE
CONFORMER CAT
VANILLA
18.75 + 0.28
19.07 1077000 0.28
19.34 = 0.28
19.7
0.29
X
CONFORMER CAT
VANILLA (25L)
0.24
18.71 0.28
0.28 19.01
19.21 0.28
19.51
0.28
CONFORMER CAT
CASCADE Urr
18.47
18.561
18.66
18.73 0.27
18.84 0.27
CONFORMER CAT
DROPOUT Uru
18.23
18.39 1..9
18.56 0.27
18.74 0.27
18.91 = 0.28
CONFORMER CAT
FRAME
0.27
18.75 0.27
CONFORMER CAT
DROPOUT FRAME
18.74 0.28
18.86 1 0.28
18.99 0.28
19.06 0.28
19.19 1 0.28
CONFORMER
DROPOUT UTT
+ 0.27
1.20
CONFORMER CAT
Two-PASS
0.28
18.9 0.28
18,80 0.28
19.0
19.05
0.28
CoN-LSTM CAT
CASCADE UrT
0.27
18.61 0.28
18.71 0.28
18.78 0.28
18 86
0.28
LSTM CAT
AuDIo BASELINE
0.33
24.09 1 0.32
24.09 0.32
24.09 0.32
24.99
0.32
LSTM CAT
VANILLA
0..2
21.61 = 0.3
22.73 0.32
23.7 0.32
21.83 + 0.33
LSTM CAT
CaSCADE Urr
0.3
23.35
LSTM CAT
DROPOUT UrT
21.64 0.3
22.28 1 0.3
22.98 0.31
23.73 0.32
24.36
0.32
LSTM-CON
CASCADR
19.86
0.3
21.98
0.31
CONFORMER CM
AUDIO BASKLINE
0.228
18.92 + 0.228
18.92 0.28
18.92 0.28
18.92
0.28
CONFORMER CM
VANILLA
0.31
41.3 0.77
60.07 - 0.85
79. 0.71
98.31
0.05
CONFORMER CM
CASCADE UTT
18.71 0.27
18.76
0.27
18.83 0.27
18.87 0.27
18.9 1 0.27
CONFORMER CM
DROPOUT Urr
21.15 0.3
21.65 + 0.33
22.07 + 0.32
22.47 0.32
22.95 + 0.33
Table 17: Robustness to TherFrome for 10db. Columns three to seven represent the amount of dropped video frames.
ARCHIEECTRRE
METHOD
0.0
0.25
0.5
0.75
1.0
ROBUST
CONFORMER CAT
Aubia BASKLANK
19.05
0.28
19.05 + 0.28
19.05
0.28
19.05 0.28
19.05
0.28
CONFORMER CAT
VANILLA
18.46 0::66:00 0.27
18.51 + 0.27
0.27 18.69
19.0 0.28
19.7 + 0.29
X
CONFORMER CAT
VANILLA (25L)
CONFORMER CAT
CASCADE Urr
18.47 0.27
18.51 1 0.27
18.63 1 0.27
18.8 0.27
18.84
1 0.27
CONFORMER CAT
DROPOUT Urr
18.23 0.27
18.33 0.27
18.6 0.27
18.6
18.91
0.28
CONFORMER CAT
CASCADE PRAME
18.41 0.27
18.39
0.22
18.55 0.27
0.28 18.66
18.91
0.28
CONFORMER CAT
DRAPOCT FRAME
18.74.1..00 0.28
0.28 18.65
18.82 0.28
0.28 18.88
19.11
19.19
1: 0.28
CONFORMIE CAT
AV DROPOUT UrT
0.22
18.37 0.27
0.28
CONFORMER CAT
Two-PASS
18.83 0.28
18.81
18.92 0.28
19.07 0.28
19.05 + 0.28
LSTM CAT
CASCADE
0.28
LSTM CAT
AUDIO HASELINE
24.09 0.32 0.32
24.09 0.33
24.09 1: 0.32
24.09 0.32
24.09 1: 0.32
LSTM CAT
VANILLA
0.229
21.12 0.3
22.03 0.31
23.47
D.32
24.83
0.33
LSTM CAT
CASCADE Urr
21.2710..29
0.3
22,97
0.31
23,66
0.32
24.05
0.32
LSTM CAT
DROPOUT Urt
0.3
22.4 0.31
23.19 0.31
23.95 = 0.32
24.36
4 0.32
LSTM-CON CAT
CaSCADE Urr
19.86
20.66
23.0
0.32
CONFORMER CM
AUDIO BASELINE
18.92 0.28
18.92 0.28
18.92 0.28
18.92 = 0.28
18.92
1 0.28
CONFORMER
23.32
25.0
0.34
0.39
98.31
CONFORMKR CM
Cascade Urr
18.71
18.77 4 0.27
18,80 1 0.28
18.93 0.28
18.9 - 0.27
CONFORMER CM
DROPOUT Urr
21.15 0.3
27.76 1. 0.37
0.44 35.48
0.53 55.64
22.95 0.33
X
26

---

Published in Transactions on Machine Learning Research (08/2022)
Table 18: Robustness to Tstart for 10db. Columns three to seven represent the amount of dropped video frames.
ARCHITECTURE
METROD
0.0
0.25
0.5
0.75
1.0
ROBUST
CONFORMII CAT
AUDIO BASKLINK
19.05 + 0.28
19.05 + 0.28
19.05 + 0.28
19.05 = 0.28
19.05 + 0.28
CONFORMER CAT
VANILLA
18.46 0.27
18.67
0.27
18.99 0.28
19.31 0.28
19.7 0.29
CONFORMER CAT
VANILLA (25L)
18.45
18.63 + 0.28
18.92 0.28
19.21 0.28
19.51 + 0.28
CONFORMER CAT
CASCADE UTT
18.47 0.27
18.6 0.27
18.88 1. 0.27
18.8 0.277
18.84 1 0.27
CONFORMRI CAT
DROPOUT Urr
18.23
18.39 0.27
18.54 0.27
0.27
18.91 0.28
CONFORSIER CAT
CASCADE FRAME
1.24
18.45
18,58 0.27
18.82 = 0.28
18.91 = 0.28
CONFORMER CAT
DROPOUT FRAME
18.74
18.82
0.28
CONFORMER CAT
AV DROPOUT UrT
18.29
18.49 0.27
18.74 0.27
19.03 0.28
19.25 + 0.28
CONFORMER CAT
TWO-PASS
0.28
18.91
19.01
0.28
19.07 = 0.28
19.05
0.28
CoN- LSTM CAT
CASCADE UTT
0.25
18.63 0.27
18.68 0.28
18.77 0.28
18.86
1 0.28
LSTM CAT
AUDIO BASELINE
2.339
24.09 1 0.32
24.09 1: 0.32
24.09 = 0.32
24.09
1 0.32
LSTM CAT
VANILLA
0.29
21.58 + 0.3
22.68
23.7
24.83
0.33
LSTM CAT
CasCADE UrT
0.22
22.07 = 0.3
22.81 0.31
23.53 = 0.32
21.05 + 0.32
LSTM CAT
DROPOUT UTT
2.3
22.35
0.31
23.86 0.32
LSTM-CON CAT
CaSCADE Urr
21.01 09.8
22.13 1313 0.3
23.25 = 0.31
21.08 + 0.32
CONFORMER CM
AUDIO BASELINE
0.29
18.92 0.228
18.92 0.28
18.92 = 0.28
18.92 0.28
CONFORMER CM
VANILLA
0.33
23.02 0.332
24.87
0.34
30.24 1 0.4
98.31
0.05
CONFORMER CM
CasCade UTT
0.277
18.78 0.27
18.89 0.28
18.91 = 0.28
18.9 1 0.27
CONFORMER CM
DROPOUT Urr
21.15
26.22
33.04 0.36
54.45 0.54
22.95 0.33
Table 19: Robustness to Tiidd for 10db. Columns three to seven represent the amount of dropped video frames.
ARCHITECTURE
METHOD
0.0
0.5
ROBUST
CONFORMER CAT
AUDIO BASELINE
19.03
0.28
19.05
+ 0.28
19.05
0.28
19.05 = 0.28
19:05 + 0.28
CONFORMER CAT
VANILLA
18.46
0.27
18.67
1 0.27
18,98
0.28
19.38 0.28
19.7 1 0.29
CONFORMER CAT
VANILLA (25L)
0.227
18.61 1 0.27
18.93
0.28
19.2 10.2::00 0.28
19.51 1 0.28
CONFORMER CAT
CASCADE Urr
18.47
18.56
0.27
18.69 0.27
18.79 0.27
18.84 0.27
CONFORMER CAT
DROPOUT UTT
0.22
18.38 1 0.227
18.57 0.27
18.78 0.28
18.91 1: 0.28
CONFORMER CAT
CASCADE FRAME
18.41
18.45
18.58
18.87 = 0.28
18.91
CONFORMER CAT
DROPOUT FRAME
18.74 0.28
18.78&0.22
18,97 = 0.28
19.17 0.28
19.19 - 0.28
CONFORMER CAT
AV DROPOUT UTT
18.29 1:00::000 0.27
18.49 0.27
18.81 0.28
19.05 0.28
19.25
0.28
CONFORMER CAT
TWSS-PASS
18.83 0.28
18.88 0.28
18,99 0.28
19.07 0.28
19.05 1 0.28
CON-LSTM CAT
CaSCADE UTT
0.227
18.63 0.28
18.71 + 0.28
18.8 1.28
18.86 1 0.28
LSTM CAT
AUDIO BASELINE
0.32
+
24.09
21.09 0.32
21.09
0.32
LSTM CAT
VANILLA
0.229
21.67 +: 0.3
22.82 0.31
23.81 0.32
24.83 + 0.33
LSTM CAT
CASCADE Uri
0.22
0.3
22.89 0.31
23.61 0.31
24.05
0.32
LSTM CAT
DROPOUT UrT
21.61
22.5 0
23.28 0.31
23.99 0.32
21.36
- 0.32
LSTM-CON CAT
CASCADE UrT
19.86 1.229
20.99 1
22.09 0.3
23.19 D.31
24.08
1 0.32
CONFORMER CM
AUDIO BASELINR
18.92
1..22
18.92
0.28
18.92 0.28
18.92
0.28
CONFORMER CM
VANILLA
0.37
23.29 022900.00
25
31.25 = 0.42
98.31
+ 0.05
X
CONFORMER CM
CasCADE Urr
0.27
18.76
18.89
0.27
CONFORMER CM
DROPOUT Urr
21.15 0.3
27.38 0.337
37.39 0.45
56.38 0.53
22.95 + 0.33
X
Table 20: Robustness Tend for 10db. Columns three to seven represent the amount of dropped video frames.
ARCHTBETTURE
МETROD
0.0
0.25
0.5
0.75
1.0
ROBUST
CONFORMER CAT
AUDIO BASELINE
19.05
0.28
19.05
1 0.28
19.05 0.28
19.05
0.28
19.05
1 0.28
CONFORMER CAT
VANILLA
18.46
0.27
18.78 1 0.27
19.1 + 0.28
19.41 = 0.28
19.7 + 0.29
CONFORMER CAT
VANILLA (25L)
18.45
18.74 0.27
19.24
0.28
19.51
CONFORMER CAT
CASCADE Urr
18.47 0.27
1.61+0.29
18.7 + 0.27
18.8 1 0.27
18.84
+ 0.27
CONFORMER CAT
DROPOUT Urr
0.27
18.48
18.63 0.27
0.28 18.78
18.91
0.28
CONFORMER CAT
CASCADE FRAME
18.41 0.27
18.55
0.27
18.72 + 0.27
0.28 18.97
18.91
: 0.28
CONFORMER CAT
DROPOCT FRAME
18.74 0.28
18.79 1 0.28
18.99 1: 0.28
19.16
0.28
19.19
1 0.28
CONFORMER CAT
AV DROPOUT UTT
18.29 0.27
0.227
18.87 0.27
0.28 19.09
19.25
0.28
CONFORMER CAT
Two-PASS
18.83 0.28
18.84 1 0.28
18.92 0.28
0.28 19.03
19.05
1 0.28
CON-LSTM CXT
CASCADE
18.54
18.64 0.28
18.73
18.79
0.28
18.86
LSTM CAT
AuDIa BASKLANK
24.09
24.09 0.33
24.09 0.32
24.09 0.32
24.09
+ 0.32
LSTM CAT
VANILLA
0.29
2.3
22.96 0.31
23.8 0.32
21.83
0.33
LSTM CAT
CASCADE UrT
21.27 11.27:0.29
22.08 + 0.3
22,83 0.31
23.53 0.31
24.55
0.32
LSTM CAT
DROPOUT UrT
21.64 1.3
22.41 0.3 0.3
23.25 1 0.31
23.92 - 0.32
24.36
1 0.32
LSTM--CO CAT
CASCADE Urr
0.229
20.8 0.29
21.97 0.3
23.03 0.31
24.08 0.32
CONFORSER CM
AUDIO BASELINE
0.29
18.92 0.22
18.92 0.28
18.92 0.28
18.90 + 0.28
CONFORMER
VanILlA
0.31
23.21
24.99
29.99 0.4
98.31 4
CONFORMER CM
CaSCADE Urr
18.71
18.76 + 0.27
18.84 0.28
18.9
18.9 0.27
CONFORMER CM
DROPOUT Urr
21.15 0.3
26.62 + 0.35
35.14 0.42
51.16 0.51
22.95 - 0.33
X
27

---

Published in Transactions on Machine Learning Research (08/2022)
Table 21: Robustness Trate for 10db. Columns three eight represent the amount of dropped video frames.
AncuractuRa
Mirmoo
RODUST
Cossocask CAT
Aumo BASELINK
19.05
28
19.00
0.28
18.05
19.05
1 2.
19.05 1
Cossorosk CAT
VANILLA
18.48
18.45
0.27
18.44
0.27
18.84 4 0.27
19.7
0.20
COSPORMER CAT
YANELA (258)
18.40
18:44
0:27
18:46
0:27
18:44
0.27
18.74
96
19:51 1
CosFoRsER CAT
Caccane Urr
18:47
18
0:27
18:47
0:27
18:49
0.
18.03
15:84 3.
COSTORSER CAC
Deoroct Err
18.23
18.
0.27
18.23
0.27
18.2
0.27
16.34
0.27
15.91
COSTORSER CAC
CAsCADE Pluors
18.41
18.41
0.27
19.89
0.27
18:34
0.27
18.6
0.27
15.91
COSTORMER CAT
Droroct FRAME
18.74
18.
0.28
19.72
0.29
0.28 18.09
16.76
0.25
19.10
Diaroyy Urz
18.25
0.27
18.3
18:31
19.25
CONFORSER
Two-Pass
18.83
0:28
18.81
19.05
Cos-LSTM
0.27
0:27
Urr
21.27
0 0.29
+
24.05
LSTM CAT
21.69
23.19
24.36
UrT
0.000.25
19.
0:29
0.0.0
21.54
COSPORSER C3
Almoo BASELINE
18.92 +: 0.28
18.02
0.28
18.92 0.28
18.92
0.28
18.92
0.25
15.92 0.25
COSTORMER C3
VANILLA
21.79 + +.331
21.79 0.31
21.92 0.31
22.51
0.32
25.36
0.96
95.51
0.05
Cooneers CMl
Caerane Urr
18.71
0.33
18.08 0:27
18.7 0.27
16.71
0.27
18.58
0.28
18.9 18.9.100 0.37
COoOcCR e CM
Deaney Ure
21.15.2
21.24
0.3
21.81
0:51
24.37
0.31
33.36
22,93 1 0.33
Table 22: Robustness to TeerUtt for Odb. Columns three to seven represent the amount of dropped video frames.
ARCHITECTURE
METROD
0.0
0.25
0.5
0.75
1.0
ROBUST
CONFORSER CAT
AUDIO BASELINE
33.54 + 0.43
33.54 + 0.43
33.54 + 0.43
33.54 = 0.43
33.54 + 0.43
CONFORMER CAT
VANILLA
24.96
CONFORMER CAT
VANILLA (25L)
24.75 1:7511000 0.35
27.23 0.33
29.8 1 0.41
32.35
0.42
34.88
1 0.43
CONFORMER CAT
CASCADE UrT
26.12 2.312:00.36
0
28.58 0.38
29.78 = 0.39
31.08
CONFORMER CAT
DROPOUT UTT
24.33 0.34
26.14
27.94 0.38
29.81 0.39
31.67
0.4
CONFORMER CAT
CaSCADE FRAME
28.04 0.38
29.5 19.5..0..
31.08 0.41
32.71 0.43
34.17
0.44
X
CONFORMER CAT
DROPOUT FRAME
27.58 0.37
0.38
30.11 0.39
31.23 0.41
32.54
0.41
CONFORMER CAT
AV DROPOUT UrT
23.64 0.33
26.13 0.36
28.58 0.38
31.05 0.4
33 + 0.11
CONFORMER CAT
TWo-PASS
CMM- CAT
CaSCADe UrT
27.9 0.39
28.81 0.4
29.68 0.4
30.53 1 0.4
31.4 1 0.41
LSTM CAT
AUDIO BASELINE
16.38 0.44
46.38 0.44
46.38 D.44
46.38 0.44
LSTM CAT
VANILLA
30.93 10331.0.50 0.39
36.45 1
41.02 0.48
46.55
0.47
51.53
0.45
LSTM CAT
CasCade UrT
33.67 + 0.4
36.92 1192:000 0.42
40.27 0.45
43.23 0.45
46.55
0.44
LSTM CAT
DROPOUT Urr
33.41
37.06
40.87
41.44 0.46
LSTM-CON CAT
CasCADE Urr
29.37 0:37::000 0.37
33.74
37.79 0.45
42.22 = 0.45
16.55
0.44
AUDIO BASELINE
31.79
31.79
CONFORMEI CM
VANILLA
34.91 24.911 0.433
51.42
66.58 0.78
82.34 0.68
98.34 1 0.05
CONFORMER CM
CASCADE UTT
27.89 1:399::00 0.37
0.37
28.83
0.33
29.7 1 0.39
30.6 D.4
31.56 0.4
CONFORMER CM
DROPOUT Urr
33.11 0.43
36.12 0.45
39.36 0.47
42.43 0.48
45.43 0.47
Table 23: Robustness to TherPrame for 0db. Columns three to seven represent the amount of dropped video frames.
ARCHITECTURE
0.5
ROBUST
CONFORMER CAT
AUDIO BASELINE
33.54
0.43
33.54 1: 0.43
33.54
1: 0.43
33.54 :: 0.43
33.54 1 0.43
CONFORMER CAT
VANILLA
24.96
+ 0.35
26.61
35.51
CONFORMER CAT
VANILLA (25L)
24.75 117551000 0.33
0.35
25.39
1 0.36
26.8 1 0.37
29.6
0.39
34.88
1 0.43
CONFORMER
CASCADR
0.322 0.36
27.5
0.37
29.07
30.44 0.39
31.08
CONFORMER CAT
DROPOUT Urr
24.33
25.08
0.35
26.48 0.36
28.88
0.38
31.67
+ 0.4
V
CONFORMER CAT
CaSCADe FRAME
28.04
28.84
30.17 0.41
32.19
0.43
31.17
0.44
X
CONFORMER CAT
DROPOUT FRAME
27.58 0.33
26.74
28.0 1 0.38
30.23 1 0.4
32.54
0.41
X
CONFORMER CAT
AV DROPOUT UTT
23.64 0.33
24.25 0.33
25.83 0.35
29.2 0.38
33.55 1: 0.41
CONFORMER CAT
TWO-PASS
CON-LSTM CAT
CASCADE UrT
27.9 1. 0.39
29.59 1. 0.4
30.52 1. 0.4
31.15 0.41
31.4 1 0.41
CAT
AUDIa BASKLINE
46.38
46.35 0.44
46.38
46.38
LSTM CAT
VANILLA
30.93
33.55 0.411
38,28 0.43
45.45 0.46
51.53 - 0.45
LSTM CAT
CASCADE Urr
33.67 0.4
38.23
42.3 0.43
45.23 0.44
46.55 0.44
LSTM CAT
DROPOUT UTT
$3.41
37.63 0.42
42.18 0.44
45.98 0.45
48.13 0.44
L.SM--CON CAT
CaSCADR Urr
29.37 0.33
33.23
38.42 0.42
43.30 0.44
46.55
0.44
CONFORMICR CM
AuDIO BASELANE
0.44
0.44
31.79
31.79
31.79
CONFORMER CM
VANILLA
4..43
42.36 0.447
49.48 0..
61.87 - 0.51
98.34
0.05
CONFORMER
CASCADE Urt
0.37
29.23
30.6+
31 0.4
3..
CONFORMER CM
DROPOUT Urr
0..11
55.38 0.52
0..77
79.58 = 0.46
43.43 0.17
X
28

---

Published in Transactions on Machine Learning Research (08/2022)
Table 24: Robustness to Tstart for Odb. Columns three to seven represent the amount of dropped video frames.
ARCHITECTURE
METROD
0.0
0.25
0.5
0.75
1.0
ROBUST
CONFORMER CAT
AUDIO BASKLINK
33.54 0.443
33.54 + 0.43
33.54 54
+ 0.43
33.54 = 0.43
33.54
+ 0.43
CONFORMER CAT
VANILLA
21.96 0.35
26.68 + 0.36
29.06
0.37
32.07 0.1
35.51
0.43
CONFORMER CAT
VANILLA (25L)
24.75+0..5
26.38
0.36
29.13
0.38
31.97
0.41
3188
0.43
CONFORMER CAT
CaSCADe UTT
26.12 0.366
27.49 0.38
28.83
0.38
30.16 0.39
31.08
0.4
CONFORMRI CAT
DROPOUT Urr
24.33
0.33
27.79 0.37
29.79 0.39
31.67
0.4
CONFORSIER CAT
CASCADE FRAME
0.30
29.24
30.96 0.42
33.01 = 0.43
34.17 = 0.44
X
CAT
DROPOUT FRAME
27.58
29.55
0,41
CONFORMER CAT
AV DROPOUT UrT
0.36
25.67 + 0.34
28.27 0.37
31.04 0.39
33.55
+ 0.41
CONFORMER CAT
TWO-PASS
30.06
32.0 0.43
33.01 0.43
33 54 0.13
CON-LSII LSTM CAT
CASCADE UTT
27.9
2
29.64 4:44:00
30.66 D.41
31.4 + 0.41
ISTM CAT
AUDIO BASELINE
46.38 0.44
46.38
0.44 46.38
46.38 0.44
46.38
0.44
LSTM CAT
VANILLA
0.39
0.42
0.45
47.03 0.45
51.53 0.45
LSTM CAT
CasCADE UrT
33.67 0.1
37.38
41.07 0.13
11.11 0.44
16.55
0.14
LSTM CAT
UTT
33.41
LSTM-CON CAT
CaSCADE Urr
29.37
31.07 0.4
38.71 11.1+00 0.42
43.41 0.44
16.55 + 0.44
CONFORMER CM
AUDIO BASELINE
31.79 0.44
31.79 0.42
-
31.79
0.42
CONFORMER CM
VANILLA
0.443
41.37 0.445
49.26 0.48
61.27 0.52
98.34
0.05
CONFORMER CM
Cascade UTT
0.33
28.96 0.339
29.97 0.39
31.02 0.4
31.56 4 0.4
CONFORMER CM
DROPOUT Urr
0.44
0.52
61.91
75.46
45.43
0.47
Table 25: Robustness to Tiiid for Odb. Columns three to seven represent the amount of dropped video frames.
ARCHITECTURE
METHOD
0.0
0.5
ROBUST
CONFORMER CAT
AUDIO BASELINE
33.34 0.43
33.54 + 0.43
33.54
03
33.54 = 0.43
33.54
0.43
CONFORMER CAT
VANILLA
24.96
0.35
26.95 1 0.36
29.51
0.38
32.5 0.4
35.51
0.43
CONFORMER CAT
VANILLA (25L)
24.75 1:75:::00 0.35
0.35
26.84
0.33
29.55
0.38
32.27 = 0.41
34.88
1 0.43
CONFORMER CAT
CASCADE Urr
26.12
2..74
29.11 0.38
30.32 0.39
31.08 +
CONFORMER CAT
DROPOUT UTT
24.33 1.431.0.00 0.34
26.24
28.18 0.37
30.22 0.39
31.67
1: 0.4
V
CONFORMER CAT
CASCADE FRAME
28.04
29.26
30.99
33.03 = 0.43
34.17
CONFORMER CAT
DROPOUT FRAME
27.38
2
29.82 0.39
31.35 0.4
32.51
+ 0.41
CONFORMER CAT
AV DROPOUT UTT
0.33
25.9 + 0.34
28.63 0.37
31.42 D.39
33.55
0.41
CONFORMER CAT
TWSS-PASS
30.06 0.42
31.17 0.42
32.16 0.43
33.13 0.43
33.54
0.43
CON-LSTM CAT
CaSCADE UTT
27.9 0.39
29.02 + 0.39
29.99 0.4
30.85 = 0.41
31.4 + 0.41
LSTM CAT
AUDIO BASELANE
16.38 0.41
46.38 0.44
46.38
46.38 0.44
46.38
0.44
LSTM CAT
VANILLA
00.93+0.00
36.53 0.41
42.25 0.43
47.31 0.44
$1.53 4 0.45
LSTM CAT
CASCADE Urt
33.67 - 0.4
37.95
41.73 0.43
44.84 0.44
46.55
0.44
LSTM CAT
DROPOUT UrT
$3.41
42.25 0.43
45.93 0.44
18.13
0.44
LSTM-CON CAT
CASCADE UTT
29.37 0.37
34.44
39.16 0.42
43.59
0.43
46.55
0.44
CONFORMER CM
AUDIO BASELINR
31.79.1
31.79 0.44
31.79
0.42
31.79 0.42
31.79
0.42
CONFORMER CM
VANILLA
34.91
42.26
49.84 0.48
62.43 0.52
98.34
0.05
X
CONFORMER CM
CasCADE Urr
29.07 0.399
30.17
31.1
31.56
0.1
CONFORMER CM
DROPOUT Urr
33.11 0.44
50.73 0.551
64.11 0.51
75.94 0.47
45.43 4 0.47
X
Table 26: Robustness to Tend for 0db. Columns three to seven represent the amount of dropped video frames.
A ARCHTBECTURE
МETROо
0.0
0.25
0.5
0.75
1.0
ROBUST
CONFORMER CAT
AUDIO BASELINE
33.54
0.43
33.54
1 0.43
33.54
0.43
33.54
0.43
33.54
1 0.43
CONFORMER CAT
VANILLA
24.96
0.35
27.36
+ 0.36
30.03 1: 0.38
32.67 + 0.4
35.51
4 0.43
CONFORMER CAT
VANILLA (25L)
24.75
0.35
27.27
29.97
1 0.4
CONFORMER CAT
CASCADE Urr
0.366
27.77
0.37
29.13 0.38
30.26 = 0.39
31.08
4 0.4
CONFORMER CAT
DROPOUT Urr
0.33
26.5
28.56 0.37
30.33 0.39
31.67
CONFORMER CAT
CASCADE FRAME
08.04:0..3
31.18 1188000
33.01 0.42
31.17 : 0.44
X
CONFORMER CAT
DROPOCT FRAME
27.58 0.37
28.26 0.37
29.96 0.39
31.64 0.4
32.54
0.41
CONFORMER CAT
AV DROPOUT UTT
0.33
26.3
0.34
29.04 0.36
31.57 0.39
33.55
0.41
CONFORMER CAT
Two-PASS
30.06 1006:1100 0.44 0.42
30.94 1944:00 0.42
31.93
0.43 30.06
32,87 = 0.43
$3.54
0.43
CON-LSTM CAT
CASCADE
27.9
30.88
LSTM CAT
AuDIa BASKLINK
46.38 0.44
46.38 0.44
46.38 0.44
46.38 - 0.44
LSTM CAT
VANILLA
0.39
36.39
41.91
46.81 0.43
51.53 0.15
LSTM CAT
CaSCADE UrT
33.67 -
$7.11-0.
40.88 0.42
41.33 0.43
46.55
0.44
LSTM CAT
DROPOUT Urr
33.41 0.4 0.4
3.4
41.68 1. 0.42
45.49 - 0.43
48.13
0.44
LSTM--CON CAT
CASCADE Urr
20.37
0.38
38.41
43.07 0.43
46.55
0.44
CONFORSER CM
AuDIO BASELINE
31.79 0.41
31.79 0.42
31.79 0.42
31.79
0.12
CONFORMER
VanILlA
0.43
0.45
60.59
0.05
CONFORMER CM
CaSCADE Urr
0.337
29.06
30.2 0.39
2..
31.56 $1.56 0.4
CONFORMER CM
DROPOUT Urr
0.41
18.17
61.11
74.33 7L43=0.0
45.43 - 0.47
X
29

---

Published in Transactions on Machine Learning Research (08/2022)
Table 27: nobustness to Trate for Odb. Columns three to eight. represent the amount of dropped video frames.
AncuraeTuRs
VANELA (25L)
Coumwore CAT
Casrane Urr
26.12
0:36
26 25
0.39
0.4
Comwwork CAT
DRoiorY Uve
24.33
0.34
24 35
0:34
0-31
24.48 0.31
26.86
36
COSTORSER CAT
CascABE FRAArt
28.04 + 0.88
28.05
0.38
28.08
0.38
28.28
0.39
30.14
0..42
34.17 + 0.44
COSTORSER CAT
Droroct FRAME
27.08 + 0.37
27 47
0.87
27.11
0.36
26.01
0.36
27.50 + 0.35
32.04 2:
COSTORMER CAT
AV DRoroct Ur7
23.64 0.33
23.62
0.33
23.63
0:33
28.8 2 0.33
25:36
4 0.35
25.00 2
ComPORSER CAT
Two-Pass
30.06
0.42
90.00 05
0.42
30.11
0.42
30.47 2:47-2000 0.43
$1.01
1 0.43
35.54
Cos-L9TM CAT
CASCADE Uvz
27.9
0.30
27.36
0.59
28.11
0.39
0.39 28:54
30.7
0.41
+ 0.4L
Have
46.38
0:44
0:64
46.38
30.83 0.39
30.88
0.39
0.38 34.15
0:41 32.06
37
51.53
33:71 0-4
VANILLA
COSTORMER
Copowos CM
DROOeY Ure
0.43
33. 0.42
15.43 1
30

---

Published in Transactions on Machine Learning Research (08/2022)
F Experiments on TED LRS3
We present additional results on the TED LRS3 dataset for just the Conformer CAT and LSTM CAT Cascade Utt models. The architectural configurations are the same as the ones detailed in Section S with the exception of differences optimization. The models are trained with a constant learning rate of 0.0001 at batch size 512 for 35k steps. Like Afouras et al. 2018)) and Ma et al. (021bb, we initialize our models from pre-trained weights. This was done using the training data described Section Like the main paper, we augment the TED LRS3 test set with artificially added babble noise from the Noise corpus (Varga & Steeneken 1993 at. different SNR levels: clean, 20db, 10db, Odb. Like in the main paper, all the Cascade Utt models
found to be robust on all the test suites. To our knowledge, the 0.92 WER we are reporting on the clean TED LRS3 test set by the Conformer CAT Cascade Utt model is also the best number that has been reported on this dataset.
Table 28: Robustness Ube for Conformer and LSTM Cascade Utt models. Columns three seven represent the amount of dropped video frames.
ARCHITECTURE
NOISE
0.0
0.25
0.5
0.75
1.0
AO BASELINE
ROBUST
CONFORMER CAT
0.92
0.26
0.9 0.25
0.99
0.27
0.95
0.26
1.01
0.27
0.99 0.26
LSTM CAT
CLEAN
1.8 0.35
2.69 :: 0.47
3.32
0.49
1.16 = 0.58
4.9 0.61
4.82 = 0.61
20DB
LSTM CAT
2008
0.35
2.64 1: 0.45
3.26 0.51
4.16 0.57
5.25 + 0.65
5.16 0.64
CONFORMER CAT
1008
1.1
1.13 0.27
1.16
0.27
0.28
141 0.31
LSTM CAT
10D8
2.11 0.38
3.31 0.52
4.62
5.57
0.68
6.79 0.74
6.6 0.73
CONFORMER CAT
0Dn
0.36
2.53 0.43
3.22 0.5
3.91 0.55
4.41 1 0.58
4.11 + 0.57
LSTM CAT
1.32 2
8.95
1.41
25.01
21.54 1.45
Table 29: Robustness to TherProme for Conformer and LSTM Cascade Utt models. Columns three to seven represent the amount of dropped vidco frames.
ARCHITECTRRE
NoIs:
BASBLINE
ROBUST
CONFORMER CAT
CLRAN
0.92 1 0.26
1.0 1 0.26
1.0 0.27
0.99
1 0.27
1.01
0.27
0.99 1: 0.26
LSTM CAT
1.800 0.35
2.07 + 0.36
2.85 0.44
3.69
0.49
4.9
4.82 : 0.61
CONFORSIER CAT
200a
1.02
0.95 0.25
0.97 0.26
1.03
0.27
0.26 0.99
1.11 0.29
LSTM CAT
20DB
0.32
0.65
5.16
CONFORMER CAT
10D8
1.1 0.27
1.12
1.13
1.27 1 0.29
0.29
1.19 0.28
1.41 0.31
10na
0.45
3.66
0.64
0.74
6.6
CONFORMER CAT
Opu
1.9
0.36
2.32 0.4
3.38 0.51
4.0 0.55
4.41
1111:00 0.58
441 0.57
L.TTM CAT
0PB
4.32 1: 0.55
8.26
0.82
12.93 1.03
19.69 1.3
25.04 1 1.5
24.54 1.45
Table 30: Robustness to Tstart for Conformer and LSTM Cascade Utt models. Columns three to seven represent the amount of dropped video frames.
ARCHUTECTRE
NOISE
0.0
0.25
0.5
0.75
1.0
AO BASELINE
ROBUST
CONPORMER CAT
CLEAN
0.92 0.26
0.93
+: 0.25
1.05 + 0.27
1.06 =: 0.27
1.01 0.27
0.99 +: 0.26
LSTM
CLEAN
0.35
3.04 + 0.46
4.03
4.6 0.58
0.61
4.82 0.61
CONPORMER CAT
2008
1.02
0.26
0.99
0.26
1.04 0.26
1.09 0.27
0.99 0.26
1.11 0.29
LSTM CAT
20be
1.35
0.48
4.13 0.56
5.01 0.62
5.25 0.66
5.16 0.64
CAT
1.08 1
1.3
1.19
LSTM CAT
10DB
2.11 0.38
0.38
3.61 0.5
4.87 0.6
6.31 1: 0.7
6.79 0.77
6.6 1 0.73
CONFORMER
1.9
0.58
4.41
LSTM CAT
Ons
4.32
0.55
9.76 - 0.88
16.05 + 1.19
22.26
1.44
1.44
25.04 0.04+110 1.5
24.54 1.45
31

---

Published in Transactions on Machine Learning Research (08/2022)
Table 31: Robustness to Tidd for Conformer and LSTM Cascade Utt models. Columns three to seven represent the amount of dropped video frames.
ARCHITECTURE
NOISE
0.0
0.25
0.5
0.73
1.0
AO BASELINE
ROBUST
CONFORMER CAT
CLRAN
0.26
1.04 + 0.27
1.09
0.28
1.04 + 0.27
1.01 + 0.27
0.99 + 0.26
LSTM CAT
CLEAN
0.35
2.63
+ 0.42
3.46
0.49
4.37 + 0.57
0.61
4.82 = 0.61
CONFORMER CAT
20DB
0.26
1.02
+ 0.26
1.08
0.27
1.01 0.22
0.26 0.99
1.11 - 0.29
LSTM CAT
20DB
1.8 1.35 0.35
2.52
1: 0.42
3.59 0.5
4.58 1: 0.59
0.65 5.25
5.16 1 0.64
CONFORMIR CAT
10DB
0.27
1.09
0.27
1.28
1.26
1.19
1.41 0.31 0.31
LSTM CAT
10Du
2.33
3.28
+ 0.48
0.6
6.0
6.79 0.74
6.6 0.73
CONFORMER CAT
0.36
3.26
1.41
441 0.57
LSTM CAT
UDB
4.32 0.55
10.57
0.94
16.98 = 1.23
22.22 1.39
25.04 + 1.5
24.54
1.45
Table 32: Robustness to Tend for Conformer and LSTM Cascade Utt models. Columns three seven represent the amount of dropped video frames.
ARCHITECTURE
NOISE
0.0
0.25
0.5
0.75
1.0
AO BASELINE
ROBUST
CONFORMER CAT
1.02
1.01
0.28
1.01
0.26
LSTM CAT
CLEAN
1.8
0.35
1.92 1 0.36
2.78 1: 0.43
3.74
U.52
4.9 0.61
4.82 : 0.61
20DB
1:
0.27
LSTM CAT
2008
0.35
0.37
2.79 0.45
3.96 0.54
5.25 0.65
5.16 0.64
CONFORMER CAT
1008
1.27
1.09 0.277
1.11 0.27
1.2 0.25
0.29
141
LSTM CAT
100R
2.11 111::.03
2.33 0.38
3.65
5.31
0.66
6.79 1.79
6.6
CONFORMER CAT
0DU
0.36
2.59 0.42
3.39 0.49
3.95 0.54
0.54
4.41 0.58
4
LSTM
1.32
7.76
13.45
1.5
2
Table 33: Robustness to Trate for Conformer and LSTM Cascade Utt models. Columns three to cight represent the amount of dropped video frames.
AnCuTFCCTRR
128
LSTM CNT
1.8.2
1.77
0.35
1.76
1.04
2.47 0.30
4.9
0.61
CAT
1.02 0.226
1.02 0.26
1.0.
1.02
0.337
0.90
0.226
1.11.:00 0.20
LSTM CAT
2008
1.8.2.0.35
1.78220.33
1.76
6.76.
1.94 0.36
2.30 0.441
8.25 0.08
5.18 0.84
COSTORSER CAT
1008
1.1
1.127...7
1.27
1.11 +.2.
1.07 0.227
1.08 0.266
1.19 +
1.41+00 0.331
LSTM CAT
1000
2.11 0.38
2.12
2.15 0.4
2.38
3.1 0.45
4.70+
6.0 0.74
CIAFARUUE: CAT
Dnn
1.5
0.36
1.9 1.9.1.00 0.33
1.95
1 0.37
2.35 0.34
2.91 0.36
2.91
4.41 0.58
4.41.1..33 0.337
1STM CAT
Doe
4.32
0.55
4.33 0.55
4.77
2.0. 0.63
5.95
0.887
11.93
1.02
25.04 1 1.5
24.51 1.45
32


Reviews:
[{"review_id": "WA0cPgo0Or", "reviewer": "Reviewer_gReF", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n- The proposed framework is novel as far as I can tell, and could be very useful for the community. \n- The cascade approach seems also novel and is a significant finding, as it is the most robust approach.\n- Overall it's a very good paper, it is a significant contribution to the field of audio-visual speech recognition.\n\nWeaknesses:\n- My main concern is that the results and the framework are not reproducible: (1) the training data seems to be a private dataset as no reference or link are provided and (2) the parameters of the test suits are not clearly documented. Section 4.4 gives examples of parameters, but it's not clear that the examples are the actual values used in the experiments. It's a big issue for me, as one of the goals of such a benchmark is to provide a way for the entire community to evaluate and compare their model, which is not possible currently. Can the authors clarify the experimental setup and comments on their plan to share their data?\n- The paper is generally well written it has some issues in terms of structure and clarity:\n    - in Section 1.1, after the end of the c) paragraph, it's not clear if the following two paragraphs are still part of c) or part of 1.1.\n     - Section 2 is a bit confusing, the title suggests that the section is going to be about the task, thus will describe the related work, but it's really the description of the proposed model. Why not call it something like \"Methodology\" or \"proposed approach\" ? It is also really short and could benefit from more information (see \"Requested Changes\").\n     - The related work is scattered in different parts of the paper: in Section 1, in 3.1 and 4.2. I think the paper could benefit from having a clearer separation between related work and novel contributions by having a dedicated \"Related Work\" Section.\n     - The methods evaluated in Section 6 (vanilla, cascade utt, etc) are hard to find, as they are briefly described at the end of Section 3.2. I think it would be better to describe them more clearly, for instance the same way the architectures are described in Section 5.\n     - The conclusion is very short, a summary of the findings in Section 6 would be beneficial.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper presents a novel methodology to evaluate robustness of audio-visual speech recognition models to missing videos frames. A framework is introduced, where a model is considered robust when performance without video is not worse than performance of an audio-only model. The paper provides empirical results for well known architectures and approaches, and shows that the Cascade approach, which has an audio-only path used when the video is missing, is the most robust approach. ", "claims_and_evidence_yn": "", "claims_explanation": "", "audience_yn": "", "audience_explanation": "", "requested_changes": "- Section 2: \n    - Change title.\n    - Define the terms used in Equation (1).\n    - Please provide the definition of the RNN-T loss.\n- Clarify the parameters of the test suites.\n- Review the structure of the related work.\n- Clarify the methods used in Section 6.\n- Expand the conclusion.\n", "broader_impact_concerns": "The Broader Impact Statement is present and sufficiently addresses the ethical implications. ", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "rbCzOeBz4T", "reviewer": "Reviewer_EdkC", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths\n1. Multimodal learning for a variety of problems, including speech recognition, are gaining significance, and understanding robustness in terms of availability of different modalities is an important problem. The lack of consistent evaluation strategies, as pointed out in the paper, does make it difficult to understand the robustness of different methods. The paper is a step in the right direction - the robustness and evaluation strategies should be well defined. The paper is also well-written and presented. \n\n2. The robustness framework makes sense and does rectify the problem of lack of robustness definition and test conditions. The test conditions presented cover most of the common cases. This should make the paper a good read on the topic of AV speech recognition. \n\n3. The empirical results and analyses are also good.  I do have some concerns around them but they can be rectified. \n\n\nWeaknesses \n\n1. One primary concern is regarding the dataset used in experiments. The paper uses only one dataset and that too a private training and test dataset sourced from YouTube which will not be available for the research community in general. Other AV speech recognition dataset are available and analyses of results on those would have helped paint a better picture and improve reproducibility. Given that the paper is aimed to study “robustness” - studying multiple datasets will go a long way in establishing the claims. \n\n2. The robustness framework mentions T_all test suite but this is not really empirically studied. In general any composition of the 6 different test conditions are realistically possible. From the definition of the robustness framework also, it is not clear to me what would it mean to think of robustness when 2 or more of these test conditions are simultaneously present. It would also be good to see some empirical results on this. \n\n3. In some cases, the CASCADE UTT ends up performing better than AO, even when video is completely missing. For example, in Table 2, it is better than Audio baseline by almost 2%. Given the way Cascade is defined in  Eq 5, why should this be possible ? In section 3.2 authors mention that Cascade “explicitly disentangles” AV from AO representations. \n\n4. For the Cascade framework, why is CASCADE FRAME not robust ? Some discussion on it would be helpful . \n\n5. Not sure if its only me but the terms ``Train-Time Robustness” and “Test-time robustness” are a bit confusing. \n\n6. Section 4.4 presents different test conditions. Discussing the connection between these and the robustness definition would be really helpful for readers. What does a given test condition with different rates mean in terms of robustness for a given model ? \n\n7. In Section “Cascade Utt achieves model capacity” authors mention that one pass uses half the resources of two pass training. I am not clear about this. In the two passes optimization,  AM is trained first and then frozen, and then AVM is trained. WHy will it take twice the resources ? \n\n8. Are the results in Tables for T_end miss-labeled. The columns should ready  1.0, 0.75, 0.5, 0.25, 0 ?? \n\n9. Several of the references in section 1, the paragraph about prior works of missing modality, are not really about missing modality. They seem to be just multimodal learning frameworks, examples Nagrani 2021, Gogate 2021 etc. \n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper focuses on robustness of speech recognition systems, in particular audiovisual speech recognition systems when visual signals may or may not be available. \n\nThe paper argues that the current approaches for robustness evaluations do not provide a complete picture and a more formal approach is necessary. The paper presents a formal definition of robustness to missing video through order theory and then defines common case test conditions (of missing video) under which different models are evaluated. \n\nTo enhance the robustness itself, the paper argues for a Cascade framework in which an audio-only (AO) model is cascaded by audiovisual model and the framework ensures that in case of missing video - the system is at least as good as an audio-only model. \n", "claims_and_evidence_yn": "", "claims_explanation": "", "audience_yn": "", "audience_explanation": "", "requested_changes": "I would propose that authors address the weaknesses pointed out in the previous section. \n\nFor 1 & 2. It would be best if these are rectified through additional experiments. I think adding them would be important in arguing for acceptance of the paper. \n\nFor 3 to 9. I would urge the authors to respond to these comments and make appropriate changes in the text to address them when necessary. \n", "broader_impact_concerns": "I do not have broader impact concerns. \n", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "WM45HeZlfR", "reviewer": "Reviewer_6k5k", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n\nS1) The proposed evaluation framework is intuitive and sensible. However, please see MW1.\n\nS2) Although the cascaded architecture is not novel, its application to audiovisual speech recognition is sound and intuitive.\n\nS3) The paper provides an extensive comparison of several methods under several conditions of missing video and noise levels. The cascaded architecture does produce more robust results (as initially claimed). While I find this to be a strength, I think the definition of robustness needs to be expanded to warrant publication.\n\nS4) The paper is well structured and easy to read. The mathematical framework is clear and cogent.\n\nMajor Weakness:\n\nMW1) In my view, the major weakness of this framework is that it only addresses robustness to known missing video information, ie, when frames are dropped due to low bandwidth or when a user turns off their camera. However, in all these cases, we know that video frames are missing, which means that we could trivially swap to an audio-only model, instead of relying on an audio-visual model with zeroed-out video input. This trivial approach would be robust under this paper’s definition of robustness. \n\nPerhaps a more realistic and challenging definition of robustness would be robustness to unknown missing video. This would occur when for example the crop of the speaker's lips fails, or if the speaker looks away or their mouth becomes occluded. In these cases, we might have video information, but it is not relevant to the task at hand. Since we may not know which frames are relevant, we couldn’t just swap to an audio-only model. Another potential source of corruption that models should also be robust would be robustness to common corruptions like jitter and blur (as acknowledged by the authors in the limitation section). \n\nGiven that the source of corruption studied in this work (known missing frames) could be easily addressed with an audio-only model, it seems to me that expanding the definition of robustness is necessary to warrant publication.\n\nWeaknesses:\n\nW1) Lacking information about used backbone models. I understand the paper does not propose a new backbone architecture. Instead, it uses LSTMs and Conformers proposed in previous work. However, more information is necessary to fully describe the models used in the experiments. For example, the language model and joiner are not specified.\n\nW2) According to Section 5 “Training data”, the training dataset is collected by selecting videos in which user uploaded transcripts match a production quality ASR system. It seems to me that the (already existing) ASR system is the upper bound to any model trained on this data. Are these systems simply relearning the predictions of the “production quality” model? In which case, what’s the point? Having said that, I understand that this may be common practice in this area, and this paper does not seek to address this problem.\n\nW3) Although acknowledge as a limitation in the paper, I believe that the binary “robust/not robust” metric does not provide enough information about the robustness of a model. After reading the paper, I began to think of robustness as a constraint, which can be violated with different levels of severity. However, the binary “robust/not robust” metric does not capture this. For example, we may be willing to forego a bit of robustness in order to obtain higher accuracy. Since establishing such a metric could be accomplished without any extra experiments or evaluation settings, I believe it should have been included in the manuscript.\n\nOther comments:\n\nO1) While I do agree with the requirement for robustness to missing video information, it is worth noting to point out that, one of the most reliable speech recognition systems (the human brain) is not fully robust. Take for example the widely known McGurk effect.\n\nO2) Section 4.3 corollary 4.7. While clearly true, it is unclear to me why this corollary is mentioned. I didn’t see any applications or important conclusions drawn from this corollary.\n\nO3) It would be worth pointing out in related work that robustness in multi-modal learning has been studied beyond audio-visual speech recognition, eg [A] Morgado et al. Robust audio-visual instance discrimination. CVPR. 2021. or [B] Han, et al. Noise-Tolerant Learning for Audio-Visual Action Recognition. arXiv:2205.07611, 2022.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper deals with audiovisual speech recognition. It shows that current techniques while capable of leveraging visual data to improve speech recognition, they are often not robust to missing video data, ie when models trained on audio-visual data are tested on data with missing video information, they can underperform audio-only (unimodal) models. The paper seeks to formalize how to evaluate robustness to missing video information, and makes three main contributions. \n\n1) A sensible robustness evaluation framework, spanning a variety of realistic conditions under which missing video could arise. Under this framework, a model is robust if its performance improves as it is given more video information (ie less missing frames).\n\n2) An evaluation of previously proposed robustness techniques (mostly focused on modality dropout). This evaluation shows that robustness to missing video information is not always guaranteed with modality dropout. This is an interesting observation.\n\n3) A cascaded audio-visual architecture that fuses audio-only predictions with audio-visual predictions. At inference time, the cascaded model can be only partially used, if only audio inputs are available, or fully used if both audio-visual information is available. By training the model with both unimodal and multimodal inputs, the cascade is shown to be robust to missing video in a variety of noise conditions and using a variety of backbone components.", "claims_and_evidence_yn": "", "claims_explanation": "", "audience_yn": "", "audience_explanation": "", "requested_changes": "Critical changes are described in MW1, W1 and W3 (especially MW1). This however entails significant changes to the whole manuscript, and I’d like to review it again before publication. If these changes are done properly, I believe this could be a strong paper. Properly evaluating robustness is a contribution that would be of interest to a subset of TMLR readers.", "broader_impact_concerns": "A broader impact statement should be added, acknowledging that by the nature of the dataset collection process, the performance of the model with under-represented groups is not being measured and that optimizing for the specific benchmark metrics proposed here may have unknown effects (either positive or negative) on the performance of models within such groups.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: Two reviewers are leaning to accept, but one reviewer has raised some concerns on the fact that the proposed cascaded method relies on knowing exactly which frames are missing during evaluation. In this case, simply swapping between an audio-only model and audio-visual model can be a trivial solution. I agree with this reviewer to some degree. However, considering the follow reasons, I recommend acception but with some revisions. 

Reasons:
1. The contribution of this paper include both a systematic evaluation framework for robustness and a cascaded model to achieve robustness. The overall study and analysis on robustness is of some value to the community, even if there are some concerns on the proposed cascaded framework.
2. When knowing exactly which frames are missing during evaluation, I think simply swapping between an audio-only and audio-visual models is non-trivial. Assume that there are some frames with video missing, I am not sure whether we can simply concatenate the outputs of two models to form the final output, since the alignment between text output and audio/video input is not accurate enough. How to decide when to switch to another model? 

Revision:
1. The authors should revise the paper carefully according to each reviewer's comments. 
2. The authors should comprehensively discuss this issue (the proposed cascaded method relies on knowing exactly which frames are missing during evaluation) and propose some possible solutions. 

Decision: Accept (poster)