Parameter Efficient Multimodal Transformers for Video Representation Learning
Authors: Sangho Lee
Youngjae Yu
Gunhee Kim
Thomas Breuel
Jan Kautz
Yale Song
Venue: ICLR 2021 Poster (2021)
Keywords: Self-supervised learning
audio-visual representation learning
video representation learning

Abstract: The recent success of Transformers in the language domain has motivated adapting it to a multimodal setting, where a new visual model is trained in tandem with an already pretrained language model. However, due to the excessive memory requirements from Transformers, existing work typically fixes the language model and train only the vision module, which limits its ability to learn cross-modal information in an end-to-end manner. In this work, we focus on reducing the parameters of multimodal Transformers in the context of audio-visual video representation learning. We alleviate the high memory requirement by sharing the parameters of Transformers across layers and modalities; we decompose the Transformer into modality-specific and modality-shared parts so that the model learns the dynamics of each modality both individually and together, and propose a novel parameter sharing scheme based on low-rank approximation. We show that our approach reduces parameters of the Transformers up to 97%, allowing us to train our model end-to-end from scratch. We also propose a negative sampling approach based on an instance similarity measured on the CNN embedding space that our model learns together with the Transformers. To demonstrate our approach, we pretrain our model on 30-second clips (480 frames) from Kinetics-700 and transfer it to audio-visual classification tasks.

Published as a conference paper at ICLR 2021
PARAMETER EFFICIENT MULTIMODAL TRANSFORM- ERS FOR VIDEO REPRESENTATION LEARNING
Sangho Lee, Youngjae Yu, Gunhee Kim Scoul National University (sangho. lee, yj- yu) @vision. snu acu.ackkr gunhee@snu.ac.
Thomas Breuel, Jan Kautz
Yale Song
NVIDIA Research (tbreuel, jkautz ( @nvidia com
Microsoft Research ya yalesonggemcrosoffe
ABSTRACT
The recent success of Transformers the language domain has motivated adapting it to multimodal setting, where new visual model is trained in tandem with an already pretrained language model. However, due to the excessive memory requirements from Transformers, existing work typically fixes the language model and train only the vision module, which limits its ability to learn cross-modal infor- mation in end-to-end manner. In this work, focus reducing the parameters of multimodal Transformers in the context of audio-visual video representation learning. We alleviate the high memory requirement by sharing the parameters of Transformers across layers and modalities; we decompose the Transformer into modality-speciif and modality-share -hhared parts so that the model learns the dynamics of each modality both individually and together, and propose a novel parameter sharing scheme based on low-rank approximation. We show that our approach reduces parameters of the Transformers up to 97%, allowing to train our model end-to-end from scratch. We also propose a negative sampling approach based instance similarity measured on the CNN embedding space that model learns together with the Transformers. To demonstrate our approach, we pretrain model on 30-second clips (480 frames) from Kinetics- and transfer it audio-visual classification tasks.
INTRODUCTION
Learning multimodal representation from unlabeled videos has received considerable attention (Bal- Audio-visual learning is of particular interest due to the abundance of videos with natural audio-visie co-occurrence (Owens & Eiros] 2018 Owens et al.) 2018]: Arandhelovic
trušaitis al. 2018).
& Zisserman 2018 Ephrat al., 2018 Gao & Grauman, 2019 Alwassel al. (0.9)). However,
existing approaches learn localized representations from short videos (hundreds of milliseconds to just under a few seconds), capturing only short-term dependencies in data. While this is useful for certain applications, e.g., source separation (Ephrat et al. 2018)) and atomic action recognition (Gu al. 2018), learning representation that captures long-term dependencies is equally important, e.g.,
for activity recognition (Kay et al. 2017 Carreira et al. 2019 Sigurdsson et al. 2016). Unfortunately. processing long videos requires large memory resource and capturing long term dependencies is
long-standing problem (Hochreiter & Schmidhuberi 1997 Cho et al. 2014 Vaswani et al.) 2017).
In language understanding, strong progress has been made in large-scale learning of contextualized
language representations using Transformers ( (Vaswani et al. 2017 Howard & Ruder
[018]] Peters
ct al.) 2018 Radford et al. 20188 2019: Devlin et a 2019 Liu al. 2019 Yang et al. 2019). Riding on the success of Transformers, several recent works have extended it to the multimodal setting by adding an additional vision module to the Transformer framework (Sun et al. 2019b: Lu et al. 2019). However, these models typically not end-to-end trained; they rely BERT (Devlin et al,, 2019). which fixed throughout, and train only the visual components. While the pretrained BERT helps accelerate convergence and brings reliable extra supervision signal the
language- pretrained

---

Published as a conference paper at ICLR 2021
Parameter Sharing in Transformers
Multimodal Transformer: Layers
Layer
Layer
(n) Visual Transformer: layers
(b) Audio Transfrmerr: L layers
Visual
Audio
Multimodal
Transformer
Transformer
TransSormer
--
Color
Sharee
Figure 1: (Left) Our model consists of CNNs encoding short-term dynamics of each modality and Transformers encoding long-term dynamics of audio-visual information from videos. (Right) To alleviate excessive memory requirements, we propose an efficient parameter sharing scheme based on matrix decomposition with low-rank approximation, which allows to train our model end-to-end.
vision component, this partial learning setup can undesirable the text data comes from different distributions (of topics, dialects, or foreign languages) or if we want to apply it to different modalities (e.g., vidio-visual). Unfortunately, end-to-end training of such multimodal Transformer architectures challenging for most existing compute environments due to the excessive memory requirement.
In this work, we make three key contributions. First, we propose an end-to-end trainable bidirectional transformer architecture that learns contextualized audio-visual representations of long videos. Our model, shown in Figure [ consists of audio/visual CNNs, audio/visual Transformers, and a multi- modal Transformer. The CNNs operate short (e.g., one second) video clips and are intended to capture short-term dynamics within each modality. The Transformer layers operate on long video se- quences (e.g., 30 seconds), capturing long-term dynamics. To enable end-to-end training. we propose novel parameter reduction technique that shares parts of weight parameters across Transformers and across layers within each Transformer. We show that this results in up to 97% parameter reduction. enabling end-to-end training of our model, with minimal performance degradation. To the best of our knowledge, our work is the first to report end-to-end trained multimodal Transformers, and the first apply Transformers for audio-visual representation learning.
The quality of negative samples crucial in contrastive learning, which is part of our learning objective. As our second contribution, we propose contenl-aware negative sampling strategy that favors negatives sufficiently similar to a positive instance. Our approach measures the similarity by reusing the CNN embeddings obtained during model training, and thus do not introduce extra parameters to learn. We show that this improves performance over the standard sampling strategies.
Our third contribution is systematic evaluation of different modality fusion strategies. Existing works on multimodal BERT (all using vision-and- san-lang data) typically apply one fusion strategy without thoroughly comparing with alternatives, e.g., some works perform early fusion (Sun et al.) 2019b (S et al. 2020) while others perform mid-level fusion (Lu et al. 2019 Tan & Bansal 2019). As result, it is unclear how different fusion methods affect the final performance. In this work, compare three fusion strategies (carly, mid, late) and show the superiority of mid-level fusion.
To demonstrate our approach, we pretrain our model on long (30-second) video clips from Kinetics- 700 (Carreira et al., (019) and finetune it on various video classification tasks. One benefit of the modular design of our architecture flexibility: once pretrained, we can use any of the subnetworks for downstream tasks depending on the modalities involved (audio-only, visual-only. audio-visual. and video lengths (short and long). To show this, we evaluate our model on UCF101 (Soomro) et al., 2012) and ESC-50 (Gemmeke et al. 2017) for short-term visual/audio classification, and Charades (Sigurdsson et al.] (0066) and Kinetics-Sounds (Arandjelovic & Zisserman, 2017) for long-term audio-visual action recognition.
APPROACH
Figure shows an overview of the proposed model architecture. The input to our model is sequence of visual clips VI.T and the corresponding sequence of audio streams a For example, each
2

---

Published as a conference paper at ICLR 2021
sequence is a 30 second-long video divided into 30 non-overlapping clips (each clip is one second long). We divide our model into three parts with different characteristics, which are explained below.
Local Feature Embedding. We feed each of T video clips to visual CNN fv to obtain xi.T E IR''D and each audio stream to an audio CNN (a() to obtain x E I TRTXDI Intuitively, the CNN outputs are temporally local embeddings as they have access only short-range temporal window of the entire video sequence. Thus, they are suitable for representing short-range atomic actions (e.g., sit down, raise arms) that constitute long-range events (e.g., gym workout). We use the SlowFast network (Feichtenhofer et al. 2019) with a ResNet-50 backbone (He et al., (016) as a visual CNN fy. and a ResNet-50 50 as an audio CNN fA- The weights of both CNNs are randomly initialized and trained end-to-end with the Transformer layers.
Unimodal Contextualized Embedding. The local feature embeddings capture short-term dynam- ies but lack long-term contextual information. We use Transformers (Vaswani et al. (017) to enrich the embeddings with sequence-level context. We start by learning unimodal contextualized representations using the visual Transformer gv and the audio Transformer ya, respectively.
The Transformer consists of L layers, each with two sub-layers: a multi-head attention layer and a feed-forward layer. Given input sequence of embeddings x E RRTxD and A attention heads, the j-th head in the attention layer computes the output embedding sequence a, E RRTx,, , = D/A as
aj- = -softmxx
Q - -==xW3 K x V = X
(I)
where W W W E DxY are weight matrices for computing the (query, key, value) triplet given the input x. This operation is repeated for each attention head, and the outputs are combined (with concatenation followed by one linear layer with weights W E DDDD), producing a E RT'D Next, the feed-forward layer takes this intermediate output and computes o E 2 using a two- layer fully-cconected network with weights W E PxF and Wd FXD. The output of each sub-layer is computed using a residual function followed by layer normalization (Ba et al. 2016). i.e., LayerNorm(z (xr + Sublayer(z)). In this work, we set the number of layers /. = 6, the number of attention heads A = 12, the feature dimension D = 768 and the intermediate dimension F = 3072. For simplicity, we use this design for all layers across all three Transformers in our model.
Before feeding local embeddings x" and x to unimodal Transformers, augment them with "positional" embeddings. Specifically, we append to the beginning of each sequence a special vector BOS (beginning of sequence), i.e., xi for visual and x5 for audio streams; their dimensions are same as x and x", respectively. We also define positional embeddings Pu:r encoding time indices (we call this "time" embedding). This is necessary to preserve information about temporal ordering of local feature embeddings, which is otherwise lost in Eqn. T
We combine them via layer normalization,
u = LayerNormxx" + P u = LayerNorm(x} + Pi), Vt E T]
(2)
We initialize (x0, x6: Po.r ) to the normal distribution and train them with the rest of the model.
We feed the augmented visual embeddings into the visual Transformer gv and obtain füe gv (w and similarly obtain yö.r = ga (ugz)). The embeddings at each time step has a direct access to the entire input sequence regardless of their position (it has one-step signal path during forward and backward inference). Multiple layers of such feature transformation thus allow the resulting embedding to be deeply contextualized in the time dimension. We denote the output embeddings corresponding to the BOS positions by BOSY = yö and BOS; = y8. and designate them as the summary embeddings representing the sequence of each modality.
Multimodal Contextualized Embedding. The unimodal embeddings capture long-term temporal context but miss out on cross-modal information. The final step in forward inference is to use multimodal Transformer Av to obtain embeddings contextualized in the audio-visual space.
We first augment the embeddings yor and yor with modality and time embeddings. The modality embeddings m" and m" are vectors of the same dimension as y and yf, respectively. We share m² (and m²) across all the unimodal embeddings yorr (and yo:r); thus, they add modality-discriiiaan information to the Transformer. We also add time embeddings po:r as before; however, unlike in
For notational simplicity, we drop the subscripts refer the entire sequence unless distinction is necessary.
3

---

Published as a conference paper at ICLR 2021
the previous step, share the same Po:r between embeddings from the two modalities to correctly indicate the time indices. We augment the modality and time embeddings via layer normalization,
W LayerNorm(y, + +P: + m"), w = LLyerNorm(y, + Pt + m²), Vt E [0, T]
(3)
We feed the augmented visual embeddings w and audio embeddings w to the multimodal
Transformer Av,, one after another, and obtain +2:++++1) hv (v ([ w Worl) ). We again denote
the output embeddings corresponding to the BOS positions by BOST; = zī(= 2o) and BOS; z
zr+1), and use them as summary embeddings encoding multimodal context.
We emphasize the importance of feeding w and w after another. An alternative would be concatenating them before feeding them to Avv and obtaining an output Zo:T (instead of (0)(2++1) However, this restricts the Transformer time slices, which could be problematic when there is a temporally asynchronous relationship between the two modalities (e.g., visual clip matches with sound captured few times steps before) (Kazakos et al., 2019; Morgado et al. 2020). By arranging the two sequences one after the appropriate audio- visual embeddings in an asynchronous manner. Another practical concern with the alternative approach is that it significantly increases the model size; the weight matrices Wg, Wk: W, grow quadratically with the input feature dimension D. Serializing the input resolves both issues.
access audio-visual embeddings only from the same
other, the Transformer can mix-and mox--nd--atee
2.1 SELF-SUPERVISED PRETRAINING OBJECTIVES
Task 1: Masked Embedding Prediction (MEP). BERT (Devlin al. 2019) is trained using the masked language model (MLM) task, which randomly selects input tokens and replaces them with mask token. The model is then trained to predict the original (unmasked) tokens by solving classification task with a cross-entropy loss. However, inputs to our model are real-valued audio-
visual signals (rather than discrete tokens) P so applying the MLM task requires input discretization, which causes information loss (Lu et al., 2019; Sun et al., 2019a). We instead train our model to identify the correct visual clip or audio stream compared set of negative samples in a contrastive manner, which does not require input discretization.
We formulate our MEP task using InfoNCE (Oord et al. 2018 which is the softmax version of the noise contrastive estimation (NCE) (Gutmann & Hyvärinen, 2010). Let o be the t-th output of any of the three Transformers obtained by masking the f-th input x,. Our InfoNCE loss is then defined as
1(x,, ot)
(xC ō) = -Ex Σ
log I(x,, o,) + Ezenego I(x;, ō,)
(4)
where neg(t) are negative sample indices and the compatibility function I(x,, o,) is,
I(xe, o.) = exp (FFN (o:)Wix:),
(5)
where W, E PRDx (P= 256) and FFN is two-layer feed-forward network. The use of a non-linear
prediction head has shown to improve the quality of the representations learned in a contrastive
learning setup (Chen et al. | 2020); following the recent work in Transformers (Devlin al. 2019 Liu
et al. 2016 in FFN. Optimizing Eqn. 4 enforces I(x,, o,) approximate the density ratio p(x. pix_ [o.).
2019 Lan et al. 2020), we use GELU non-linear activation function (Hendrycks & Gimpel
this
p(x]]
can be seen as maximizing the mutual information between x, and ö, (Oord al., 2018). Intuitively, this encourages the Transformer to capture the underlying dynamics of x from each modality without explicitly learning generative model lo.)
Negative sampling. We find that a good negative sampling strategy is essential for the model's convergence. Existing approaches either use all but x, (positive) within a mini-batch as negative samples or limit it to the current sequence only. However, both these methods ignore the data content and thus can miss useful negatives. Oord al. (2018) showed that leveraging prior knowledge about data can improve the negative sample quality (e.g., by sampling negatives from the same speaker as the positive). Unfortunately, such prior knowledge is often not available in unlabeled videos.
II the form of RGB images and log-mel-scaled spectrograms.
4

---

Published as a conference paper at ICLR 2021
Transfommer
Transformer N
Transfommer
TranSoomerr
Transfomnnee
Transfomer
Transfommrt N
Tayer
Layer 2
Layer
(a) shuring
Cross-layer sharing [37]
All sharing
Ours
Comparison of parameter sharing schemes. Ours combines (b) and (c) but decomposes Figure 2:
weights in each layer into private and shared parts so only the latter is shared across Transformers.
We propose a content-aware negative sampling strategy that favors negatives sufficiently similar to positive instance in the CNN embedding space; call our approach CANS-Simi misiiiir....... Our approach is inspired by [yynnov et al. (2018) who showed that randomly initialized CNNs provide a strong prior over natural images due the inductive bias already built into the design of the CNNs. This suggests that our local feature embeddings x" (and x") can capture the underlying statistical regularities in video clips (and audio streams) right from the beginning, which can be sufficient to assess the similarity/dissiiiiaan approximate content dissimilarity well (and this will improve as the training progresses).
between clips. Therefore, the distance measured on them can
Motivated by this, we sample the negatives based on local feature embeddings x" (and x"). Specif- ically, we compute pairwise & distance between x; (positive) and all other instances within mini- and normalize them to the [O, 1] interval. To remove samples that are either too similar or too different from the positive sample, we discard instances that fall outside the 95% confidence interval in the normalized distance space. We then sample the negatives from the remainder using the normalized distance as sampling probability. This makes instances similar to the positive instance have more chance to become negatives. We emphasize the importance of sampling, instead of deterministically taking top most similar samples; the stochasticity allows our model to be robust to potentially inaccurate distance estimates because samples with low probabilities will still have chance to be selected as negatives.
Finally, our MEP loss is the InfoNCE loss computed all three Transformers,
LMEP Lxcu(x", 5²)) Lxce(x", y") + (xcx((x"; x"],
(6)
Task 2: Correct Pair Prediction (CPP). The MEP task encourages our model learn the underlying dynamics within each modality. To help our model learn mrosl dynamics, we design task that predicts whether a pair of audio-visual embeddings is from the same video. Specifically, we define two binary classifiers, one for the two unimodal Transformers and another for the multimodal Transformer. Each classifier takes as input either S, = [yö: yöl (and [z5: zfi), a pair of audio-visual "summary" embeddings corresponding to the BOS positions, S₁2 yyf (or [z," zf]), the output
embeddings sampled at random positions (we take two random positions E [1, TT). The classifier
predicts p( cls) indicating whether the pair is from the same video (c 1) from different videos (c = 0). We train the classifiers with a binary cross-entropy loss,
LCPP -xx, [c- logg(c(s,) c- log p(c[s:)]
where is the inner product. We generate a random derangement of the input mini-batch so that the number of positive and negative pairs are guaranteed to be the same.
Overall Pretraining Objective. We train model end-to-end from scratch by optimizing CMEP+ aLCPP with a balancing term a. We find our model is insensitive to this term, so we set a 1.0.
2.2 PARAMETER REDUCTION
Optimizing our model is challenging due to the large memory requirement. The most expensive part is the Transformers, which take up 82% model parameters. One could reduce the model size by making the Transformers shallower, but the depth of Transformers has shown to be crucial to get good performance (Devlin et al.) (201)). We propose to reduce the model size by aggressively sharing parts of weights across Transformers as well as layers within each Transformer (see Figure (d))
Sharing across Transformers. We first consider sharing weights across Transformers. Each Trans- former encodes data coming from different distributions: gv encodes x", Ja encodes xa, and
5

---

Published as a conference paper at ICLR 2021
hAv encodes (y",y") y These input distributions may each exhibit different dynamics, yet to- gether share certain regularities because they all come from the same videos. Motivated by this, we decompose Transformer weights into shared and private parts that different patterns can be learned in a parameter-efficien manner. Recall that each layer of a Transformer contains weights {W", W W W We W We decompose each of these weights into W UEVT, where W E RMMN U E Mx0x - E g0x0, .V E RN We perform low-rank approximation of W by setting the rank O < M, and share U across Transformers while keeping Σ and V private each Transformer. This helps reduce parameters because MO + 3(0² + NO) < 3M N. We experimented with different matrix ranks O but the differences were small; we set = 128 (M, N 768 or 3072).
The decomposition converts a linear projection of input Wx into series of (unconstrained) linear projections UEVTX x. However, this can cause numerical instability during optimization (Nocedal & Wright, 2006). We could perform the Singular Value Decomposition (SVD) over W so that it performs rotation (V stretch (E), and rotation (U) with orthogonal basis vectors in U/ and V. Unfortunately, solving the full SVD has a computational complexity of O (mmxx(M 2) (Golub Van Loan 2012). Here, we put an orthogonality constraint only on Σ and perform projection (V ), rotation (E), and projection (U) of input x. In addition, put V x in unit sphere (via Ey- normalization) before rotating it with Σ. This not only improves numerical stability, but also removes magnitude information in V x and keeps angular information only, which has been shown to provide sample discriminative information (Chen et al.) 2019a). To impose the orthogonality constraint on Σ, we use the Padé approximation with scale- squaring trick of Lezcano--oasss (2019). Intuitively, we linearly project x onto a unit sphere (V x) and rotate it (EV x) in each Transformer so that captures the dynamics of each input distribution independently. We then project it to the shared space via U, capturing shared regularities across all three Transformers.
& Martinez-Rubio
Sharing across Layers. Recently, (Bii et al. (2019a) showed that sharing parameters across layers in deep neural networks does not hurt the representational power of the network. Furthermore, (Lan et al. 2020) demonstrated that cross-layer parameter sharing in the Transformer leads to a lighter and taster-to-train model without sacrificing the performance on various language understanding benchmarks. Motivated by this, we let each Transformer share parameters across different layers.
EXPERIMENTS
We pretrain our model on Kinetics-700 (Carreira et al. 2019) or AudioSet (Gemmeke et al., 2017) and finetune it various downstream tasks. The official release of Kinetics-700 contains 10-second clips only, so we download 410K original videos from YouTube and take 30-second clips from each video. For fair comparison with prior work, we use 10-second clips from the official release of AudioSet (we used 1.8M clips). We pretrain our model on 64 NVIDIA Tesla V100 GPUs with batch size of 256 for 220K iterations. For downstream tasks, we evaluate on short- video/addooo classification using UCF-101 (Soomro et al. 2012) (13K clips from 101 classes; 7.2 seconds on average) and ESC- 50 (Gemmeke et al.) 2017) (2K clips from 50 classes; 5 seconds), and on long-video classification using Slnetics-Sounde (Arandjelovic & Zisserman 2017) (23K videos from 32 classes; 10 seconds on average) and Charades (Sigurdsson et al.] 2016) (10K vidcos from 157 classes; 30 seconds on average). We describe various details about experimental setup in Appendix.
3.1 RESULTS AND DISCUSSION
Multimodal Fusion Methods. To evaluate different fusion methods on the quality of learned representation, we test the following settings: (i) Early uses a single multimodal Transformer with 2 x I. layers, (ii) Mid is our approach described in Figure [1 (iii) Late uses two unimodal Transformers each with 2 x T. layers. All the methods are pretrained on audio-visual data using CPP and MEP losses, except for (iv) -wwaaaatt---eeeee the pretrained models on audio-visual, audio-only, and visual-only scenarios. For fair comparisons across different fusion methods, do not perform parameter sharing in this ablation setting.
where we use only the MEP loss. We finetune
Table I (a) shows that Early and Mid outperform Late on the audio-viste scenario. This suggests the importance of encoding cross cross-modal information. Note that Late-w/ WCCPP gets modal sel--supervision, which gives marginal performance improvement over Late-w/c -w/o-CP; however, both methods miss the opportunity encode any cross-modal relationship, leading inferior results.

---

Published as a conference paper at ICLR 2021
Fusion Method
Audio Visual
Audio-only -/-
Visual-only -/-
bJ Sampling Methed
top-1
Top-5
Sarly
64.97890 89.8
Current Bequence
64.6
89.8
Late w/ CB2
52.3/808
41.0/113
Current MiniBatch
90.8
Late-w/c-CaP
60.6/ 87.6
50.5 / 79.9
40.7/71.7 42.5
VANS-Dissimill
66.2
91.1
Mid
65.7 89,9
53.5 / 82.7
CANS Similar"
67,5
92.3
C) Model
X.-T
Params
top-175
d) Model
X-L
X-I
Params
top1//5
Molt hi-ti-2
7M
60.3/88.9
Vis
14M
41.47100
Multi-6
X
21M
65.7
Vi5-2
5
x
7M
41.2/72.9
MulLi
((A11)
67.1/92.3
Vis 6
43M
43.8/74.2
t1-6
ViPart
4M
67.5/92.3
Vis-E
x
7M
43.5/73.7
Table I: Ablation study on Kinetics- Sounnd comparing: (a; top-left) multimodal fusion methods, (b; top-right) negative sampling strategies, and (c & d; bottom) parameter sharing schemes. X.-L: Cross-layer, X.-T: Cross-Tree Tooss-Transfommr sharing. We report top-1 and top-5 accuracy (%).. T: Ours.
While both Early and Late perform similarly in the audio-visual scenario, only Late can be used in unimodal downstream scenarios (c.f., Early requires the presence of both modalities). This has practical implications: Mid and Late can effectively handle missing modalities, i.e., once pretrained on audio- audiol data, we can use it on any audio-visual audio-only, and visual-only scenarios. Our Mi d fusion approach enjoys both the advantages, i.e., learning cross-modal relationship and being robust missing modalities, achieving overall the best performance.
Negative Sampling Strategies. We compare four strategies: (i) Current-Sequent takes all but the positive instance from the same sequence as negatives, (ii) Current MeazenttMiniin MiniBatch takes all but the positive instance in the mini-batch as negatives; this subsumes Current Seerrent-Sequee Sequence, (iii) CANS-Dis ssim aar stochastically samples negatives using a modified version of our content- aware negative sampling (CANS) that favors dissimilar samples, and (iv) CANS-Simil Similar is our proposed CANS approach that favors negatives that are similar to the positive instance.
Table (b) shows Current- Sequence is the least effective: It makes MEP too difficult because negatives are (sometimes too much) similar to positives. As a result, the training dynamics is dominated by CPP, which is relatively easier, leading inferior performance. We make quite the contrary observations from Current Sonntt-iniiiie MiniBatch. the inclusion of negatives from different videos makes MEP easier and thus makes it dominate the training dynamics. Our CANS approach solves both these issues by eliminating negatives that are either almost identical to or trivial to distinguish from the positives, based on the 95% CI over the CNN embedding distances. It also samples negatives in stochastic manner so a wide variety of samples can be included as negatives. Our proposed CANS-51 a r can be considered as a "softened" version of Current - it samples negatives that are similar to positives with high probability (this can be considered online hard negative mining), but it also takes instances from different videos with a lower probability. This balances out hard and easy negatives, making the MEP task effective.
Parameter Sharing Schemes. Our parameter reduction scheme reduces the number of parameters from 128M to 4M (by 97%) (Table (c)). We reduce the model size by sharing weights across Transformers and across layers. We validate these ideas in two sets of experiments. Table [ (c) compares cross- cross-Transfer Transforrrr weight sharing schemes. We Multi-6 that uses all three Transformers with 6 layers each, and compare four methods that correspond to Figure (a)-(d). Note that No sharing is too large fit in a Tesla V100 GPU (16GB) even with samples, we define Multi 2 that uses three Transformers with 2 layers each, and with the reduced number of attention heads A 5, the feature dimension D to 320 and the intermediate dimension E 1280. We see that our proposed approach, Part, achieves the best performance with the least number of parameters. One might ask how Part leads to a smaller model when All 1 shares all the weights across Transformers: We decompose weights W = UVV with low-rank approximation and share only across Transformers, while the ΣVT part learns modality-speeifi dynamics. Table 1 (d) compares cross-layer weight sharing schemes using the visual Transformer with either 2 (Vis-22 or (Vis-6) 6) layers. The results show that sharing weights across layers does not hurt the performance, confirming the observations by Lan et al. (2020) in the audio-visua setting.
Pretraining Objectives. To evaluate the importance of MEP and CPP tasks, we test two settings: (i) Mid-w/o-CPP and (ii) Mid-w/o-MB On Kinetics-Soundl these achieve 65.9% and 64.6%, respectively; ours achieve 67. 67.5% (top-1 accuracy). The result show that the MEP task plays an important role during pretraining, confirming the findings from Sun al. (2019a) that the InfoNCE
7

---

Published as a conference paper at ICLR 2021
Model
Net
Data
UCF
Model
Ner
Data
ESC
Model
Charades
KS
ST-Puzzle
3D-R18
K400
65.8
SVM
MCP
39.6
Random
5.9
ClipOrder
R(2+DD
UCF
72.4
ConvAE
CNN-4
18.3
DPC
3D-R34
K400
75.7
RF
MLP
44.3
ATF (OF)
22.4
CBT
S3D
79.5
Conv-Net
CNN
V- CNN
18.7
43.8773.3
MultiSens
3D-R18
AS
82.1
SoundINet
CNN-8
FS
74.2
A-COI
18.9
49.4/76.9
AVTS
MC3-18
K400
85.8
L.-Nee
CNN.8 8
FTS
79.3
M CNN
23.1
59.4/83.6
AVTS
MC3-18
89.0
DMC
VGG-ish
79.8
V-BERT
26.0
49.5/78.9
V-UKN
SlowFast
K700
85.2
AVTS
VGG-M
80.6
A BERT
27.4
/8.9
V CKN
FlowFFas
AS
86.1
A CNN
R50
AS
81,5
M-BERT!
29.5
75.6 / 94,6
Datases,
Kimetcs. AS: AudioSet.
SoundNei, KS: Kinetics Sounds Baxelines Puzze
2010 ClipOrder
Xu
2019
DPC [Ban RE
2019.
CBT
Sun e all 20 MuliiSens (Owens & Efros[2013) AVTS (Karbur et 2018. AE Dotar st
[2016] LVM (Sscaal
Przza
ComoNet
Piezak: 2015s SoundNat AAyter al. p016). -.a-Ner Arandjslonic Zisserman
2017
DMC
Hu 2019)
(Siuuddooon
2017
Table 2: (a; left): Short video classification results on UCF101 (mean accuracy (%)). (b; cen-
ter): Short audio classification results on ESC-50 (mean accuracy (%)). (c; right): Long video classification results on Charades (mAP) and Kinetics-Sounds (KS; top- 1/5 accuracy (%)). ( Ours.
loss, as deployed in CBT, is effective in the cross-modal setting. The result also shows that augmenting MEP with CPP provides further performance improvement by learning cross-modal correspondence.
Downstream Evaluation. We pretrain our model with Mid fusion using MEP and CPP tasks (with and employ Part weight sharing. We use either Kinetics-700 AudioSet for fair comparisons with prior work. Table 2 (a)/(b) shows short-video/audio classification results on UCF-101/ESC-50. For fair comparisons to the baselines, we use only the visual/audio CNN (no Transformers): we finetune linear classifier on top of the visual CNN end-to-end for UCF-101, and train a multi-class one-vs-all linear SVM on top of the fixed audio CNN for ESC-50. Although our model is pretrained on long video clips with no direct supervision to the CNN layers (gradients must flow through Transformers), outperforms most of the baselines (except for AVTS on UCF-101) that received direct supervision from short video clips. We note that, similar to ours, CBT (Su et al., 2020) is a multimodal Transformer pretrained on long video clips and thus is the most meaningful comparison to ours; ours outperform CBT on UCF-101 by 5.7%. For sound classification, our approach outperform all existing published results.
(c) shows video classification results Charades and Linetics-Soundee (KS) when pre- trained on Kinetics-700. We test Vi Visual-only (V),Audo Audio-only (A), , and Mult imodal (M) settings to verify the benefit of multimodal learning. Because there is no published self- supervised learning results on these datasets, we demonstrate long-term representations by comparing CNNs (CNN; short-term) to Transformers (BERT; long-term) on KS that contains 10-second clips. Since CNNs process 1-second clips, we feed 10 non-overlapping clips to CNNs and average the pre- diction output. In all settings. we add a 2-layer MLP with softmax classifier on top. The results show that Transformers outperform CNNs on Linetics-Sounds suggesting the superiority of long-term representations. We also see that combining audio-visual information performs the best. We notice that audio representations are generally stronger than visual representations; we believe that learn- ing discriminative visual representations is generally more challenging. especially when the CNNs receive (self- Jsupervision signals only indirectly through Transformers. We believe that providing seeff-)supervision directly to CNNs, e.g., by first pretraining CNNs on 3D rotation prediction (Jing et al. 2018) and then jointly training the whole model (as was done in CBT (Sun al. 2019a)), could further improve performance. Incorporating contrastive learning (Chen et al. 2020) over the CNN embeddings and training the whole model end-to-end is another promising direction for future work.
4 RELATED WORK
Multimodal BERT. Extending BERT (Devlin et al.
2019) to vision-and and wand-language has been actively
studied. Existing work typically adopt early fusion (Li et al. 2019b: Li et al.] 2020; Zhou et al., 2020 Su et al.] 2020 C Chen et al. 2019b Zhu & Yang 2020) or
2019 Alberti et al. 2019 Sun et al..
mid fusion (Tan & Bansal 2019 Lu et al. 2019 Sun et al. 2019a Luo et al. 2020) without thorough
validation, and they train only visual components while relying on a Iinuuage-pretraine
BERT.
Although there have been some efforts to leverage the Transformer architecture (Vaswani et al. 2017) for audio and visual inputs (Boes& & Van hamme 2019: Tian et al.) 2020), our approach is the first to demonstrate multimodal audio-visual BERT trained from scratch in an end-to-end manner. This is enabled by our novel parameter reduction technique, which is one of our main technical contributions.
8

---

Published as a conference paper at ICLR 2021
Audio-Visual Learning. Early work in audio-visual learning focused speech signals, improving
audio-visuue speech recognition than unimodal approaches (Ngiam et al. 2011; Srivastava & Salakhut dinov 2012). Recent approaches leverage unlabeled videos from specthic domains (Owens et al.
2016 Gao & Grauman 2019 Zhao et al. 2018 Ephrat et al. 2018, Alwassel et al., 2019) Miech et al.
2020 Piergiovanni al. 2020) and often demonstrate on audio-visual source separation, localization, and co-segmentation. However, these approaches rely on short-term audio-visual correspondence and thus may not generalize long-term video recognition that requires global context (as was suggested in (Hjelm et al. 2019)) ) which this work focuses on.
Parameter Reduction. Network pruning (Reed, 1993 Caron al.. 2020) trains a large model and then reduces its size while maintaining performance. Reducing the size of CNNs for mobile
applications an active research area (Rastegari et al. 2016 Howard al., 2017 2019 Zhang et al., 2018) et al. 2016). Our work is closely re lated to the work that shares parameters across layers deep neural networks. Trellis network (Bai et al. 2019b) a temporal convolutional architecture with weight-tying across time and depth. Similar to ours, Universal Transformer (Dehghani et al. 2019). RSNMT (Dabre & Fujita 2019), DEQ (Bai et al. 2019a), ALBERT (Lan et al. (0200 share weights across layers in Transformers. We combine this idea with novel cross- Tros-Trrasffomer weight sharing, which decomposes weight matrices with low-rank approximation.
Negative Sampling. Hard negative mining has been shown to be crucial for contrastive Icarn- ing (Arandjelovic & Zisserman, 2017, 77, Owens & Efros 2018 Korbar et al., 2018 [Schroff et al.) 2015 [015|/Zhang 2019) Morgado et al.) [220)] Wu et al.] (020). Korbar al. (2018) use the time difference between clips to approximate clip similarity (i.e., clips that are further apart are deemed more different). However, such an assumption may not hold for real-world videos, e.g. periodic actions such as push-ups. Unlike this line of approaches, we directly use the feature embeddings learned by our model. Several apparoaches adapted a similar idea (Schroff' et al.) 2015 Zhuang et al.) 2019) Morgado et al., 2020; Wu et al. (0200). Different from prior work, we bring the stochasticity to sampling procedure by using the content similarity as the sampling probability; this helps reduce potential errors especially during the early stage of training.
CONCLUSION
We introduced a multimodal bidirectional Transformer architecture for self-supervised learning of contextualized audio-visual representation from unlabeled videos. Our main technical contributions include: (1) we propose parameter efficient multimodal Transformers based on matrix decomposi- tion with low-rank approximation; (2) we propose a novel content-aware negative sampling technique for contrastive learning. We demonstrate a successful end-to-end training of multimodal Transformers for audio-visus learning (which is, to the best of our knowledge, the first time in the literature). We also report comprehensive evaluation of various design decisions in multimodal learning.
Acknowledgements. This work was partially supported by Institute of Information & communi- cations Technology Planning & Evaluation (ITTP) grant funded by the Korea government (MSIT) Video Turing Test, N0.2019-0-01082, SW StarLab) and the international coopera- tion program by the NRF of Korea NNRE2201822AA2AAI10000
REFERENCES
Chris Alberti, Jeffrey Ling, Michael Collins, and David Reitter. Fusion of Detected Objects in Text for Visual Question Answering. In EMNLP-IJCNLP. 2019.
Humam Alwassel, Dhruv Mahajan, Lorenzo Torresani, Bernard Ghanem, and Du Tran. Self- Supervised Learning by Cross-Modal Audio-Video Clustering, arXiv preprint arXiv: 1911.12661 2019.
Relja Arandjelovic and Andrew Zisserman. Look, Listen and Learn. In ICCV, 2017.
Relja Arandjelovic and Andrew Zisserman. Objects that Sound. In ECCV, 2018.
Yusuf Aytar, Carl Vondrick, and Antonio Torralba. SoundNet: Learning Sound Representations from Unlabeled Video. In NeurIPS, 2016.

---

for Published as a conference paper at ICLR 2021
Jimmy Lei Ba, Jamie Ryan Kiros, and Geoftrey E Hinton. Layer Norrmalizaannnn In NeurlPS Deep Learning Symposium, 2016.
Shaojie Bai, J. Zico Kolter, and Vladlen Koltun. Deep Equilibrium Models. In NeurIPS, 2019a.
Shaojie Bai, Zico Kolter, and Vladlen Koltun. Trellis Networks for Sequence Modeling. In ICLR, 2019b.
Tadas Baltrušaitis, Chaitanya Ahuja, and Louis- Phillipp Morency, Multimodal Machine Learning: Survey and Taxonomy. IEEE Transactions on Pattern Analysis and Machine Intelligence, 41(2): 423 443, 2018.
Wim Boes and Hugo Van hamme, Audiovisual Transformer Architectures for Large-Scale Classifica- tion and Synchronization of Weakly Labeled Audio Events. In ACM MM, 2019.
Mathilde Caron, Ari Morcos, Piotr Bojanowski, Julien Mairal, and Armand Joulin. Pruning Convolu- tional Neural Networks with Self-Supervision. arXiv preprint arXiv:2001.03554 2020.
Joao Carreira, Eric Noland, Chloe Hillier, and Andrew Zisserman. A Short Note on the Kinetics-700 Human Action Dataset. arXiv preprint arXiv: 1rxiv:197.009990 2019.
Beidi Chen, Weiyang Liu Animesh Garg, Zhiding Yu, Anshumali Shrivastava, Jan Kautz, and Anima Anandkumar. Angular Visual Hardness. In ICML Workshop Identifying and Understanding Deep Learning Phenomena, 2019a.
Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A Simple Framework for Contrastive Learning of Visual Representations. arXiv preprint arXiv:2002.05709, 2020.
Yen-Chun Chen, Linjie Li, Licheng Yu, Ahmed El Kholy, Faisal Ahmed, Zhe Gan, Yu Cheng, and Jingjing Liu. UNITER: UNiversal Image-TExt Representation Learning. arXiv preprint arXiv: 1740, 2019b.
Kyunghyun Cho, Bart Van Merriënboer, Caglar Gulcehre, Dzmitry Bahdanau, Fethi Bougares, Holger Schwenk, and Yoshua Bengio. Learning Phrase Representations using RNN Statistical Machine Translation. In EMNLP, 2014.
Raj Dabre and Atsushi Fujita. Recurrent Stacking of Layers for Compact Neural Machine Translation Models. AAAI, 2019.
Mostafa Dehghani, Stephan Gouws, Oriol Vinyals, Jakob Uszkoreit, and Eukasz Kaiser. Universal Transformers. In ICLR, 2019.
Jacob Devlin, Ming-Wer Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. In NAACL-HLT, 2019.
Ariel Ephrat, Inbar Mosseri, Oran Lang. Tali Dekel, Kevin Wilson, Avinatan Hassidim, William Freeman, and Michael Rubinstein. Looking to Listen at the Cocktail Party: A Audio-Vissee Model for Speech Separation. ACM Transactions on Graphics, 37(4), 2018.
Christoph Feichtenhofer, Haoqi Fan, Jitendra Malik, and Kaiming He. SlowFast Networks for Video Recognition. ICCV, 2019.
Ruohan Gao and Kristen Grauman. 2.51D Visual Sound. In CVPR, 2019.
Jort F. Gemmeke, Daniel P. W. Ellis, Dylan Freedman, Aren Jansen, Wade Lawrence, Channing Moore, Manoj Plakal, and Marvin Ritter. Audio Set: An Ontology and Human-Labeled Dataset for Audio Events. In ICASSP, 2017.
Gene H Golub and Charles F Van Loan. Matrix Computations, volume 3. JHU press, 2012.
Chunhui Gu, Chen Sun, David A Ross, Carl Vondrick, Caroline Pantofaru, Yeqing Li, Sudheendra Vijayanarasimhan. George Toderici, Susanna Ricco, Rahul Sukthankar, et al. AVA: A Video Dataset of Spatio-tyyporally Localized Atomic Visual Actions. In CVPR, 2018.
10

---

Published as a conference paper at ICLR 2021
Michael Gutmann and Aapo Hyvärinen. for Unnormalized Statistical Models. In AISTATS, 2010.
Estimation: A New Estimation Principle
Tengda Han, Weidi Xie, and Andrew Zisserman. Video Representation Learning by Dense Predictive
Coding. In ICCV Workshop on Large Scale Holistic Video Understanding, 2019.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep Residual Learning for Image Recognition. In CVPR, 2016.
Dan Hendrycks and Kevin Gimpel. Gaussian Error Linear Units (GELUs). arXiv preprint
1606. 2016.
Devon Hjelm, Alex Fedorov, Samuel Lavoie Marchl-a Karan Grewal, Phil Bachman, Adam Trischler, and Yoshua Bengio. Learning deep representations by mutual information estimation and maximization. In ICLR, 2019.
Sepp Hochreiter and Jürgen Schmidhuber. Long Short-Term Memory. Neural computation, 9(8):
1997.
Andrew Howard, Mark Sandler, Grace Chu, Slang-Chich Chen, Bo Chen. Mingxing Tan, Weijun Wang, Yukun Zhu, Ruoming Pang, Vijay Vasudevan, et al. Searching for MobileNetV3. ICCV, 2019.
Andrew G Howard, Menglong Zhu, Bo Chen, Dmitry Kalenichenko, Weijun Wang, Tobias Weyand, Marco Andreetto, and Hartwig Adam. MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications. arXiv preprint 1rXiv:100..04811, 2017.
Jeremy Howard and Sebastian Ruder. Universal Language Model Fine-tuning for Text Classification. In ACL, 2018.
Di Hu, Feiping Nic, and Xuelong Li. Deep Multimodal Clustering for Unsupervised Audiovisual Learning. In CVPR, 2019.
Forrest N landola, Song Han, Matthew W Moskewiez, Khalid Ashraf, William Dally, and Kurt Keutzer. SqueezeNet: AlexNet-level accuracy with 50x fewer parameters and <0.5MB model size. arXiv preprint 4602:0 2016.
Longlong Jing, Xiaodong Yang, Jingen Liu, and Yingli Tian. Self-Supervised Spatiotemporal Feature Learning via Video Rotation Prediction. arXiv preprint arXiv: 1387, 2018.
Will Kay, Joao Carreira, Karen Simonyan, Brian Zhang, Chloe Hillier, Sudheendra Vijayanarasimhan, Fabio Viola, Tim Green, Trevor Back, Paul Natsev, al. The Kinetics Human Action Video Dataset. arXiv preprint 1mXiv:1705.06955 2017.
Evangelos Kazakos, Arsha Nagrani, Andrew Zisserman, and Dima Damen. EPIC-Fusion: Audio- Visual Temporal Binding for Egocentric Action Recognition. In ICCV, 2019.
Dahun Kim, Donghyeon Cho, and So Kweon. Self-Supervised Video Representation Learning with Space- Time Cubic Puzzles. In AAAI, 2019.
Bruno Korbar, Du Tran, and Lorenzo Torresani. Cooperative Learning of Audio and Video Models from Self-Supervised Synchronization. NeurlPS, 2018.
Zhenzhong Lan, Mingda Chen, Sebastian Goodman, Kevin Gimpel, Piyush Sharma, and Radu Soricut. ALBERT: A Lite BERT for Self-supervised Learning of Language Representations. In ICLR, 2020.
Mario Lezcano-Casade and David Martinez-Rubio. Cheap Orthogonal Constraints in Neural Net- works: A Simple Parametrization of the Orthogonal and Unitary Group. In ICML, 2019.
Gen Li, Nan Duan, Yuejian Fang. Min Gong, Daxin Jiang, and Ming Zhou. Unicoder-VLL A Universal Encoder for Vision and Language by Cross-modal Pre-training. In AAAI, 2020.
Liunian Harold Li, Mark Yatskar, Da Yin, Cho-Jui Hsich, and Kai- Wei Chang. VisualBERT: A Simple and Performant Baseline for Vision and Language. arXiv preprint
2019.
11

---

Published as a conference paper at ICLR 2021
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. RoBERTa: Robustly Optimized BERT Pretraining Approach. arXiv preprint arXiv:1907.11922 2019.
Ilya Loshchilov and Frank Hutter. Decoupled Weight Decay Regularization. In ICLR, 2019.
Jiasen Lu, Dhruv Batra, Devi Parikh, and Stefan Lee. ViLBERT: Pretraining Task-Agnostic Visiolin- guistie Representations for Vision-and-Languaae Tasks. In NeurlPS, 2019.
Huaishao Luo, Lei Ji, Botian Shi, Haoyang Huang, Nan Duan, Tianrui Li, Xilin Chen, and Ming Zhou. UniViLM: A Unified Video and Language Pre- Pre-Training Model for Multimodal Understanding and Generation. arXiv preprint arXi::2002.0653 2020.
Antoine Miech, Jean-Baptiste Alayrac, Lucas Smaira, Ivan Laptev, Josef Sivic, and Andrew Zisser- man. End-to-End todEnd Learning of Visual Representations from Uncurated Instructional Videos. In CVPR, 2020.
Pedro Morgado, Nuno Vasconcelos, and Ishan Misra. Audio Visual Instance Discrimination with Cross-Modal Agreement. arXiv preprint Ttt1:2004.129444 2020.
Jiquan Ngiam, Aditya Khosla, Mingyu Kim, Juhan Nam, Honglak Lee, and Andrew Ng. Multi- modal Deep Learning. In ICML, 2011.
Jorge Nocedal and Stephen Wright. Numerical Optimization. Springer Science & Business Media, 2006.
Aaron van den Oord, Yazhe Li, and Oriol Vinyals. Representation Learning with Contrastive Predictive Coding, arXiv preprint arXiv:/807.03740 2018.
Andrew Owens and Alexei Efros. Audio-Visual Scene Analysis with Self- Supervisedd Multisensory Features. In ECCV, 2018.
Andrew Owens, Phillip Isola, Josh MeDermott, Antonio Torralba, Edward H Adelson, and William Freeman. Visually Indicated Sounds. In CVPR, 2016.
Andrew Owens, Jiajun Wu, Josh H McDermott, William T Freeman, and Antonio Torralba. Learning Sight from Sound: Ambient Sound Provides Supervision for Visual Learning. International Journal of Computer Vision, 126(10): 120-11177 2018
Daniel Park, William Chan, Yu Zhang, Chung-Cheng Chiu, Barret Zoph, Ekin D Cubuk, and Quoc Le. SpecAugment: A Simple Data Augmentation Method for Automatic Speech Recognition. In INTERSPEECH, 2019.
Mandela Patrick, Yuki M Asano, Ruth Fong, João F Henriques, Geoffrey Zweig, and Andrea Vedaldi. Multi-modal Self-Supervision from Generalized Data Transformations, arXiv preprint arXiv:2003.0m 04298, 2020.
Matthew E Peters, Mark Neumann, Mohit lyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. Deep Contextualized Word Representations. In NAACL-HLT, 2018.
Karol Piczak. ESC: Dataset for Environmental Sound Classification. In ACM-MM, 2015a.
Karol J. Piczak. Environmental Sound Classification with Convolutional Neural Networks. In MLSP, 2015b.
AJ Piergiovanni, Anelia Angelova, and Michael S Ryoo. Evolving Losses for Unsupervised Video Representation Learning. In CVPR, 2020.
Alec Radford, Karthik Narasimhan, Tim Salimans, and Ilya Sutskever. Improving Language Under- standing by Generative Pre- Trainingg Techincal Report, OpenAI, 2018.
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. Language Models are Unsupervised Multitask Leamers. OpenAl Blog, 1(8), 2019.
12

---

Published as a conference paper at ICLR 2021
Mohammad Rastegari, Vicente Ordonez, Joseph Redmon, and Ali Farhadi. XNOR-Net: ImageNet Classification Using Binary Convolutional Neural Networks. ECCV, 2016.
Sashank J Reddi, Satyen Kale, and Sanjiv Kumar. On the Convergence of Adam and Beyond. In ICLR, 2018.
Russell Reed. Pruning Algorithms-A Survey, IEEE Transactions Neural Networks, 4(5):740-747, 1993.
Florian Schroff, Dmitry Kalenichenko, and James Philbin. FaceNet: A Unified Embedding for Face Recognition and Clustering. CVPR, 2015.
Gunnar A Sigurdsson, Gül Varol, Xiaolong Wang, Ali Farhadi, Ivan Laptev, and Abhinav Gupta. Hollywood in Homes: Crowdsourcing Data Collection for Activity Understanding. In ECCV, 2016.
Gunnar A Sigurdsson, Santosh Divvala, Ali Farhadi, and Abhinav Gupta. Asynchronous Temporal Fields for Action Recognition. In CVPR, 2017.
Khurram Soomro, Amir Roshan Zamir, and Mubarak Shah. UCFI01: A Dataset of 101 Human Action Classes From Videos in The Wild. CRCV-TR-12-01, 2012.
Nitish Srivastava and Russ Salakhutdinov. Multimodal Learning with Deep Boltzmann Machines. In NeurlPS, 2012.
Weijie Su, Xizhou Zhu, Yue Cao, Bin Li, Lewei Lu, Furu Wei, and Jifeng Dai. VL-BERT: Pre-training of Generic Visual-Linguisti Representations. In ICLR, 2020.
Chen Sun, Fabien Baradel, Kevin Murphy, and Cordelia Schmid. Learning Video Representations using Contrastive Bidirectional Transformer. arXiv preprint arXiv:1906.05743 2019a.
Chen Sun, Austin Myers, Carl Vondrick, Kevin Murphy, and Cordelia Schmid. VideoBERT: A Joint Model for Video and Language Representation Learning. In ICCV, 2019b.
Hao Tan and Mohit Bansal. LXMERT: Learning Cross-Modality Encoder Representations from Transformers. In EMNLP-IJCNLP, 2019.
Yapeng Tian, Dingzeyu Li, and Chenliang Xu. Unified Multisensory Perception: Weakki-Superrvied Audio- Audio-Visual Visual Video Parsing. In ECCV, 2020.
Dmitry Ulyanov, Andrea Vedaldi, and Victor Lempitsky. Deep Image Prior. In CVPR, 2018.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention Is All You Need. In NewrlPS, 2017.
Mike Wu, Chengxu Zhuang, Milan Mosse, Daniel Yamins, and Noah Goodman. On Mutual Information in Contrastive Learning for Visual Representations. arXiv preprint arXiv:2005. /3149, 2020.
Dejing Xu, Jun Xiao, Zhou Zhao, Jian Shao, Di Xie, and Yueting Zhuang. Self-supervised Spatiotem- poral Learning via Video Clip Order Prediction. In CVPR, 2019.
Zhilin Yang, Zihang Dai, Yiming Yang, Jaime Carbonell, Russ R Salakhutdinov, and Quoc V Le. XLNet: Generalized Autoregressive Pretraining for Language Understanding. In NeurIPS. 2019.
Xiangyu Zhang, Xinyu Zhou, Mengxiao Lin, and Jian Sun. ShuffleNer: An Extremely Efficient Convolutional Neural Network for Mobile Devices. CVPR, 2018.
Hang Zhao, Chuang Gan, Andrew Rouditchenko, Carl Vondrick, Josh McDermott, and Antonio Torralba. The Sound of Pixels. In ECCV, 2018.
Luowei Zhou, Hamid Palangi, Lei Zhang, Houdong Hu, Jason J Corso, and Jianfeng Gao. Unified Vision-Languaae Pre-Training for Image Captioning and VQA. In AAAI, 2020.
13

---

Published as a conference paper at ICLR 2021
Linchao Zhu and Yi Yang. ActBERT: Learning Global-Local Video- Text Representations. In CVPR, 2020.
Chengxu Zhuang, Alex Lin Zhai, and Daniel Yamins. Local Aggregation for Unsupervised Learning of Visual Embeddings. In /CCV, 2019.
A IMPLEMENTATION DETAILS
A.1 ARCHITECTURES OF VISUAL/AUDIO CNNS
Table 3 shows the architectures of visual and audio CNNs we use for our model. For the visual CNN. the SlowFast network (Feichtenhofer et al. 2019) with ResNet-50 backbone (He al. 2016). We use the speed ratio = 8 and the channel ratio / = 1/8 for the SlowFast architecture, so Ty 8 x T.. We use different values of T. and Ty for different tasks. During pretraining, set T. = 4 and T = 32. During finetuning, we use T, = 8 and Ty = 64 for short-video action classification on UCFI01 (Soomro al. 2012) while we use T. action classification on Charades (Sigurdsson et al. 2016) and Jinetics-Soundee (Arandjelovic & Zisserman, 2017). For the audio CNN, we use ResNet-50 without the downsampling layer pooli to preserve information along both frequency and time axis in early stages. We use different values of To for different training phases. We set T. = 220 for one-second clip during pretraining while we use T, 440 for two-second clip during finetuning.
and T, 32 for long-vides
Visual CNN
Stage
Audio CNN
Slow pathway
Fast pathway
raw clip
x T. X 112"
x T x 112
128 x T.
1 X 72, 64
x 72,8
9 x 9,
conv
stride 1,
stride 1,2²
stride 1,
1 1x32 3 max
1x3 3 max stride 1,2²
pool
stride 2
x 12, 64
3 1x 8
1 1, 32
res2
1 32 64
x3
1 1 32
x3
3 3,
x3
1 x 12 256
1 12 32
[ X 1, 128
1 12, 128
3 12,16
1,
1 x 32, 128
x4
3x
x4
3 x 3, 64
x4
X 512
1 1x
1 1, 256
3 12, 256
3 x 12,32
[ 1, 128
reS4
1 x 32 256
X6
1 1x 32,32
x6
3 x 3, 128
x6
1 12, 1021
1 x 128
x 1,22
x 12 512
3 x 12 64
x 1, 256
ress
32 512
x3
1 x 32 64
x3
x 3, 256
x3
1 1 2048
x 1 256
1 x 1, 1024
Table 3: The architectures of visual and audio CNNs. For the visual CNN, the input dimensions are denoted by (channel size, temporal size, spatial size"), kernels are denoted by (temporal size, spatial size?. channel size) and strides are denoted by /temporal stride, spatial stride?), For the audio CNN, the input dimensions are denoted by Ifrequency size, temporal size), kernels are denoted by (frequency size, time size, channel size) and strides are denoted by (frequency stride, temporal stride).
A.2 DATA PREPROCESSING
We preprocess the data by dividing 7 -eccndd clips into 7'' non-overlapping parts (T' 30 for Kinetics- 700 (Carreira et al. 2019) and T 10 for AudioSet (Gemmeke et al. (017 and sampling 16 frames from each. For audio stream, take waveform sampled at 44.1 kHz and convert it to log- mel--seed spectrogram. We augment audio data with random frequency/time masking using SpecAugment (Park) al. 2019), and visual data with color normalization, random resizing, random horizontal flip, and
14

---

Published as a conference paper at ICLR 2021
CANS-Oissmiaa Simitar
38-
CANS Dissimlar
92.5
Multi-
75-
36
SWNS-Similar
87.5
Mutth Part
Multi Patt
70-
34-
85.0
02.5
80.0
60-
53-
75.0
26-
72.5
30k
40k
10cc
20c
40k
50k
10c
20k
30k
40k
Iterations
berationn
Reeations
(a-1) MEP CPP
(a-2) Visual MEP
MEP CPP
(a) Ablation: negative sampling
(b) Ablation: parameter sharing
Figure 3: Loss curves during pretraining under different ablative settings. (a) compares Content- Aware Negative Sampling (CANS) that favors negatives that are dissimilar vs. similar the positive instance. (b) compares different cross- Trnss-Trranfommer weight sharing schemes; see the text for details.
random cropping to obtain 112 x 112 pixel frames; for test data, we resize videos to 128 pixels on the shorter side and take three equidistant crops of 128 128 pixels to cover the entire region. We also apply audio-visual synchronized temporal jittering (Patrick et al. 2020).
A.3 DOWNSTREAM EVALCATION
For evaluation on UCF101, we follow the test protocol of (Feichtenhofer et al. 2019): We sample 10 clips from each test video uniform time interval, and for each sampled clip, we take three equidistant spatial crops, resulting in a total of 30 views. We use each of the 30 views as input to our visual CNN and average the prediction scores from all 30 views to obtain the final prediction result. For evaluation on ESC-50 (Piczak 2015a), we extract 10 equally spaced 2-second clips from each test audio sample. We use each of 10 clips as input to our audio CNN and average the prediction scores obtain the final prediction result. For evaluation on Charades and Kinetics-Sounds, we use three audio-visual sequences with different spatial crops from a test video and max- max-poollaaverge the prediction scores from each sequence, respectively.
A.4 OPTIMIZATION
In all experiments, use the AMSGrad (Reddi al.) 2018) variant of AdamW (Loshchilov & Hutter 2019) optimizer with 31 = 0.9, Br = 0.98, L2 weight decay of le-4. We use a learning rate warm-up for the first 6% of iterations followed by a linear decay of learning rate.
From the observations of Lezcano-Casado and Martinez-Rubio [Lezcano-Casado & Martinez- Rubio 2019), we have 10 times less learning rate for the orthogonal parameters than that for the non- orthogonal parameters: we use lc-5 for the former and lc-4 for the latter.
We pretrain our model on Kinetics-700 (Carreira et al. 2019) with batch size 256 for 220K iterations and AudioSet (Gemmeke et al., (017)) with a batch size 300 for 220K iterations in the main experiments; for the ablation study, we use much smaller batch size of and pretrain our model on Kinetics-700 for 80K iterations.
For finetuning on UCF101, we train our model for 40K iterations with batch size of 64 and learning rate of 0.02. For evaluation on ESC-50, we train a multi-class one-vs-all linear SVM on top of our fixed audio CNN for 38K iterations with a batch size of 128 and learning rate 0.003. For finetuning on Charades, we train for 40K iterations with batch size of 8, with learning rate of 0.001 for the classifier and CNN parameters, 1e-5 for the orthogonal parameters and le-4 for the rest parameters. For finetuning on Sinetics-Sounde we train for 24K iterations with batch size of 32, with learning rate of 0.005 for the classifier and CNN parameters, le-4 for the orthogonal parameters and 1e-3 for the rest parameters.
15

---

Published as a conference paper at ICLR 2021
B EXTRA RESULTS FROM THE ABLATION STUDY
B. NEGATIVE SAMPLING STRATEGIES
We proposed the content-aware negative sampling strategy (CANS) using pairwise l2 distances between CNN embeddings. We introduced two variants of CANS: MANS-Dissimila that favors negatives that are dissimilar to the positive instance and CANS Siimsmmmllli lar that favors negatives that similar to the positive instance. We chose to PANS-Simel lar based on the results from ablation study presented in the main paper, Table 1 (b).
Figure 3 (a) in this appendix provides additional evidence that supports decision. We that the loss of CANS-Dis milar initially drops rapidly but starts increasing around iteration 7K and continues to increase until around 15K; this is mainly caused by the visual MEP loss shown in Figure 3 (a-2). One explanation for this might that
Diiiimmlii lar is easier to solve than CANS Similar, which causes the loss landscape of CANS-Dis -iis--iiimiaas to contain too many shallow local minima compared to that of CANS-Simi
Recall that we use a learning rate warm-up for the first 6% of iterations during pretraining; this roughly equals the first 13K (out of 220K) iterations. Given this, we speculate that the model got out of local minima around iteration 7K (most likely due to the increasing learning rate), and then eventually settled in another (bad) local minima after the warm-up period ended. Compared to this, we observe much milder learning dynamics with CANS- Simi mar: the loss decreases relatively slowly but steadily, and eventually leaps around 35K to go below the loss of CANS-Disimilen this is because CANS Simi M aar is more difficult to solve than CANS-Di: imi lar (as shown by the slower decrease in loss values), which caused the resulting loss landscape contain steeper local minima. Our model eventually found one of those after round 40K of iterations, resulting in a better performing model in the downstream tasks shown in Table I (b) of the main paper (the loss kept slowly decreasing after iteration 50K).
We, again, believe that
B.2 PARAMETER SHARING SCHEMES
We proposed a cross- cross-Transfore Transformer weight sharing technique, which decomposes weight matrices with low-rank approximation. Recall that each layer of a Transformer contains the multi-head attention layer weights (Wo, W* W Wo and the feed-forward layer weights (We, Wd.. We chose to share all six weight matrices across Transformers, though we could have shared any combination of them. To justify this design choice, we empirically compared three variants: (i) Mu 1 i -6 6 that do not share parameters across Transformers, (ii) Multi- 6-Part-rtt-t that shares only {W', W W W)3 (but not {W W*1) and (iii) Multis i 6-Part that shares all six weight matrices. Figure 3 (b) shows that there is not much difference between all the variants in terms of the loss curves; we chose to use Multa ti- -6 P-rr that requires the least number of parameters. We showed that our approach outperforms Multi- the ablation study (Table 1 (c-left) the main paper).
B.3 JUSTIFICATION FOR THE MEP LOSS FORMULATION
Since the multimodal Transformer hAv has access to both visual and audio inputs, one might think that the model could "leak" information about visual input into 2" and information about audio input into z", which could make MEP trivial to solve. Here we show that this is not the case. By construction, we mask the same positions in audio and visual streams when designing the MEP task, so the model has no access to the masked input even a cross-moda manner. Empirically, removing the third term in Eq. (LNCE (x"; x"], Z)) leads to performance degradation in Kinetics-Sounds i.c., top-1 accuracy 66. 66 vs. ours 67.5% (see Table T. which suggests that solving the MEP task in the multimodal Transformer is beneficial to model.
B.4 JUSTIFICATION FOR THE CPP LOSS FORMULATION
Recall that our CPP loss has two terms; the first term uses the summary embeddings S, and the second term uses output embeddings Sy, sampled at random positions; see Eq. 7 One could argue that the two terms are redundant bidirectional Transformers have "one-step" access to all the input embeddings, and thus solving CPP only with the summary embeddings (the first term) would be enough. This is not the case. We encode s with position-specific information through the time
16

---

Published as a conference paper at ICLR 2021
embeddings Pt, which makes every Sy, different compared to Sy. Empirically, we find that removing the second term Eqn. (s)) in our CPP loss leads to an inferior accuracy 66.9% vs. ours 67.5% on Kinetics-Sounin suggesting its importance in learning.
B.5 USE OF MODALITY EMBEDDINGS IN THE MULTIMODAL TRANSFORMER
We use modality embeddings m" and m as part of input to the multimodal Transformer in order distinguish embeddings coming from visual and audio Transformers. They are learnable weights trained end-to-end with other parameters. Conceptually, incorporating poolalty-discriminatiy em- beddings is crucial because our aggressive weight sharing scheme. Without them, the multimodal Transformer will the output from audio/visual Transformers (y" and y²) if they coming from the same distribution because the two Transformers share a large part weights. Using modality embeddings encourages model to preserve modality-specion information in the final output, and this empirically leads to performance improvements: ours 67.5% vs. without modality embeddings 67.15 % on Senetics-Sounds.
B.6 ON THE IMPORTANCE OF END-TO-END PRETRAINING
Previous work in multimodal visual-and lann--aan- tasks (Tan & Bansal 2019 Lu et al. 2019) point that using partially fixed Transformers of different modalities is detrimental to multimodal representation learning (c.f.,/un et al. (2019a b)). We make the same observation in our audio- visual learning scenario. We compare two variants Multi-6 Part Table T (c), each of which pretrains only the audio (or visual) CNN/Transfommer in the first half of pretraining stage and then continues pretraining the remaining weights while fixing the weights of the audio (or visual) CNN/Transformer in the second half. This leads to inferior performance (audio-fixed 62.8% and visual-fixed 63. 1% vs. ours 67.5%% which consistent with the results reported in Tan & Bansal (2019): Lu et al. (0099)
17


Reviews:
[{"review_id": "bwnouZTvtm", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary: \n\nIn this paper, the authors propose a multimodal transformer network for audio-visual video representation learning.  To reduce parameters,  a new parameter sharing scheme is introduced. They further propose a Negative Sampling strategy to improve model training. Experiments are performed on several audio and video benchmarks.\n\nStrengths:\n\n(1) The proposed parameter sharing can reduce model sizes and preserve performance. \n\n(2) The negative sampling strategy is content-aware and can improve data quality.\n\nWeaknesses:\n\n(1)  The technical novelty is limited. The main contributions are the proposed parameter sharing and negative sampling strategies. However, as discussed in the paper, parameter sharing across layers has already explored in previous works. In addition, improving negative sampling for audio-visual learning is also not a new idea. AVTS has already explored it. Thus, the contributions are pretty incremental.\n\n(2) Transformer-like audio-visual networks have been used in recently published papers. For example, [1] uses an audio-visual transformer for audio event classification, and [2] proposes a new joint audio-visual transformer module with both self-attention and cross-modal attention. \n\n[1] Boes, Wim, and Hugo Van hamme. \"Audiovisual Transformer Architectures for Large-Scale Classification and Synchronization of Weakly Labeled Audio Events.\"ACM MM. 2019.\n\n[2] Tian, Yapeng, Dingzeyu Li, and Chenliang Xu. \"Unified Multisensory Perception: Weakly-Supervised Audio-Visual Video Parsing.\" ECCV (2020)\n\n(3) Unfair comparison in Table 2(b). The authors validate the effectiveness of learned audio features on the ESC-50 dataset. Previous methods including SoundNet, L3, DMC, and AVTS extract features from pre-trained models and use an SVM to predict event categories without fine-tuning of audio networks.   However, the authors fine-tune the model on the ESC-50, which makes the comparison become unfair. \n\n(4) With an additional multimodal transformer model, the proposed method fails to greatly improve performance. The previous methods such as AVTS and DMC, they only have an audio net and a visual net to compute loss functions and learn representations. However, the proposed model has a large additional multimodal transformer after slowfast visual net and ResNet-50 audio net. Even so, I fail to observe significant improvements over recent approaches. \n\n*** Post-Rebuttal ***\n\nSome of my concerns are addressed by the authors' rebuttal. The unfair experimental comparison has been fixed. \n\nThe proposed model indeed has some merits (e.g., parameter sharing and negative sampling). However, to me, the technical novelty of the paper is incremental.  In addition, with additional large transformer networks, the proposed model only achieves limited improvements over previous methods. The authors claim that the proposed model is more effective in handling long videos. But, only results on Charades and KS are shown without extensive comparisons. \n\nThus, I would like to keep my rating unchanged.", "extra_scores": {}, "extra_text": {}}, {"review_id": "D2MzNArxHIk", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper studies modeling and training choices when designing a single model based on ConvNets and transformers for audio-visual representation learning. It proposes ablations for which weights/layers to share across modalities, when/where to fuse/join both modalities, and other modeling details (that matter). It also completes pre-training with 3 InfoNCE (audio-audio, visual-visual, audio-visual) with a binary classification loss about if two pairs of audio-visual are from the same or different videos. As strategies for negative sampling in audio and videos are different, it proposes to sample negatives that are similar in the ConvNets' embeddings. The models are pretrained on Kinetics-700 and AudioSet and evaluated on UCF101, ESC-50, and Kinectics-Sounds.\n\nContributions:\n+ A sound, extensive set of ablations of the choices made along the way of modeling and training.\n+ Good performance (overall best reported but for UCF101).\n\nLimitations:\n- The article is at times slightly difficult to follow. A few important pieces are mentioned too quickly: e.g. how exactly to do the content-aware negative sampling (bottom paragraph page 4), or how exactly to get UΣV from (maintain Σ orthogonal with a scale-squaring trick + first project Vx on the unit sphere). Some explanations come after the first introduction of the term.\n- There is a small flaw in the parameters sharing experiments (Table 1, bottom) as the models that are compared have vastly different capacity, so maybe not the same training regimes. It would be more rigorous to also have an experiment where one reduces the (parameters) size of the model without sharing from 155M to ~34M (or 31M, and decompose W=UΣV for those weights too), and compare this to the all/part parameter shared models.\n- Not a negative point in itself, but related to how this article is written: content-aware negative sampling is not new, see e.g. FaceNet (Schroff et al. 2015), On Mutual Information in Contrastive Learning for Visual Representations (Wu et al. 2020). \n\nOverall, the paper presents multiple ablations and gives results that show that the model leverages both modalities, on challenging datasets. But it feels a bit too much like a tech report, where the most important bits of the contribution are given a too light algorithmic treatment (negative sampling, parameter sharing).\n\nNitpick: in Table 1 the left / right tabulars are way to close, put some \\quad.\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "Cy0h6XdS7yk", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "**Summary**\nIn this work, the authors present a method for learning audiovisual (AV) representations from videos using a Transformer-based model architecture. Since both video processing and Transformer-based model are memory-intensive, a parameter-reducing scheme is proposed, which facililates training the model end-to-end. The AV representations are learned by training the network to solve two self-supervised pertaining tasks, and subsequently evaluated on various audio/visual downstream tasks. An ablation analysis is performed to demonstrate the efficacy of the various contributions.\n\n**Strengths**\n- Leveraging unlabeled videos to learn powerful audiovisual representations is an important problem, and judging by the many recent papers on this topic, of much relevance to the community.\n- The use of a Transformer-based model architecture is well-motivated, given its success on other multi-modal problems. To the best of my knowledge, videos of such long duration (30 sec.) have not yet been used in this setting, and the ability to process such long videos seems doable due to the contributions of this paper.\n- The ablation study presented does a good job of highlighting the contribution of the design choices, and can be of importance to future works which seek to build upon this one.\n- Empirical results on both audio and video understanding tasks demonstrate that the proposed method does indeed learn useful representations, and that multimodal training provides the expected boost to results.\n\n\n**Concerns**\n- The authors imply that using a partially fixed model (as has been done with multimodal vision/language tasks) is inferior to end-to-end training, hence the motivation for the proposed parameter-reducing technique, which is the major technical contribution of this work, as presented by the authors. This may very well be the case, but I would have liked to see evidence of that. Obviously, comparing fixed vs. end-to-end training of vision/language models using the proposed method would be very interesting to see, but is out of the scope of this work. However, perhaps comparing the proposed end-to-end audiovisual model with a similar fixed model would provide the insight necessary to determine the importance of your contribution.\n- It seems to me that there are some previously reported self-supervised results missing from Table 2:  \n[1] obtain 93% accuracy on UCF101 and 85.8% on ESC using AudioSet for pretraining.  \n[2] obtain 91.3% on UCF101, and [3] 93.8% on UCF101, both using larger datasets.  \nThis obviously a crowded space, with new results being published often, but I would have hoped to see larger performance gains on tasks that require more global reasoning, such as action recognition, given the ability of the proposed model to process very long sequences.\n- Perhaps I didn't understand this correctly, but in Table 1, shouldn't CANS-dissimilar perform less well, since negative samples are \"easy\", therefore causing MEP to dominate?\n- Nit: In equation (1), I think Q, K, and V might be missing the subscript 'i'?\n\n**Additional comments**\n- Abstract: The reader needs to read almost halfway through the abstract to get to what this work is about. IMO, either move a significantly reduced version of the \"why\" to after the \"what\", or perhaps just leave the motivation for the introduction.\n  \n  \n[1] Alwassel, Humam, et al. \"Self-supervised learning by cross-modal audio-video clustering.\" arXiv preprint arXiv:1911.12667 (2019).  \n[2] Miech, Antoine, et al. \"End-to-end learning of visual representations from uncurated instructional videos.\" Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2020.  \n[3] Piergiovanni, A. J., Anelia Angelova, and Michael S. Ryoo. \"Evolving Losses for Unsupervised Video Representation Learning.\" Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2020.", "extra_scores": {}, "extra_text": {}}, {"review_id": "3WSzXXnNIy5", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper is a nice read. It builds on a line of research on multi-modal video understanding that utilises transformers where these works: 1) fix one of the transformer models (e.g. BERT) and 2) utilise tokens and thus do not train the approach in an end-to-end fashion. This typical trend is due to the memory requirements for training a multi-modal transformer end-to-end.\nTo allow for end-to-end learning, the paper argues for shared parameters (across the network), primarily:\na) sharing weights in CNNs of the same model [understandable]\nb) sharing weights between layers of transformer for the same modality\nc) sharing weights between modality transformers\nd) performing mid-fusion by having modality-specific transformer followed by cross-modality transformer,\ne) sharing position encoding parameters between modalities and transformer layers\nf) decomposing transformer weights, so some are distinct and others are shared\n-- among other suggestions [I didn't list fully].\nIn addition to the above, the paper showcases the need for context-aware negatives, rather than random sampling, in line with a number of concurrent works that address this issue, some submitted to ICLR [which I reviewed as well coincidentally].\n\nThe paper then runs a number of experiments to prove their approach, all showcasing the combined advantage of end-to-end learning with shared parameters. This is tested on the \"usual suspect\" set of datasets: Kinetics, Audio Set, with downstream tasks on short-range datasets (e.g. UCF) as well as long-term (e.g. Charades). Performance improvement over the baseline is consistent.\n\nTwo aspects of the paper are disappointing,\nFirst, the motivation that the approach will enable end-to-end learning with transformer, should be re-written to say: \"This would enable end-to-end learning with multi-modal transformer for a handful of labs who have 64 V-100 GPUs which can be trained for 220K batches [I'm presuming that's many days/weeks].\" It is quite impossible for almost all researchers to utilise the findings of this paper. It's true that the number of parameters has dropped significantly, but in any forward/backward pass, the memory requirements of multiple slow-fast (ResNet-50) with all transformers in memory and a necessarily large batch size keeps the same limitation of an \"end-to-end-to-end\" approach more likely to be used by the community. Apart from knowing of this finding, I am not sure how this will transform the community's go-to solutions.\n\nSecond, the experimental results (tables and commentary on tables) are not designed for easy consumption. This makes the readability of the experimental section below acceptable bar IMO. This should be fixable, but disappointing that it is submitted in the current form. Let me give you a few examples of how difficult it was to read the tables of results:\nEx1: Table 1, the caption talks about a, b and c but these are not referenced in the actual tables. It took me several minutes to realise you are referring to the two tables on the first row as the gap between the two tables is not easy to observe. \nEx2: Table 1, names of sampling methods \"similar/dissimilar\" do not align with how the paper describes hard negatives. Using varying terminology you need to rely on the brief description to know what's happening.\nEx3: Table 2, the decision to list the dataset references like this makes the table and checking the references an impossible task, many abbreviations (e.g. KS for Kinetics-Sounds) are not common. On the first two table within Table 2 you use Ours, which I presume is M-BERT in the right table? It is not clear why V- and A- were not tested independently for the tables on the left, but are ablated on Charades and Kinetics Sound. \nThese tables were very hard to follow/read and check for correctness. \n\nA few minors:\n*) I am not sure the results on Charade represent the SOA on this dataset. They seem to only reference the first 2017 paper as a baseline? The same for Kinetics-Sounds, this seems to be a very old baseline?\n*) The manuscript talks about the audio being a \"real valued audio signal\" and it's only in the appendix that the log-mel-scaled spectrogram is explained. This can be deceiving to the reader.\n*) Given very few can replicate these results, the fact that an input of 1 second was only tried in all experiments limits our knowledge of the impact of this critical parameter. \n*) On the issue of Task 2 (correct pair prediction) other works have discussed the need for asynchronous understanding from the audio-visual signal that are worth referencing, e.g.\nKazakos and Zisserman (2019). Audio-Visual Temporal Binding for Egocentric Action Recognition. ICCV\nMorgado et al (2020). Audio-Visual Instance Discrimination with Cross-Modal Agreement. ArXiv April 2020 [recent work understandably] https://arxiv.org/abs/2004.12943v1 ", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: Three out of four reviewers are positive about the paper after the author response and during the discussion.

Strengths include
* The proposed method for parameter reduction in transformers allows end-2-end learning cross-modal representations especially on long videos, which has not been possible before
* Good performance on audio and video understanding
* Extensive set of ablations

Concerns include a somewhat incremental nature of the paper and the still large computational resources to run the experiments.
I think, both, the ideas and results are interesting to the community and recommend accept.
Decision: Accept (Poster)