Support-set bottlenecks for video-text representation learning
Authors: Mandela Patrick
Po-Yao Huang
Yuki Asano
Florian Metze
Alexander G Hauptmann
Joao F. Henriques
Andrea Vedaldi
Venue: ICLR 2021 Spotlight (2021)
Keywords: video representation learning
multi-modal learning
video-text learning
contrastive learning

Abstract: The dominant paradigm for learning video-text representations – noise contrastive learning – increases the similarity of the representations of pairs of samples that are known to be related, such as text and video from the same sample, and pushes away the representations of all other pairs. We posit that this last behaviour is too strict, enforcing dissimilar representations even for samples that are semantically-related – for example, visually similar videos or ones that share the same depicted action. In this paper, we propose a novel method that alleviates this by leveraging a generative model to naturally push these related samples together: each sample’s caption must be reconstructed as a weighted combination of a support set of visual representations. This simple idea ensures that representations are not overly-specialized to individual samples, are reusable across the dataset, and results in representations that explicitly encode semantics shared between samples, unlike noise contrastive learning. Our proposed method outperforms others by a large margin on MSR-VTT, VATEX, ActivityNet, and MSVD for video-to-text and text-to-video retrieval.

Published as a conference paper at ICLR 2021
SUPPORT-SET BOTTLENECKS FOR VIDEO-TEXT REPRESENTATION LEARNING
Mandela Patrick; Po-Yao Huang, Florian Metze & Andrea Vedaldi
Facebook AI
[mandel apatrick htt.ddaaaaaarii..eeeeeeaaaaa fmetze, veda 1d d @.f..com
Alexander Hauptmann
Yuki M. Asano & João Henriques
Language Technologies Institute
Visual Geometry Group
Camegie Mellon University alexêcs cmu, edu
University of Oxford
(yuki, joao) @robots. ox ac. uk
ABSTRACT
The dominant paradigm for learning video-text representations. noise contrastive learning- increases the similarity the representations pairs samples that are known to be related, such as text and video from the same sample, and pushes away the representations of all other pairs. We posit that this last behaviour is too strict, enforcing dissimilar representations even for samples that semantically- related for example, visually similar videos or ones that share the same depicted action. In this paper, we propose novel method that alleviates this by leveraging a generative model to naturally push these related samples together: each sample's caption must be reconstructed ples' visual representations. This simple idea ensures that representations are not overly-specialized individual samples, are reusable across the dataset, and re- sults in representations that explicitly encode semantics shared between samples, unlike noise contrastive learning. Our proposed method outperforms others by a large margin on MSR-VTT, VATEX, ActivityNet, and MSVD for video-to-text and text-to-video retrieval.
weighted combination of other support sam-
INTRODUCTION
Noise contrastive learning (Gutmann & Hyvärinen 2010) is emerging as one of the best ap- proaches to learn data representations both for supervised (Khosla et 2020) and unsupervised regimes (Chen al. 2020c). The idea is to learn representation that discriminates any two data samples while being invariant to certain data transformations. For example, one might learn a repre- sentation that identifies specific image up to arbitrary rotations (Misra & van der Maaten 2020)). In a mudti-modal setting, the transformations can separate different modalities, for example, by extract- ing the audio and visual signals from video. The resulting noise contrastive representation asso- ciates audio and visual signals that come from the same source video, differentiating others (Patrick) al. 2020).
The noise contrastive approach is motivated by the fact that the transformations that are applied to the data samples leave their 'meaning' unchanged. For example, rotating image does change the fact that it contains a cat or not (Gidaris et al.] 2018). However, in most cases, we expect to find many data samples that share the same content without being necessarily related by simple transformations (c.g. think of any two images of cats). Existing noise contrastive formulations are unaware these relationships and still try assign different representations to these samples (Wu) et al.) 2018), 8), despite the fact that they are semantically equivalent. If the representation is learned for a downstream task such as semantic video retrieval, this might degrade performance.
This suggest that there might be other learning signals that could complement and improve pure contrastive formulations. this paper, we explore this idea in the case of learning from two modali-
Joint first authors.

---

Published as a conference paper at ICLR 2021
(a) Contrastive learning
(b) Cross-catiioinng
Contrastive attraction
Contrastive repalsion
"throwing, the ball"
"Urowing
Generative attraction
the
"putting the shot"
hheaving
ball"
"putting
shot"
"hesving
hall
Support-set
Fig. 1: Cross-modal discrimination and cross-captioning, Our model learns from two comple- mentary losses: (a) Cross-modal contrastive learning learns strong joint video-text embeddings, but every other sample considered a negative, pushing away even semantically related captions (orange arrows). (b) We introduce generative task of cross-captioning, which alleviates this by learning to reconstruct a sample's 's text representation as a weighted combination support-set, composed of video representations from other samples.
ties: videos and text, in the form of video transcripts or captions. Given a state-of-the-an contrastive formulation that learns from these modalities, we investigate complementary pretext objectives to improve it. First, we consider the (instance) captioning task, namely mapping a video to the corresponding text, casting this conditional stochastic text generation problem. We show that this brings only a modest benefit.
We observe that the captioning task highly sample-specific. as the goal is to produce a caption which describes a specific video and not any other video, and thus it suffers from the same disad- vantages (discouraging concept sharing among samples) as contrastive learning. Thus, we propose to address this issue by switching to a different text generation task. The idea is to modify the text generator to take as input learnable mixture a support-set of videos, which we call cress-instance captioning. The mixture weights are generated by comparing the learned video representations to captions' representations in an online way over the batch. The limited set of support samples acts bottleneck that encourages extraction of shared semantics. In this manner, the embeddings can associate videos that share similar captions even the contrastive loss tries to push them apart.
We show that, when the captioning task is added in this manner, it brings sensible improvement to already very strong video representation learning results, further improving our own state-of-the baseline by significant margin.
RELATED WORKS
Learning data representations from unlabelled data has been long standing goal of machine learn- ing. These approaches are called "self-supervised learning" because the learning signals, termed pretext tasks, obtained from the data itself. In the image and video domain, pretext tasks include colorization (Zhang et al. 2016), rotation (Gidaris et al. 2018), or clustering (Asano et al. 22220a b Caron al., 2018 et al. 2218), while in the natural language domain, masked language model- ing (Devlin et al. 2019). and next word prediction (Mikolov et al. [2033] Pennington et al., 2014) are extremely popular. These pretext tasks can be broadly classified into two classes: generative and discriminative.
Discriminative approaches learn representations by differentiating input samples, using objectives such as the contrastive loss (Gutmann & Hyvärinen 2010, Hadsell et al., 2006). Discriminative
approaches have proven to be particularly successful for image (Chen et al. 2020c He ct al. 2020 Misra & van der Maaten 2020 Wu al.) 2018)) and video (Han al. 2019 Morgado et al. 2020
Patrick et al., (0200) representation learning. Generative approaches, on the other hand, try to re- construct its input. GANs (Donahue & Simonyan 2019 Goodfellow et al. 2014 Radford et al. 2015). autoencoders (Hinton & Salakhutdinov (0066)
(006) and sequence-to-see wewueence-to-eqqeene models (Huang al. 2020; (Sutskever al. (204)) are popular generative models. In this work, show the im- portance of combining both discriminative and generative objectives to learn effective video-text representations.
The success of representation learning has also been due to advances in model architectures, such as the Transformer Vaswani et 2017). BERT (Devlin al. 2019) demonstrated that transformer
2

---

Published as a conference paper at ICLR 2021
(a) Training pipeline
(b) Attention for reconstruction
Positive sample
"A throws shot put"
S
V
Identity
Text encoder
Video encoder
Cross-instance captioning
Support-set
Full
Text decoder
Video encoder
Cross
"A person throws a ball"
Attention for reconstruction
P
Generative attraction (similar samples)
Hybrid
Contrastive attraction (same sample) Contrastive repulsion (all other samples)
Fig. 2: (a) Our cross-modal framework with the discriminative (contrastive) objective and the gen- erative objective. The model learns to associate video-text pairs in common embedding space with text and video encoders (top). Meanwhile, the text must also be reconstructed as weighted com- bination of video embeddings from support-set (bottom), selected via attention, which enforces representation sharing between different samples. (b) Weights matrices (attention maps) used in each cross-captioning objective (see section 3.2.2..
architecture pretrained on large-scale textual data can learn transferable text representations that can be fine-tuned on a variety of downstream tasks. Subsequent works (Clark et al. 2020; Lewis et al.) 2020alb) by Radford et al. 2019: Raffel et al.) 2019) have improved upon the transformer architecture training objective to learn even better representations. Inspired by the success of transformers in the NIP domain, several works have leveraged transformers to learn transferable image (Chen et al.
2020a Desai & Johnson 2020 Sariyildiz et al. 2020) multi-modal image-text (Chen al. [2199]
Li et al. [2220a] 2019 Lu et al. 2019 Su et al. 2019 Tan & Bansal 2019) and video-multilinuall
text (Huang et 2021 representations, In this work, leverage the transformer architecture to better encode and represent text and video.
Large-scale training data has enabled the more effective pretraining of image (Sun al.) 2017
Yalniz et al. 2019), video (Ghadiyaram et al., 2019 Thomee et al. (016) and textual representa- tions (Raffel al. 2019). The release of the How To100M dataset (Micch al. 2019). a large-scale instructional video dataset, has spurred significant interest in leveraging large-scale pretraining to improve video-text representations for tasks such as video qnestion-answeriin
(Lei et al.) 2018))
text-video retrieval (Liu et al.,, (201)) and video captioning (Zhou et al. (2018b) smaller datasets such as YouCookII (Zhou et al. (2018a), MSVD (Venugopalan et al. 2015a). MSR-VTT (Xu et al.)
2016 LSMDC (Rohrbach et al. (017). DiDeMo (Hendricks et al. 2018) and ActivityNet (Krishnal ct al. 2017). Although semantically rich and diverse, instructional videos from the web are super noisy and therefore few approaches have been proposed combat this. A few works (Luo et al. 2020
2019a|b Zhu & Yang, 2020) extend the BERT model to accept both visual and textual tokens learn high-level semantic video-text representations. Other works have leveraged the contrastive loss (Miech et al. 2020) and show that using the raw audio (Alayrac et al. 2020) Rouditchenko al.
Sun et al.]
2020) and other modalities (Gabeur al. 2020) can be used to better align and improve video-text representations. While all these approaches rely on a contrastive objec- tive, VidTranslate (Korbar al., 2020) shows that generative objective can also be used to learn joint video-text representations. In contrast to Korbar et al. (2020), we show that combining con- trastive and generative objectives to pre-train video-text representations on large-scale data such as HowToo10M is very effective. The generative objective serves as regularizer to mitigate the strict- ness the instance discrimination task of the constrastive objective, showing benefits similar to approaches such as clustering (Caron et al. 2020) Li ct al. 2020b) and feature mixing (Kalantidis et al. 2020) which have been applied in the image domain.
3

---

Published as a conference paper at ICLR 2021
3 METHOD
We consider the problem of learning multimodal representations from a corpus C of video-text pairs (v, where is video and t its corresponding text (caption or transcription). Our goal is to learn a pair of representation maps c, ((() and Gr (t), with outputs in a d-dimensional embedding space Cu. Ct R., where semantically similar instances are close each other.
3.1 OBJECTIVE FOR LEARNING MULTIMODAL REPRESENTATIONS
We consider two learning objectives, also illustrated in Figure [ The first is the contrastive objective, pushing embeddings a and cy to close if text and video e come from the same sample and push- ing them apart otherwise. This assumes that every sample is its own class and does not benefit from modelling similiarities across instances. The second objective is generative captioning. In its most basic variant, maximizes the probability of generating the text given the corresponding video v. However, we suggest that variants that explicitly promote concept sharing between instances will result in better downstream performance, tasks such as video retrieval. These variants, illustrated in Figure 2] have in common that the caption f is reconstructed from learned weighted combina- tion over other videos 6, This form of attention (Bahdanau et al., 2014) which encourages the network to learn about which videos share similar semantics, compensating for the contrastive loss and grouping them implicitly.
In the following, we denote with B C C a batch of multi modal samples, i.c. finite collection of video-text pairs (1, v) E C. For simplicity, we denote the batch as {(t), ))
3.1.1 CONTRASTIVE OBJECTIVE
To define the contrastive objective, let s(a, b) Tall bJ be the similarity measure between vectors a and b. Following Faghri et al. (2018), we adopt the hinge-based triplet ranking loss with hard negative mining:
conntaast 1
Σ max Ja s(cl, s(c), + max Ja s(cl, s(c) ct) s)(.,))) (1) B
where is the correlation margin between positive and negative pairs and [++ max (0, -3 is the hinge function. In our experiments, we set a = 0.2.
3.1.2 CROSS-CAPTIONING OBJECTIVES
In the conventional captioning, the decoder seeks to optimize the negative log-likelihood of a text sequence given its corresponding video u:
Cappioon
Σ logg(t")e)).
(2)
Here, the log-likelihood is obtained via auto-regressive decoding (Vaswani et al.) 2017) from an intermediate video embedding et +'(w"). For the cross-captioning objective, we modify this loss to condition the generation process on weighted average of the embeddings of the other videos in the batch, which we call the support-set. The weights themselves, which can be interpreted as batch-wist wise attention, obtained sofumax distribution with temperature T over batch indices based on the video embeddings, as follows:
B
2== Σ exp c(/T JES, Exes, exp (ct, cb)
paross captioning
B log logg((()))
M
e
1=1
By default, the summation in the softmax is conducted over a support set S, containing all indices except i. In the experiments, we consider the following attention types for reconstruction. Iden- tity captioning (S, = {t}) generates the caption from the corresponding video and reduces to the standard captioning objective, eq. (2). Full support (S; {1...., .....,B)) B}} considers all videos as pos- sible candidates for captioning. Hybrid captioning sets the weights in eq. (3) the average of the

---

Published as a conference paper at ICLR 2021
weights for identity captioning and full support. Cross-captionigg (S, {j i}) considers all but the video that one wishes to caption. This variant forces the network to extract all information required for captioning from other videos in the batch. Figure comparess graphically these attention mechanisms.
Considering both discriminative and generative objectives for learning multimodal representations, our full objective is C = L + Amssss-capi where À balances two objectives. We set À = 10 ensure similar magnitudes for both losses in our experiments. In the training phase, we use Adam (Kingma & Ba 2015) to minimize our loss. At inference time, we directly use o(() and (v) to encode video and text representations for retrieval.
3.2 MODEL ARCHITECTURE
We now discuss the details of the encoders and decoder components in our architecture, illustrated in fig. 2 For the text decoder p(tlee) in eq. (2 and (3,, we use a pre-trained T-5 decoder RRffel
et al. 2019).
For the video representation cu = (v) (W'())) we use a video encoder e, = V'(v) followed by multi-layer transformer pooling head C2 (e,) The encoder V V'(v) concatenates the output of pretrained ResNet-152 (He et al. 2016) and R(2+1)D-334 (Tran et al. 2018) networks applied to individual video frames, resulting in a code C2 [c,1 ....... where M is the maximum duration of a video clip. For the pooling head Co = (e,),, we consider a transformer architecture to attend important context and summarize into fixed-length representation Ca- For this, we follow MMT (Gabeur et al. 2020), but with two important differences. First, while MMT uses 7 expert features that results in 7x the sequence length, we only use a transformer to attend to early-fused motion and appearance features as the video representation, thus significantly reducing the sequence length and computational cost. Second, instead of stacking 6 transformer layers to encode the visual stream in MMT, only use shallow two-layer transformer architecture with additional pre-encoders, further increasing model efficiency. As temporal ID-convolutional neural networks (CNNs) (LeCun et al. 1998) were shown to effectively capture temporal dependencies in videos (Dong et al. 2019), we integrate CNNs into our transformer pooling heads to better capture video temporal signals. In more detail, we compute Co = (e.) by chaining two transformer layers, each of the type:
(4) Here f pre-encoder that refines the video representation; we found empirically that ID CNN works well for this purpose. Then, we apply multi-head self-attention (MHA) (Huang et al., 2019 Vaswani et al. 2017) followed by a feed-forward network (FNN) with batch normalization
(e)) = (FFNN(()) + Eatm), Eatm BN(MHA(f(e) + f(e)).
(BN) (Toffe & Szegedy 2015). The architecture maps the input sequence er to a new 'contextual-
ized' sequence of representation vectors; we take the first one as Cy-
((t) = (t))(tt). The text encoder code et = [ee1 ... ., where N is the maximum length of a sentence. The pooling head e (ex))) follows the same design the video case, but f is set to a recurrent neural network (RNN) instead of a CNN. Please refer to the appendix for details.
The text representation decomposes in the same way as
& = ('(1) uses a pretrained T-5 network resulting
In practice, for computational reasons, we use eq. except the video encoder V'(v), which is fixed.
3 to finetune the parameters of all networks
4 EXPERIMENTS
We validate empirically the ability of our method to learn better representations for the downstream tasks of text- to-ttoovideo and video-to-text retrieval. First, in sec. 4.2 we ablate various model com- ponents the MSR-VTV dataset. Then, in sec. 43 show that best model significantly outperforms state-of-the-art retrieval systems on three datasets, MSR-VTT, ActivtyNet and VATEX. Finally, in sec. 4.4 we analyse qualitatively the effect of the attention mechanism used during train- ing.
5

---

Published as a conference paper at ICLR 2021
Table Model Architecture and Training Details Ablation. Vixto retrieval performance on MSR-VTT. Recall@1 5, and Median Recall are shown.
(a) Video Encoder. Stronger features and combina- tion improves performance.
(b) Feature Aggregation. Learning temporal atten- tion yields strong gains over pooling.
Feature source
R@17 f R95 1 MdR
Temporal reduction R01 f R
MdR
R-152 R(2+1)D--4
20.8
46.2
6.0
Max
21.8
49.5
8.0
23.7
53.2
Mean
22.5
51.3
R(2+1)D-34 +R-152
27.2
55.2
3.0
Multi-Head Attn
27.2
55.2
3.0
(c) Text Encoder. Stronger encoding of text improves retrieval.
(d) Text Decoder. Stronger decoding of text im- proves retrieval.
Text Encoder
R@1
R@5 t MdR +
Text Encoder Text Decoder R@1 1 R855 1 MdR
W2V (GloVe)
22.1
49.8
6.0
TS-Base
T5-Small
26.2
54.2
3.0
T5-Small
24.5
51.2
3.0
T5- Base
T5-Basc
27.2
55.2
3.0
Base
27.2
55.2
3.0
(e) Contrastive Loss. Inter- modal Triplet loss yields the best performance.
Contrastive
R@1 f RØ5 f MdR
InfoNCE (inter+intra)
10.7
28.5
15.0
InfoNCE (inter)
10.8
29.0
Triplet (inter+intra)
26.8
56.2
3.0
Triplet (inter)
27.2
55.2
3.0
(I) Support-set Size. Retrieval degrades when reconstructing from too small and too large sets.
Batch-size
Memory bank
Size
8
16
32
64
128
256
512
2k
8k
R@ 1/5
18.5/45.6
20.7/49.9
25.2/54.6 27.2/55.2
28.0/56.1
26.9/55.0
25.3/53.5
26.8/54.7
26.2/52.7
4.1 EXPERIMENTAL SETUP
Datasets. How0oM00M (Miech et al., 2019) is large-scale instructional video collection of 1.2 million YouTube videos, along with automatic speech recognition transcripts. We use this dataset for our pre- pre-trrining experiments. MSR-VTT (Xu et al., 2016) contains 10,000 videos, where each video annotated with 20 descriptions. We report results on the 1k-A split (9,000 training, 1,000 testing) in Liu et al. (2019). VATEX (Wang al.) 2019) is multilingual (Chinese and English) video-text dataset with 34,911 videos. We use the official training split with 25,991 videos and report the validation split as in HGR (Chen al.) 2020b). The ActivityNet Caption (Krishnal et al. 2017) dataset consists of densely annotated temporal segments of 20K YouTube vidcos. We use the 10K training split to train from scratch/ finetune the model and report the performance on the 5K 'vall' split. The MSVD (Chen & Dolan 2011) dataset consists of 80K English descriptions for 1,970 videos from YouTube with each video associated with around 40 sentences each. We use the standard split of 1,200, 100, and 670 videos for training, validation, and testing (Liu al. 2019 Venugopalan et al. 2015b Xu al. 2015).
Evaluation Metries. To measure the rext-to-video and video-to- text retrieval performance, we choose Recall K (R @RK) and Me- dian Rank (MedR), which are common metrics in information re- trieval.
Table 1: Effect of learning objectives. Text-+Video re- trieval on MSR-VTT.
4.2 ABLATIONS
Rait
R MdR
None
25.9
53.0
In Tab. 2 we first only ablate the crossmmodal retrieval part of our network architecture, while the generative objectives are analysed Tab. [
Identity
26.4
51.9
Full
25.8
53.9
3.0
Hybrid
26.0
54.8
3.0
Cross
27.2
55..2
3.0

---

Published as a conference paper at ICLR 2021
Video Encoder. In Tab. 2a show the effect of the choice of visual imput features. We find that for text-to-video retrieval at Recall at 1 and 5 (R@1, R05), features obtained from video R(2+1)D-34 ResNet achieve 2.9% and 7.0% higher performance compared to only image-frame based features from a ResNet-152. A further 3.5% and 2.0% can be gained by concatenating both features, yielding the strongest MdR. of 3.0%.
Feature Aggregation. While the features from both video and image- based visual encoders have re- duced spatial extent after a fully-connected layer, the temporal dimension can be reduced in various ways. In Tab. 2b] we find that our multi-head, parameterized attention reduction yields strong gains over the mean- or max max-ppolligg baselines of over 4% for R@1. This shows that learning attention over the temporal dimension of fixed feature sets can give strong gains even without fine-tuning the encoder.
Text Encoder. In Tab. 2c we find decent gains of 2.7% and 0.4% for R @ 1,5 5 for using T5-base, instead of T5-smlll. We do not use the T-5-Large model, as in Korbar al. (2020), due to the prohibitively large relative model size increase of +220%.
Text Decoder. In Tab. 2d we find that using a larger text decoder gives 1% increase in performance when using the cross-captioning objective.
Contrastive Loss. results of the InfoNCE contrastive loss (Oord et al. (018)) with triplet loss, with both the intra and inter-intra modality variants. We find that InfoNCE (Oord et al. 2018) loss does not work well in our case, likely due to the difficulty in tuning this loss to have the right combination of temperature and batch-size.
validate the choice of a triplet loss in eq. T in Tab. 2e we compare the
Support-Set Size. Lastly, in Tab. 21 show the effect of the size of the support set used for cross-instance captioning. We find that our reconstruction loss indeed acts as a bottleneck, with both smaller and very large sizes degrading the performance.
Captioning Objective. In Tab. we show the effect of the different variants of our learning objec- tive eq. B First, we find that the naive addition of reconstruction objective ("Identity") does not improve the contrastive-only baseline ("None") much. Considering reconstruction from other videos improves the performance more. particular, the "Hybrid" variant, which combines "Identity" and "Full" (sec. 3.1.2) improves Recall at and 5 from 25.9% and 53.0% to 26.0% and 54.8%. respec- tively. However, the best result by far (27.2 55.2%% is obtained forcing captions be reconstructed only from other videos, via our cross-instance attention mechanism ("Cross"). This variant cannot use information contained on the model to discover meaningful relationship between different videos. This newly- newly-proposed scheme seems to have the most beneficial effect for semantic retrieval.
video to generate the corresponding caption and thus entirely relies
Table 3: Retrieval performance on the MSR-VTT dataset. Models in the second group are additionally pretrained on HowTo100m
Text + Video
Video Text
R@11
R@5
R@101 MdR
R@I R@5t R@10f MdR
Random Baseline
0.1
0.5
1.0
500.0
0.1
0.5
1.0
500.0
JSFusion al. 2018
31.2
43.2
13.0
HT100M Miech al
2019
12.1
35.0
48.0
12.0
JPoSE Tay et al 2019)
14.3
38.1
53.0
9.0
16.4
41.3
54.4
8.7
CE (Liu et al. 2019)
20.9
48.8
62.4
6.0
20.6
50.3
64.0
5.3
MMT Gabeur
2020
24.6
54.0
67.1
4.0
21.4
56.0
67.8
4.0
Ours
27.4
56.3
67.7
3.0
26.6
55.1
67.5
3.0
VidTranslate KKrbbrr al.
2020)
14.7
52.8
HT100M Miiech al.] 2019)
14.9
40.2
52.8
9.0
16.8
41.7
55.1
8.0
NoiseEstimation (Amrami ct al.
2020
17.4
41.6
53.6
8.0
UniVI. [Luo et al. (020)
21.2
49.6
63.1
6.0
AVLnet (Kouditchenko
2020
27.1
55.6
66.6
4.0
28.5
54.6
65.2
4.0
MMT (Gabeur et al. 2020)
57.1
69.6
27.0
57.5
69.7
3.7
Ours-pretraine
30.1
58.5
69.3
3.0
28.5
58.6
71.6
3.0

---

Published as a conference paper at ICLR 2021
Table 4: Retrieval performance the VATEX dataset
Text + Video
Video Text
R
R R@10 MdR
R
R
R010f MdR
Random Bascline
0.2
0.7
1.05
2000.5
0.02
0.1
1.02
2100.5
VSE Kiros
2014)
28.0
64.3
76.9
3.0
VSE+ (raghri el al.
2018
33.7
70.1
81.0
Dual Dong et al.
2019)
31.1
67.4
78.9
3.0
HGR Chen et al.
2020b)
35.1
73.5
83.5
2.0
Ours
44.6
81.8
89.5
1.0
58.1
83.8
90.9
1.0
Ours-pretraince
45.9
82.4
90.4
1.0
61.2
85.2
91.8
1.0
Table 5: Retrieval performance ActivityNet
Text Video
Video Text
R&11 n RØ501 MdR R& R&5 f RQ050 MdRj
Random Baseline
0.02
0.1
1.02
2458
0.02
0.1
1.02
2458
Zhang al.
2018
18.2
44.8
89.1
7.0
16.7
43.1
88.4
7.0
CE
Lau
201 9
18.2
17.7
91.4
6.0
17.7
46.6
90.9
6.0
HSE (Zhang et al. 2018
20.5
49.3
18.7
48.1
MMT Gabeur et a.
2020
22.7
54.2
93.2
5.0
22.9
54.8
93.1
4.3
Ours
26.8
58.1
93.5
3.0
25.5
57.3
93.5
3.0
MMTTpretraiied
Gabeur et al. 2020
28.7
61.4
94.5
3.3
28.9
61.1
94.3
4.0
Ours-pretrained
29.2
61.6
94.7
3.0
28.7
60.8
94.8
2.0
Table 6: Retrieval performance on the MSVD dataset
Text + Video
Video -Txx
R 1 R@5t R 10 MdR, H 11 R05t H 10T MdR
VSE (irrss et al. 2014)
12.3
30.1
42.3
14.0
VSF++ |Faghni al. 2018
15.4
39.6
53.0
9.0
Multi. Cues (Mthhnn 2018
20.3
47.8
61.1
6.0
CE (Liu et al. 2019
19.8
63.8
Ours
23.0
52.8
65.8
5.0
27.3
50.7
60.8
5.0
Ours- purs-pretrained
28.4
60.0
72.9
4.0
34.7
59.9
3.0
4.3 COMPARISON TO STATE-OF -EE--------EEEEEE
In this section, we compare the results of our method to other recent text-to-video and video-to-text retrieval approaches on various datasets. In Tab. 3 to 5 we show the results of our model applied to text-to-video and video-to-text retrieval on MSR VTT, VATEX, ActivityNet and MSVD with and without pre-trainig How To (oo00000 100M. Without pre-training, our method outperforms all others in all metrics and datasets. In particular, for the VATEX dataset, our retrieval performance at recall at 1 and is 45.9% and 82.4%, exceeding recent state-of-the-art methods (Chen al. 2020b) by margin of 9%. For Net, our model outperforms MMT by a margin of 4% at recall at T. With pre-training on How NowTooO0e 100M, our performance further increases across the board. Notably, unlike MMT which uses 7 features, our model uses only 2 features and achieves state-of-the-art in most metries.
4.4 ANALYSIS
In order to better understand the effect of our learning objective, we visualize the soft attention of our best-performing cross-instance reconstruction model fig. 3 As we can see in the top-left square, which shows the pairwise attention between all pairs of videos in the batch, it is highly focused, with the model mostly attending one or two other instances in the batch.
8

---

Published as a conference paper at ICLR 2021
For the first video's caption reconstruction (second row), we find that the model solely attends to another musical performance video that is in the batch, ignor- ing the others. For the second video (third row), the model focuses another sample that shows the sea but differs in most other aspects since there are no semantically-equiialenn clips in the batch. The third video shares a similar scenario. These examples show that the bottleneck is effective at forcing the model to avoid memorising the video-caption association of each clip in isolation, and attempt to match other clips more broadly, since an exact (or very close) match is not guaranteed.
050
Fig. 3: Support-set attention map. At- tention scores of all pairs in a batch (top- left square) and subset of rows/columns (other squares) on VTT.
CONCLUSION
In this work, we studied classic contrastive learning methods such as the triplet loss to learn video- text representations for cross-model retrieval. We suggested that the contrastive approach might pull apart videos and captions even when they are semantically equivalent, which can hinder downstream retrieval performance. To mitigate this effect, propose to consider captioning pretext task an additional learning objective. In particular, we show that cross-instance captioning can encourage the representation pull together videos that share similar caption, and thus likely to be equivalent for retrieval. Leveraging these ideas, our model achieves state-of-the-al performance on the text-to-video and video-to-text retrieval tasks, on three datasets.
While we demonstrated these ideas in the specific case of text-to-video retrieval, they can in principle generalize any setting that utilizes contrastive loss, including self-supervised learning, provided that it is possible to learn reasonable conditional generators of modality or data stream given another.
ACKNOWLEDGEMENTS
We are grateful for support from the Rhodes Trust (M.P), the Royal Academy of Engineering (DFR05420, J.H), Facebook (M.P. and P.H.), EPSRC Centre for Doctoral Training in Autonomous Intelligent Machines & Systems [EP/L015897/1] (M.P. and Y.A.) and the Qualcomm Innovation Fellowship (Y.A.). P.H. is also supported by the DARPA grant funded under the GAILA program (award HR001 1190063..

---

Published as a conference paper at ICLR 2021
REFERENCES
Jean- Cean-Baptiste Alayrac, A. Recasens, Rosália G. Schneider, R. Arandjelovié, Jason Ramapuram, J. Fauw, Lucas Smaira, Dieleman, and Andrew Zisserman. Self-supervised multimodal versa- tile networks. ArXiv, abs/2006. 16228, 2020.
Humam Alwassel, Bruno Korbar, Dhruv Mahajan, Lorenzo Torresani, Bernard Ghanem, and
Du Tran. Self-supervised learning by cross-modal audio-video clustering. In NeurlPS, 2020.
Elad Amrani, Rami Ben-Ari, Daniel Rotman, and Alex Bronstein. Noise estimation using density estimation for self-supervised multimodal learning. arXiv preprini urXiv:2003.00 /86, 2020.
Yuki M. Asano, Mandela Patrick. Christian Rupprecht, and Andrea Vedaldi. Labelling unlabelled videos from scratch with multi-modal supe-vsperisi.on In NeurlPS, 2020a.
Yuki M Asano, Christian Rupprecht, and Andrea Vedaldi. Self-labelling via simultaneous clustering and representation learning. In ICLR, 2020b.
Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. Neural machine translation by jointly
learning to align and translate. arXiv preprint arXiv: (eov:1409.0475 0473, 2014.
M. Caron, I. Misra, J. Mairal, Priya Goyal, P. Bojanowski, and Armand Joulin. Unsupervised
learning of visual features by contrasting cluster assignments. In NeurlPS, 2020.
Mathilde Caron, Piotr Bojanowski, Armand Joulin, and Matthijs Douze. Deep clustering for unsu- pervised learning of visual features. In ECCV, 2018.
Joao Carreira and Andrew Zisserman. Quo vadis, action recognition? a new model and the kinetics dataset. In proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp- 6299-6308, 2017.
David L Chen and William B Dolan, Collecting highly parallel data for paraphrase evaluation. In Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Hu- man Language Technologies- Volume I, pp. 190-200. Association for Computational Linguistics,
2011.
Mark Chen, Alec Radford, Rewon Child, Jeff Wu, and Heewoo Jun. Generative pretraining from pixels. In ICML, 2020a.
Shizhe Chen, Yida Zhao, Qin Jin, and Qi Wu. Fine-grained video-text retrieval with hicrarchical graph reasoning. 2020b.
Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. ICML, 2020c.
Yen-Chun Chen, Linjie Li, Licheng Yu, Ahmed El Kholy, Faisal Ahmed, Zhe Gan, Yu Cheng, and Jingjing Liu. Uniter: Learning universal image-text representations. arXiv preprini
arXiv:1909. wttp:9999.1740.... 11740, 2019.
Kevin Clark, Minh- Thang Luong, Quoc V. Le, and Christopher D. Manning. Electra: Pre-training text encoders as discriminators rather than generators, 2020.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale
hierarchical image database. In CVPR. pp. 248-255, 2009.
Michael Denkowski and Alon Lavie. Meteor universal: Language specific translation evaluation for
any target language. In Proceedings of the ninth workshop on statistical machine translation, pp. 376-380, 2014.
Karan Desai and Justin Johnson. Virtex: Learning visual representations from textual annotations.
arXiv preprint arXiv:2006.06666 2020.
Jacob Devlin, Ming- Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. In ACL, 2019.
Jeff Donahue and Karen Simonyan. Large scale adversarial representation learning. In Neurlps, 2019.
Jianfeng Dong, Xirong Li, Chaoxi Xu, Shouling Ji, Yuan He, Gang Yang, and Xun Wang. Dual encoding for zero-example video retrieval. In CVPR, 2019.
Haodong Duan, Yue Zhao, Yuanjun Xiong, Wentao Liu, and Dahua Lin. Omni-sourced webly- supervised learning for video recognition. In ECCV, 2020.
Fartash Faghri. David J Fleet, Jamie Ryan Kiros, and Sanja Fidler. Vse++: Improving visual- semantic embeddings with hard negatives. In BMVC, 2018.
Valentin Gabeur, Chen Sun, Karteek Alahari, and Cordelia Schmid. Multi-modal transformer for video retrieval. In ECCV, 2020.
Deepti Ghadiyaram, Du Tran, and Dhruv Mahajan. Large-scale weakly-supervised pre-training for video action recognition. In CVPR, 2019.
10

---

Published as a conference paper at ICLR 2021
Spyros Gidaris, Praveer Singh, and Nikos Komodakis. Unsupervised representation learning by
predicting image rotations. ICLR, 2018.
Jan J. Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial networks, 2014.
Michael Gutmann and Aapo Hyvärinen. Noise-contrastive estimation: A new estimation principle for unnormalized statistical models. In AISTATS, 2010.
Raia Hadsell, Sumit Chopra, and Yann LeCun. Dimensionality reduction by learning an invariant mapping. In CVPR, 2006.
Tengda Han, Weidi Xie, and Andrew Zisserman. Video representation learning by dense predictive coding. In ICCV Workshops, 2019.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp- 770-778, 2016.
Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick.
Momentum contrast for
unsupervised visual representation learning, 2020.
Lisa Anne Hendricks, Oliver Wang, Eli Shechtman, Josef Sivic, Trevor Darrell, and Bryan Russell. Localizing moments in video with temporal language. In Empirical Methods in Natural Language Processing (EMNLP), 2018.
Geoffrey E Hinton and Ruslan R Salakhutdinov. Reducing the dimensionality of data with neural networks. Science, 2006.
Jingyi Hou, Xinxiao Wu, Wentian Zhao, Jiebo Luo, and Yunde Jia. Joint syntax representation learning and visual cue translation for video captioning. In ICCV, 2019.
Po- Yao Huang, Xiaojun Chang, and Alexander Hauptmann. Multi-head attention with diversity for learning grounded multilingual multimodal representations. In Proceedings of the 2019 Confer- ence on Empirical Methods in Natural Language Processing and the 9th International Joint Con- ference on Natural Language Processing (EMNLP-JJCNLP, pp. 1461-1467, November 2019. Po- Yao Huang, Junjie Hu, Xiaojun Chang, and Alexander Hauptmann. Unsupervised multimodal neural machine translation with pseudo visual pivoting. In Proceedings of the 58th Annual Meet- ing of the Association for Computational Linguistics, pp. 8226-8237. Association for Computa- tional Linguistics, July 2020.
Po-Yao Huang, Mandela Patrick, Junjie Hu, Graham Neubig, Florian Metze, and Alexander Hauptmann. Multilingual multimodal pre-training for zero-shot cross-lingual transfer of vision- language models. In Meeting of the North American Chapter the Association for Computational Linguistics (NAACL), June 2021.
Sergey loffe and Christian Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. arXiv preprint arXiv:1502.03 2015.
Xu Ji, João F. Henriques, and Andrea Vedaldi. Invariant information clustering for unsupervised image classification and segmentation, 2018.
Yannis Kalantidis, Mert Bulent Sariyildiz, Noe Pion, Philippe Weinzaepfel, and Diane Larlus. Hard negative mixing for contrastive learning. In NewrlPS, 2020.
Prannay Khosla, Piotr Teterwak, Chen Wang, Aaron Sarna, Yonglong Tian, Phillip Isola, Aaron Maschinot, Ce Liu, and Dilip Krishnan. Supervised contrastive learning. 2020.
Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In ICLR, 2015. Ryan Kiros, Ruslan Salakhutdinov, and Richard S Zemel. Unifying visual-semantic embeddings with multimodal neural language models. arXiv preprint arXiv: 141.2394 2014. Bruno Korbar, Fabio Petroni, Rohit Girdhar, and Lorenzo Torresani. Video understanding as ma- chine translation, 2020.
Ranjay Krishna, Kenji Hata, Frederic Ren, Li Fei-Fei, and Juan Carlos Niebles. Dense-captioning events in videos. In CVPR, 2017.
H. Kuehne, H. Jhuang, E. Garrote, T. Poggio, and T. Serre. HMDB: a large video database for human motion recognition. In ICCV, 2011.
Yann LeCun, Léon Bottou, Yoshua Bengio, and Patrick Haffner. Gradient-based learning applied to document recognition. Proceedings of the IEEE, 86(11):2278-2 Jie Lei, Licheng Yu, Mohit Bansal, and Tamara Berg. Tvqa: Localized, compositional video ques- tion answering. In EMNLP, pp. 1369-1377, 2018.
1998.
Mike Lewis. Marjan Ghazvininejad, Gargi Ghosh, Armen Aghajanyan, Sida Wang, and Luke Zettle- moyer. Pre-training via paraphrasing, 2020a.
11

---

Published as a conference paper at ICLR 2021
Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer
Levy, Ves Stoyanov, and Luke Zettlemoyer. Bart: Denoising sequence-to-seqqeene pre-training for natural language generation, translation, and comprehension. In ACL, 2020b.
Gen Li, Nan Duan, Yuejian Fang, Ming Gong, Daxin Jiang, and Ming Zhou. Unicoder-vi: A universal encoder for vision and language by cross-modal pre-training. In AAAI, 2020a.
Junnan Li. Pan Zhou, Caiming Xiong, Richard Socher, and Steven C. H. Hoi. Prototypical con- trastive learning of unsupervised representations. 2020b.
Liunian Harold Li, Mark Yatskar, Da Yin, Cho-Jui Hsieh, and Kai-Wei Chang. Visualbert: A simple and performant baseline for vision and language, 2019.
Yehao Li, Ting Yao, Yingwei Pan. Hongyang Chao, and Tao Mei. Jointly localizing and describing
events for dense video captioning. In CVPR, 2018.
Yein Lin. Rouge: A package for automatic evaluation of summaries. In Text summarization branches out, pp. 74-81, 2004.
Yang Liu, Samuel Albanie, Arsha Nagrani, and Andrew Zisserman. Use what you have: Video
retrieval using representations from collaborative experts. In BMVC, 2019.
Jiasen Lu, Dhruv Batra, Devi Parikh, and Stefan Lee. Vilbert: Pretraining task-agnostic visiolin- guistic representations for vision-and-language tasks. In Neurips, 2019.
Huaishao Luo, Lei Ji, Botian Shi, Haoyang Huang, Nan Duan, Tianrui Li, Jason Li, Taroon Bharti, and Ming Zhou. Univl: A unified video and language pre-tran vee-raining model for multimodal under- standing and generation, 2020.
Antoine Miech, Dimitri Zhukov, Jean-Baptiste Alayrac, Makarand Tapaswi, Ivan Laptev, and Josef Sivic. Howto Learning a tixt-video embedding by watching hundred million narrated video clips. In ICCV, 2019.
Antoine Miech, Jean-Baptiste Alayrac, Lucas Smaira, Ivan Laptev, Josef Sivic, and Andrew Zis- serman. End-to-end learning of visual representations from uncurated instructional videos. In CVPR, 2020.
Tomas Mikolov, Kai Chen, Greg Corrado, and Jeffrey Dean. Efficient estimation of word represen- tations in vector space. arXiv preprint arXiv:1301.3781, 2013.
Ishan Misra and Laurens van der Maaten, Self-supervised learning of pretext-invariant representa- tions. In CVPR, 2020.
Niluthpol Chowdhury Mithun, Juncheng Li, Florian Metze, and Amit K Noy-Chowdhury, Learning joint embedding with multimodal cues for cross-modal video-text retrieval. In Proceedings of the 2018 ACM on International Conference on Multimedia Retrieval, pp. 19-27, 2018.
Pedro Morgado, Nuno Vasconcelos, and Ishan Misra. Audio visual instance discrimination with cross-modal agreement. arXiv preprint arXiv:2004.12944
2020.
Aaron van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predic- tive coding. arXiv preprini arXiv: 1807.03788 00777,0748,2. 2018
Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. Bleu: method for automatic
evaluation of machine translation. In Proceedings of the 40th annual meeting of the Association for Computational Linguistics, pp. 311-318, 2002.
Mandela Patrick, Yuki M. Asano, Polina Kuznetsova, Ruth Fong. João F. Henriques, Geoffrey Zweig, and Andrea Vedaldi. Multi-modal self-supervision from generalized data transformations, 2020.
Jeffrey Pennington, Richard Socher, and Christopher D Manning. Glove: Global vectors for word representation. In Proceedings of the 2014 conference on empirical methods in natural language processing (EMNLP), pp. 1532-1543, 2014.
AJ Piergiovanni, Anelia Angelova, and Michael S. Ryoo. Evolving losses for unsupervised video representation learning. CVPR, 2020.
Zhaofan Qiu, Ting Yao, and Tao Mei. Learning spatio-temporal representation with pseudo-3d residual networks. In proceedings of the IEEE International Conference on Computer Vision, pp. 5533-55411 2017.
Alec Radford, Luke Metz, and Soumith Chintala. Unsupervised representation learning with deep convolutional generative adversarial networks. arXiv preprini arXiv: /5 1.06444, 2015. Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. Language models are unsupervised multitask learners. OpenAl Blog, 18):9, 2019.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. arXiv preprint arXiv: /910. 10683, 2019.
12

---

Published as a conference paper at ICLR 2021
Anna Rohrbach, Atousa Torabi, Marcus Rohrbach, Niket Tandon, Chris Pal, Hugo Larochelle, Aaron Courville, and Bernt Schiele. Movie description. International Journal of Computer Vi- sion, 2017.
Andrew Rouditchenko, Angie Boggust, David Harwath, Dhiraj Joshi, Samuel Thomas, Kartik Audhkhasi, Rogerio Feris, Brian Kingsbury, Michael Picheny, Antonio Torralba, al. Avi- net: Learning audio-visual language representations from instructional videos. arXiv preprint arXiv:2006.00 199: 2020.
Mert Bulent Sariyildiz, Julien Perez, and Diane Larlus. Learning visual representations with caption annotations, 2020.
Khurram Soomro, Amir Roshan Zamir, and Mubarak Shah. UCFI0I: A dataset of 101 human action classes from videos the wild. In CRCV-TR-12-01, 2012. Weijie Su, Xizhou Zhu, Yue Cao, Bin Li, Lewei Lu, Furu Wei, and Jifeng Dai. VI-bert: Pre-training of generic visual-linguistic representations, 2019.
Chen Sun, Abhinav Shrivastava, Saurabh Singh, and Abhinav Gupta. Revisiting unreasonable ef- fectiveness of data in deep learning era. 2017.
Chen Sun, Fabien Baradel, Kevin Murphy, and Cordelia Schmid. Learning video representations using contrastive bidirectional transformer, 2019a.
Chen Sun, Austin Myers, Carl Vondrick, Kevin Murphy, and Cordelia Schmid. Videobert: A joint model for video and language representation learning. In ICCV, 2019b.
Ilya Sutskever, Oriol Vinyals, and Quoc V Le. Sequence to sequence learning with neural networks.
In Neurlps. pp. 3 3104-3112, 2014.
Hao Tan and Mohit Bansal. Lxmert: Learning cross-modality encoder representations from trans- formers. In EMNLP, 2019.
Bart Thomee, David A Shamma, Gerald Friedland, Benjamin Elizalde, Karl Ni, Douglas Poland,
Damian Borth, and Li-Jia Li. Yfcc100m: The new data in multimedia research. Communications of the ACM, 64-2))6473.. 2016.
Du Tran, Heng Wang. Lorenzo Torresani, Jamie Ray, Yann LeCun, and Manohar Paluri. A closer look at spatiotemporal convolutions for action recognition. In CVPR, 2018.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez,
Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Neurlps, 2017.
Ramakrishna Vedantam, C Lawrence Zitnick, and Devi Parikh. Cider: Consensus- basedd image description evaluation. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 4566-4575, 2015.
Subhashini Venugopalan, Marcus Rohrbach, Jeff Donahue, Raymond Mooney, Trevor Darrell, and Kate Saenko. Sequence to sequence video to text. In Pruceedings of the IEEE International Conference on Computer Vision (ICCV), 2015a.
Subhashini Venugopalan, Marcus Rohrbach, Jeff Donahue, Raymond Mooney, Trevor Darrell, and Kate Saenko. Sequence sequence video to text. In ICCV, 2015b.
Limin Wang, Yuanjun Xiong, Zhe Wang. Yu Qiao, Dahua Lin, Xiaoou Tang, and Lue Van Gool. Temporal segment networks for action recognition in videos. IEEE transactions on pattern anal- ysis and machine intelligence, (1(11):27444 (11)::74402755, 2018
Xin Wang, Jiawei Wu, Junkun Chen, Lei Li, Yuan-Fang Wang, and William Yang Wang. Vatex: A large-scale, high-quality multilingual dataset for video-and-language research. In Proceedings of the IEEE International Conference on Computer Vision, pp. 4581-4591, 2019.
Michael Wray. Diane Larlus, Gabriela Csurka, and Dima Damen. Fine-grained action retrieval through multiple pfrts-of-speech embeddings. In ICCV, 2019.
Zhirong Wu, Yuanjun Xiong, Stella X. Yu, and Dahua Lin. Unsupervised feature learning via non- parametric instance discrimination. CVPR, 2018
Saining Xie, Chen Sun, Jonathan Huang, Zhuowen Tu, and Kevin Murphy. Rethinking spatiotem- poral feature learning: Speed-accuracy trade-offs in video classification. In Proceedings of the European Conference on Computer Vision (ECCV), pp. 305-321, 2018.
Jun Xu, Tao Mei, Ting Yao, and Yong Rui. MSR-VTSTT A large video description dataset for bridging video and language. In CVPR, 2016.
Ran Xu, Caiming Xiong, Wei Chen, and Jason J Corso. Jointly modeling deep video and composi-
tional text to bridge vision and language in unified framework. Citeseer, 2015.
I. Zeki Yalniz, Hervé Jégou, Kan Chen, Manohar Paluri, and Dhruv Mahajan. Billion-scale semi- supervised learning for image classification, 2019.
Youngjae Yu, Jongseok Kim, and Gunhee Kim. A joint sequence fusion model for video question answering and retrieval. In ECCV, 2018
13

---

Published as a conference paper at ICLR 2021
Bowen Zhang. Hexiang Hu, and Fei Sha. Cross- moosll and hierarchical modeling of video and text.
In ECCV, pp. 374-390, 2018.
Richard Zhang, Phillip Isola, and Alexci A Efros. Colorful image colorization. In ECCV, 2016. Ziqi Zhang, Yaya Shi, Chunfeng Yuan, Bing Li, Peijin Wang, Weiming Hu, and Zheng-Jun Zha. relational graph with teacher-recommendee learning for video captioning. In CVPR, 2020. Luowei Zhou, Chenliang Xu, and Jason J Corso. Towards automatic learning of procedures from web instructional videos. 2018a.
Luowei Zhou, Yingbo Zhou, Jason Corso, Richard Socher, and Caiming Xiong. End-to-end dense video captioning with masked transformer. In CVPR, 2018b. Linchao Zhu and Yi Yang. Actbert:
Learning global-local video-text representations. In CVPR,
2020.
14

---

Published as a conference paper at ICLR 2021
6 APPENDIX
The appendix organized as follows: First, we provide more details about our model. Then we introduce the datasets and the experimental setup. Finally, we provide additional qualitative and quantitative experimental results for video-text retrieval and captioning.
6.1 MODEL DETAILS
Implementation details and hyper parameters. For our text encoder, we use the T5-base model pre-trained on the "Colossal Clean Crawled Corpus" (C4) (Raffel et al. 2019). We use its corre- sponding text tokenizer and encode sentence into sequence of 1024 dimensional vectors.
For our visual encoder, our model utilizes only the motion and the appearance features. For the mo- tion feature, we use a 34-layer, R(2+1)-D (Tran et al.] (0188) model pre-trained on IG65M (Ghadi- yaram et al., 2019) and apply spatial-temporal average pooling over the last convolutonal layer, resulting in a 512-dimensiona vector. For the appearance feature, we use the 2048-dimension flat- tened pool-5 layer of the standard ResNet152 (He et al. 2016) pre-trained Imagenet (Deng al.) 2009). We extract features at a rate of feature per second and simply concatenate the two features, resulting in 2560-dimension visual input stream. Noteworthily, instead of using and different types of visual features as in CE (Liu et al. 2019) and MMT (Gabeur et al., 2020 we use only the above 2 features and achieve on par or superior performance. Also, with early fusion, our model does not suffer from additional computation required for the extended sequence length in MMT. For the text decoder, we use the T5-base model decoder, also pre-trained on C4.
As illustrated in Fig. our transformer pooling head is composed of a pre-encoder, multi-heee head self-attention (MHA), and feed-forward layer (FFN), For pre-encoders, we use one-layer MLP with d-dimensional output for mapping video features into the common embedding space. We 1024-dimension bi-directional GRU as the text pre-encoder. For the ID-CNN prior, we use kernels with size [2, 3, 4, 6] as the visual and text pre-encoders. We set the embedding dimension to 1024 and use 4 attention heads in the transformer pooling layers. The hidden dimension of FFN is 2048.
FFN
MHA
Training and Inference time. Pre-training on 1.2 million How To100M videos takes around 160 GPU hours (NVIDIA V100) for 20 epochs. We speed up the pre-training process by distributing the workload over 8 GPUs. We use GPU for the fine-tume tine-tuning or training from scratch experiments. For the AISRVVTT 1k-A split, it takes 12 GPU hours to train our full model on 180K video-text pairs for 20 epochs. For Vatex, it takes 32 GPU hours train on 260K video-text pairs for 30 epochs. For ActivityNet, it takes 2.5 GPU hours to train on 10K video-text paris for 28 epochs.
x L
Fig. 4: Transformer pooling head.
For inference, the encoding speed is around 250-300 video/sec and 200-250 text query/sec. The overall text-to-video search speed on 5,000 video-text pairs (5,000 text queries over 5,000 videos) is 30 seconds including encoding. The speed of text-to-video retrieval is similar to video-to-text retrieval.
6.2 EXPERIMENT DETAILS
The margin a of the max max-margin loss 0.2, and the temperature T is set 0.1 as used in Sim- CLR Chen et al. (200cc). We use the Adam (Kingma & Ba, (015)) optimizer with a initial learning rate 5- 10 and clip gradients greater than 0.2 during the training phase. Dropout rate is 0.3 for all datasets besides ActivityNet (0.0).
As the average video/text lengths and videos available are quite different across datasets, we adjust our training scheme accordingly. When training on MSR-VTI VTT, ActivtyNet and Vatex, batch-size is set to 64. For MSR-VTT training, we sample and truncate videos to 32 seconds, text to 100 tokens and train for 20 epochs. For Vatex, videos are most 64 seconds and we train for 30 epochs. For ActivtityNet training, videos are at most 512 seconds and 256 tokens for the text part. We train
15

---

Published as a conference paper at ICLR 2021
for 28 epochs ActivityNet. For fine-tuning HowTo100M pre-trained model, we reduce training epochs into quarters.
6.3 DATASET DETAILS
HowTo100M Miech al. 2019 is large-scale instructional video collection of 1.2 million Youtube videos, along with automatic speech recognition transcripts. There are more than 100 million clips (ASR segments) defined in HowTo100M. We use this dataset for pretraining.
MSR-VTT (Xu al. 2016) contains 10,000 videos, where each video is annotated with 20 descrip- tions. For retrieval experiments and ablation studies, we follow the training protocol and defined in Gabeur et al.] (2020); Liu et al.] (2019); Miech et al. (2019) and evaluate on rext-to-video and video-to-text search tasks on the Ik-A testing split with 1,000 video or text candidates defined by Yu al. 2018). For captioning task, we evaluate on the standard testing split with 2,990 videos.
VATEX (Wang al., 2019) multilingual (Chinese and English) video-text dataset with 34,911 videos. We the official split with 25,991 videos for training. As the testing annotations private in VATEX, we follow the protocol in Chen et al. (2020b) to split the validation set equally (1,500 validation and 1,500 testing videos) for model selection and testing. For each video, 10 English and 10 Chinese descriptions are available, and we only use the English annotations.
AetivityNet Dense Caption dataset consists densely annotated temporal segments of 20K YouTube videos. Following Gabeur et al.] (2020); Zhang et al. (2018), we concatenate descriptions of seg- ments in video to construct "video-pan "arecooprraraph"" for retrieval and captioning. We use the 10K training split to train from scratch/ finetune the model and report the performance on the 5K 'vall' split.
MSVD dataset consists of 80K English descriptions for 1,970 videos from YouTube, with each video associated with around 40 sentences each. We use the standard split of 1200, 100, and 670
videos for training, validation, and testing (Liu al. 2019 Venugopalan et al. 2015).
2015b Xu al.
6.4
VIDEO CAPTIONING EXPERIMENTS
To measure captioning/text generation performance, we report BLEU4 (Papineni et al. TEOR (Denkowski & Lavie 2014), Rogue-L (Lin 2004) and CIDEr (Vedantam et al. rics. We report results on the MSR-VTT, VATEX and ActivityNet datasets.
2002), ME-
2015 met-
Table 7: Captioning performance on the MSR-VTT dataset
Captioning
BLUE4
METEOR Rogue-L CIDEr
VidTranslate (Krbbar et al.
2020
41.7
28.5
POS+VCT (Hou et al. 2019
42.3
29.7
62.8
49.1
ORG (Zhang al. 2020)
43.6
28.8
62.1
50.9
Ours, MSR-VTT only Ours, HT100M + MSR-VTI VTT
39.7
28.3
60.5
46.5
38.9
28.2
59.8
48.6
Table 8: Captioning performance on the VATEX dataset
Captioning
Blue@4
METEOR
Rogue-L
CIDEr
Shared Enc-Dec (Wang et al.
2019
28.4
21.7
47.0
45.1
ORG (Zhang et al. 2020
32.1
22.2
48.9
49.7
Ours, VATEX only
32.8
24.4
49.1
$1.2
Ours, HT100M Vatex
32.5
24.1
48.9
50.5
16

---

Published as a conference paper at ICLR 2021
Table 9: Captioning performance on the ActivtyNet dataset
Captioning
Blue@4
METEOR Roguc-L, CIDEr
DENSE Krishna et al. 2017
1.6
8.9
DVC-D-A al.
2018)
9.3
Bi-LSTM+ TempoAtin
Zhhou 2018b
10.0
Masked Transformer (Zhou et al.
2018b
2.8
11.1
Ours, ActivityNet only Ours, HT100M + ActivityNet
1.5
6.9
17.8
3.2
1.4
6.9
17.5
3.1
6.5 ZERO-SHOT RETRIEVAL EXPERIMENTS
We also evaluate our model in the zero-shot setting on MSR-VTI VTT. Vatex, ActivityNet and MSVD, after pre-training on HT 100M. While are able get reasonable results on MSR-VTI and MSVD, our results are not great on Vatex and Activity- Ceet due to significant domain gap.
Table 10: Zero-shot Retrieval performance on VATEX, MSR-VTI. MSVD and ActivityNet.
Text - Video
Video + Text
R@11
R
R
MdRl
R@11
R
R
MdRl
Zero-Shot
ActivityNet
0.06
0.2
0.5
1907.0
0.0
0.2
0.3
2238.0
VATEX
0.07
0.4
0.7
682.0
0.07
0.4
0.9
697
MSVD
8.9
26.0
37.9
18.0
21.4
46.2
57.7
6.0
MSR- VTT
8.7
23.0
31.1
31.0
12.7
27.5
36.2
24.0
6.6 ACTION RECOGNITION
Lastly, evaluate model the video action recognition task HMDB-51 (Kuehne al., 2011) and UCF-101 (Soomro et al.) 2012). For this, we use the R(2+1 D)334 (pretrained on IG65M) model well ResNet- 152 model (pretrained feature per second per video by concatenating the features from each model (2560-D), and obtain an average representation per video using either average pooling (2560-D) our proposed trans- former pooling head (1024-D) pre-trained on HT 100M using cross-captioning objective. We then train a linear classifier for 1500 epochs for HMDB-51 (500 for UCF-101) on these features using Adam (Kingma & Ba 2015) optimizer with learning rate of 1 and weight decay Ie-4 with early stopping. We also drop the learning rate by 10 at epochs 200, 400 for UCF-101 and 1000, 1200 for HMDB-51. 1, In Table [I] we show the results of training only linear-layer on features extracted from our fixed backbone with or without a learned former temporal pooling head provides significant benefits over the baseline of simply average pool- ing the features, demonstrating the effectiveness of building contextualized representations using our proposed transformer. In particular, UCF-101 by replacing average pooling with our transformer pooling head to aggregate features. We observe that naive average pooling performs significantly worse than transformer pooling under evaluation protocol. This is likely because 1) the average pooling collapses temporal information, making the linear layer based classification difficult 2) compared to the transformer pooling, it does not benefit from large-scale pretraining a wide variety of action videos of HT100M. We further compare very favorably the current state-of-the-au approaches. In particular, we outperform all other approaches, both supervised and self suef-supervisdd, except the recently introduced Omni (Duan) et al. 2020) which was finetuned on both UCF-101 and HMDB-51, while we only trained linear classifier extracted features. However, it should be noted that it is very difficult to fairly com- pare all these different approaches because they may use different modalities (images, RGB video, optical flow, audio, ASR outputs), pretraining datasets (Kinetics-40V HT100M IG65M, Imagenet), architectures (S3D, I3D, R(2+1)D, R3D), pre-training (supervised, self-supervised) and downstream training (frozen, finetuned) strategies.
Imagenet), in method. We extract
connsfomer-ppolliie head. We find that our trans-
improvements of over 7% on HMDB-51 and 34%
17

---

Published as a conference paper at ICLR 2021
Table 11: Action recognition. Results of training only a linear-layer, on features extracted from our fixed backbone with or without learned transformer-pood pooolnng--oiigggggog head. We compare to the state-of-art supervised and self-supervised pretrainig methods on the HMDB-51 and UCF-101 action recogni- tion task, for different downstream training protocols ("FT?" stands for finetuned). We report aver- age Top-1 accuracy across folds. Dataset abbreviations: AudioSet, HMDB51, HowTo 100M, Instagram65M, IMaennet- 1000, Kinetics400, OmniSource Images Videos, Sports UCF101, YouTube8M. Other abbreviations: Video modality, Flow modality, Image modality, Audio modal- ity, Transformer pooling, Average pooling
Method
Mod
Dataset
Model
FT?
H51
U101
Self-Supervised Pre-training
MIL-NCE (iech ct al.
2020
VTT
HM
S3D-G
53.1
82.7
MIL-NCE (Mecch ct al 2020
V.T
HM
S3D-G
V
61.0
91.3
MMV (Alayrac et al. 2020)
V.T.A
HM+AS
TSM-50x2
X
67.1
91.8
ELo Piergiovanni al.
20200
V.EAA
YT8M
R(2+1)D-50x3
67.4
93.8
XDC Alwassel el 2020)
V,A
IG6SM
R(2+1)D-18
95.5
GDT Patrick et al.
2020
V.A
IG65M
R(2+I)D-18
72.8
95.2
MMV Alaayrcc et a.
20200)
V.T.A
HM+AS
TSM-50x2
75.0
95.2
Supervised Pre-training
P3D
QQu et al. 2017)
VI
SIM+IM
P3D
88.6
TSN Wang
2018
VII
IM
TSN
69.4
94.2
13D Carreira Zisserman
2017
V,I
K400+IM
13D
74.8
95.6
R(2+1)D (Tran et al. 2018
V
K400
R(2+DD-33
74.5
96.8
S3D-G (Xie ct al. 2018)
V.I
K400+IM
S3D-G
75.9
96.8
13D
(Carreira & Zisserman
2017
VII
K400+IM
I3D
77.1
96.7
R(2+1)D (Tan
2018
V
K400
R(++1)D-34
76.4
95.5
R(2+1)D Tran
2018
V.F
K400
R(2+))D-34X2
78.7
97.3
Omni (Duan et al.
2020)
VII
K400+OS
Slow-8x8-R101
79.0
97.3
13D Carreira & Zisserman
2017)
V.FII
K400+IM
13Dx2
80.7
98.0
Omni Duan al.
2020
V.I.I
K400+0S
Slow- 8x8-R101x2
83.8
98.6
Ours (Avg-pooling)
V.I
IG6SM+IM
R2+1)D-34+RR52
73.7
64.3
Ours (T-pooling)
V.I
HM+1G65M+IM
R(+1))D-4+R152
81.3
98.0
STATISTICAL SIGNIFICANCE
In Table show the results of finetuning pretrained model for times the VATEX dataset. We find that the variance is quite low and our model consistently beats the state the art.
Table 12: Retrieval performance on the VATEX dataset
Text + Video
Video - Text
R@
R@5t
R@10t
MdRj
R@1t
R@5t
R@10f
MdR
Random Baseline
0.7
1.05
2000.5
0.02
0.1
1.02
2100.5
VSE (Kros et al.
2014
28.0
64.3
76.9
3.0
VSE++ (Taghn el al
2018)
33.7
70.1
81.0
2.0
Dual (Dong
2019)
31.1
67.4
78.9
3.0
HGR Chen et al.
2020b
35.1
73.5
83.5
Ours
44.9
82.1
89.7
1.0
58.4
84.4
91.0
1.0
18

---

Published as a conference paper at ICLR 2021
6.8 ADDITIONAL QUALITATIVE RESULTS
We provide addition qualitative text-to-video retrieval results on MSR-VTT VATEX, ActivityNet in Fig. 5 Given text query, in most cases, model successfully retrieves the correct videos marked in green.
person swimming in white water rapids
showing the interior of
1
(0.77)
2
(0.62)
110.71
3
10.61
(0.69)
(a)
MSR-VTT
kid riding harse while talking
skiing down the mountain smoothly
10:83)
(0.80)
2
(0.74)
(0.73)
(0.64)
(0.65
(b) VATEX
A woman is speaking to the camera while holding accordion and moving her hands around She demonstrates how to play the instrument while still speaking the camera and moving all around
close of nails and nail polish. A person is then wiping polish onto pad and rubbing the object all over her nails. She then puts a coating over the nail and shows off again
followed by shot of brushes
2
Caloeg
(0.73)
10
3
(0.64)
(c) ActivityNet
Fig. 5: Examples of top-3 Text-> Video retrieval results and similarities on the MSR-VTT, VATEX, and ActivityNet testing set. Only one correct video (colored in green) for each text query the top.
19


Reviews:
[{"review_id": "peN70Rl18Xt", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "## Summary\n\nThe paper proposes a new way to train joint representation of video and text. In particular, it argues that the contrastive based approach is too strict in the sense that it forces pairs of video and text to be far away only because they are not from the same instance without considering potential shared semantic. To address this, they propose to combine the contrastive loss with a generative loss that tries to generate the text from one video based only on *different* videos within the batch. Interestingly, adding this objective improves performance of retrieval.\n\n## Strengths\n\n* The paper is clearly written which makes it easy to understand\n* To me the fact that the Cross approach works best is not completely intuitive which makes the result quite interesting. This approach is novel to the best of my knowledge.\n* The ablation study is thorough and overall well conducted.\n* The final performance are quite impressive (albeit some difficulty to fully compare to previous work as explained in the weaknesses section). \n\n## Weaknesses\n\nThere are a few questions/concerns about the work that I hope can be answered in the rebuttal.\n\n**Is the support set idea always good?**: While this idea seems to work well on MSRVTT I wonder whether or not the same loss would work well on much larger scale and more diverse datasets. In particular the method seems to rely strongly on the fact that there are other semantically related videos/text in the batch. While this may be true for small scale datasets it might not always be the case. I see a few options to try to illustrate a bit more that point:\n\n* Is the support set idea also important when training on HowTo100M? Does it matter there? Does it hurt?\n* I would expect the support set size to depend on the dataset? Was this observed in practice? \n\n**What are zero shot performance when training on HowTo?** I would be curious to know what would be the ZS performance when just training on HowTo. In particular (related to the previous question) is the generation loss also important there at all?\n\n**Comparison to the state-of-the-art is not totally fair**: even though less features are used, the pretrained models are trained on different data (IG65M dataset), therefore it's not clear that the comparison is fair compared to MMT. Would it be possible to do the comparison using the same set of features? Currently its not clear if the best performance comes from the fact that the method is better or if the input features are better.\n\n**Technical questions about aggregation**:\n\n* For the text encoder, it does not seem to be standard to combine RNN with a Transformer. Was it really important? Similarly for the vision model, how important was it to add the convolution in the transformer?\n\n* Have you tried other options for $c_v$ (e.g. by averaging all the outputs instead of just taking the first? Asking since in that case it seems that no special [CLS] token was added so taking the first token looked arbitrary.\n\n* When averaging temporal representations of the different videos in equation (3), this somewhat assume that all videos are temporally aligned. However this is often not true. Is this a problem that the authors have considered?\n\n\n**Some details about the tables**:\n\n* In Table 1. e) why the numbers with batch size 64 do not match the other best numbers in the table (55.7 vs 55.2).\n* Table 1. d). What does inter+intra means? In particular for the intra does it mean you do a contrastive loss between video and video? Are there any augmentations that are used for that case? I might have missed this but I could not find the details of that in the paper.\n\n**Comment about the title + question**: when I first read the title I thought that visual features would also be learned end to end (due to the term representation learning). I wonder if representation learning is the best term to put in the title due to the potential confusion. Related to that, have you tried to also finetune the video networks? If yes, were there benefits? \n\n## Justification of rating\n\nThe paper presents a simple but effective idea for training joint text and video models. The paper is clearly written and the results are compelling and do support the claim of the paper. There are still some things that can be improved about the paper and I am expecting the authors to reply to my questions and concerns. \n\n## Post rebuttal comment\n\nThe authors have successfully answered my concerns and added the necessary experiments. I think the paper should be accepted as the support set idea is an important one to improve upon the simple contrastive idea (in particular to soften the set of positives). \n", "extra_scores": {}, "extra_text": {}}, {"review_id": "aGhPYBAP4rX", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper concerns the problem of video-text representation learning and its application to video-text retrieval. The source of pre-training data comes from YouTube video-ASR pairs. The main novelty is the adoption of a generative objective (i.e., video captioning) to refine the video encoder ($\\Psi''$) and the text encoder, given the paired data input. Further, the paper observes that sometimes videos belong to the same batch could share similar characteristics/semantics, and therefore traditional contrastive learning objectives that repel positive samples from these hard negative samples could potentially hurt the learned representation. To this end, this paper introduces an idea called support-set bottlenecks, which effectively alleviate this conflict through batch-wise attention.\n \nLike the review title indicated, the paper has demonstrated strong empirical results on some common benchmarks. The related work is mostly comprehensive. The method section is also described with good clarity. However, the main concerns include the somewhat weak motivation of the method and some overstated claims. The reasons are detailed as follows.\n\ni) The motivation behind using a support set is still unclear, besides the obvious reason of empirical gain. Compare to related techniques such as semi-hard negative mining, what are their relations and the advantage of the proposed approach? Also, since the idea is generic to the training objective, what is the rationale for not using it for the contrastive objective (but instead on the more complicated captioning objective)? Or if you have made attempts, what are the observations?\n\nii) The overall scope of the proposed method is somewhat overstated. Despite that the method is generic to learn video-text representation, the downstream tasks only involve a single type of problem (i.e., video-text retrieval), which raises concern on its effectiveness across a broader range of tasks. Besides, even for video-text retrieval, notable benchmarks such as [YouCook2](https://www.aaai.org/ocs/index.php/AAAI/AAAI18/paper/viewFile/17344/16367) and [MSVD](https://www.aclweb.org/anthology/P11-1020.pdf) are missing, not even in the related work section. Consider acknowledging these works.\n\niii) From Tab. 2, we know that the model variant \"Cross\" works the best. But chances are there, other videos from the same batch might be totally irrelevant to the video-of-interest (more pronounced when the batch is small). Could this potentially hurt the learned representation? Also, any insight on what intuitively makes \"Cross\" better than other model variants?\n\niv) The conclusion drawn on Contrastive Loss from this work (page six bottom) is opposite to existing works, that is, triplet loss works much better InfoNCE. Any insight or observation besides these mentioned in the paper would be appreciated.\n\nvi) Consider the relevance of [Prototypical Contrastive Learning](https://arxiv.org/pdf/2005.04966.pdf) to this paper (also alleviate the over-discrimination/\"false negative\" issue on instance discrimination), please acknowledge the work in the related work section.\n\nMinor comments:\n\ni) Please check if the font style follows ICLR'21 guideline, for example, for the title and references.\n\nii) The green arrow between the positive pair is missing in Fig. 1b.\n\n=========================== Post-Rebuttal ===========================\n\nMy questions are well addressed in the rebuttal. I acknowledge the novelty in the paper and the convincing empirical results. The authors have committed to include more datasets to enhance the result completeness and add relevant but missing related work. Taking all these into account, I am adjusting my final rating to Accept.", "extra_scores": {}, "extra_text": {}}, {"review_id": "KPvs7QWw6cf", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary:\nThe paper introduces an interesting problem when doing noise contrastive learning. In particular, the model will push away the representations of all negative pairs although these samples sometimes are semantically related. To alleviate this problem, the authors develop a generative model to push these related samples together by reconstructing the caption from the weighted feature of support samples. Overall, the insight of semantically related pairs in negative sampling is interesting for me. My major concern is about the architecture of the proposed model, the performance on other tasks, and some additional ablation studies (see below).\n\nPros: \n1. The paper takes an important issue in contractive learning: semantically related samples in negative sets.\n2. The paper proposes the cross-instance captioning to alleviate the discouraging concept sharing among samples. The comprehensive experiments on retrieval tasks show SOTA results with a margin.\n\nCons: \n1. A pretrained T5 with 12-layer encoder and 12-layer decoder is used in the model. It is doubt that why the cross-instance captioning needs such a heavy decoder. How about a shallow decoder affect the performance? \n2. The caption results shown in the appendix demonstrate a weak advantage in generation tasks. Are there some reasons for the opposite performance compared with the retrieval tasks?\n3. The model places the pretrained architectures after the initialized Transformer pooling head, which will affect the pretrained weights. What is the training strategy and learning rate to deal with such a problem?\n4. What is the possible reason for the reconstruction bottleneck that both smaller and very large sizes degrading the performance? Besides, the performance is sensitive and high-impact with different sizes.\n5. There is a doubt that the MdR of T5-Small in Table 1 (c) is the same as that of T5-Base but R@1 and R@5 have a margin. \n6. What is the meaning of `None` in Table 2? Does it mean training without cross-instance captioning?\n7. This paper sets the $\\alpha$ to 0.2 in Eq. (1). How about other numbers affect the results, e.g., 0.1 and 0.3? What is the value of temperature T in Eq. (3)?\n8. Are there some zero-shot results for the pretrained model?\n9. How about the performance of the video representation on other tasks, e.g., action recognition?\n10. The support-set is the samples of the mini-batch. How to know which one semantically relates to the anchor? \n\n11. Missing references:\n1). Linchao Zhu and Yi Yang. 2020. Actbert: Learning global-local video-text representations. In CVPR.\n2). Huaishao Luo, Lei Ji, Botian Shi, Haoyang Huang, Nan Duan, Tianrui Li, Jason Li, Taroon Bharti, and Ming Zhou. 2020. UniVL: A Unified Video and Language Pre-Training Model for Multimodal Understanding and Generation. arXiv:2002.06353.\n3). Bruno Korbar, Fabio Petroni, Rohit Girdhar, and Lorenzo Torresani. 2020. Video understanding as machine translation. arXiv:2006.07203.", "extra_scores": {}, "extra_text": {}}, {"review_id": "HkwTab6zkzD", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 9, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary: \nThis paper focuses on better learning of video-text representations. To this end, the paper introduces a new generative task of cross-captioning which alleviates the typical issue of contrastive learning by learning to reconstruct a sample’s text representation as a weighted combination of a video support set. The proposed approach performs better than previous work on various datasets. \n\nPros: \n1) The paper is well written and easy to follow. \n\n2) The proposed method sounds novel and interesting. The empirical evaluations on various datasets suggest that the proposed method is better. \n\n3) The ablations on various modules of the proposed method is very interesting and thorough. \n\n\nCons:\n1) The current approach limits to using only the videos in the current batch for the support set. One could also try retrieving the support set from the full dataset in an online way. It would be interesting to see this ablation. \n\nOverall:\nThe proposed method of cross-captioning is novel and the thorough empirical evaluations/ablations further show the superiority of the proposed method as well as the usefulness of each component.  \n\nQuestions: \n1) Please provide statistical significance scores wherever necessary, e.g., Table-4 ours vs ours-pretrained difference is statistically significant? \n\n2) Is it possible to ablate on the choice of support set from within a batch vs. full dataset?\n\n\n", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper studies the problem of learning better video-text representation learning with an application to video-text retrieval. It proposes a key innovation: it uses a new generative task of cross-captioning that addresses issues with contrastive learning by learning to reconstruct a sample’s text representation as a weighted combination of a video support set, using a novel objective function using video-set bottlenecks. It uses pre-training based on YouTube video-ASR pairs, and shows empirical results where the proposed method outperforms multiple SOTA methods.

The authors have addressed the feedback of the reviewers, especially with the following improvements:

- Experiments were run on more datasets
- Relevant work pointed out by the reviewers were added
- Concerns regarding technical details were clarified

Decision: Accept (Spotlight)