Learning Video Representations using Contrastive Bidirectional Transformer
Authors: Chen Sun
Fabien Baradel
Kevin Murphy
Cordelia Schmid
Venue:  (2020)
Keywords: self-supervised learning
video representations
cross-modal learning

Abstract: This paper proposes a self-supervised learning approach for video features that results in significantly improved performance on downstream tasks (such as video classification, captioning and segmentation) compared to existing methods. Our method extends the BERT model for text sequences to the case of sequences of real-valued feature vectors, by replacing the softmax loss with noise contrastive estimation (NCE). We also show how to learn representations from sequences of visual features and sequences of words derived from ASR (automatic speech recognition), and show that such cross-modal training (when possible) helps even more.

Under review as a conference paper at ICLR 2020
LEARNING VIDEO REPRESENTATIONS USING CONTRASTIVE BIDIRECTIONAL TRANSFORMER
Anonymous authors Paper under double-bling review
ABSTRACT
This paper proposes self-supervised learning approach for video features that results in significantly improved performance on downstream tasks (such as video classification, captioning and segmentation) compared existing methods. Our method extends the BERT model for text sequences to the case of sequences of real-valued feature vectors, by replacing the softmax loss with noise contrastive estimation (NCE). We also show how to learn representations from sequences of visual features and sequences of words derived from ASR (automatic speech recognition), and show that such cross-modal training (when possible) helps even more.
INTRODUCTION
Recently there has been a lot of progress in self-supervised representation learning for textual sequences, followed by supervised fine-tuning (using small labeled datasets) of shallow (often linear) decoders on various downstream NLP tasks, such as sentiment classification. In this paper, we build on this work and propose new method for self-supervised representation learning for videos, optionally accompanied by speech transcripts generated by automatic speech recognition (ASR). We show that fine-tune tine-tuning linear decoders together with our self-supervised video representations, can achieve state of the art results on various supervised tasks, including video classification, segmentation and captioning.
Our approach builds on the popular BERT (Bidirectional Encoder Representations from Transformers) model (Devlin al., 2018) for text. This uses the Transformer architecture (Vaswani et al., 2017) to encode long sentences, and trains the model using the "masked language modeling" (MLM) training objective, in which the model must predict the missing words given their bidirectional context. The MLM loss requires that each token in the sequence be discrete. The VideoBERT model of (Sun et al., 2019a) therefore applied vector quantization (VQ) to video frames before passing them (along with optional ASR tokens) the BERT model. Unfortunately, VQ loses fine-gras imn-igrieed information that is often critical for downstream tasks. More recently, several papers (e.g., VilBERT (Lu et al. 2019) and LXMERT (Tan & Bansal 2019h) proposed to address this limitation by directly measuring the visual similarity between frames using pre-trained visual encoders.
In this paper, we propose way to train bidirectional transformer models on sequences of real- valued vectors (e.g., video frames), IT.T, using noise contrastive estimation (NCE), without needing pre-trained visual encoders. We call our method "Contastive Bidirectional Transformer" We also develop method that combines Z1:T with an optional sequence of discrete tokens, V1.2" (e.g., derived from ASR). In contrast to the VideoBERT paper (Sun et al., 2019a), we provide "lightweight" way of combining these signals after training each modality separately. In particular, we propose a cross-modal transformer to maximize the mutual information between #1.7 and y1.7> the sequence level (rather than at the frame level). This method is robust to small misalignments between the sequences (c.g., if the words at time t do not exactly correspond to what is visually present in frame t).
CBT.
We demonstrate the effectiveness of the proposed approach for learning short-term visual representa- tions, well as longer term temporal representations. For visual representations, window of K frames using a 3D convolutional neural network S3D (Xie et al. 2018), and then pass this sequence of features to the CBT model for self-supervised pretraining with the NCE loss on the
encode each

---

Under review as a conference paper at ICLR 2020
Pre-training
ASR
Let's go open up our camperddoro
Yue
BERT
embedding
ye
Leross
HowTo]U)
Transformer
Video
HowToluf Kinenks|U]
Lvisual
If arr
Leiass
50Salads(L) Breakfast(L)
Acteet)))
Downstream Tasks
Captioning
Leeg
YouCook2[1)
Temporal
Segmentation
Lseq
COINI))
3r Classifier
Lelass
UCF103/LI
HMDE51(L)
Summary of our method for training and evaluation. The blocks above the line are pre- trained in an self-supervised way. The solid blocks represent the BERT language model, which is pre-trained on web text and frozen (see section 3.1 The black CBT visual block is trained using
Figure I:
NCE loss on unlabeled HowTo or Kinetics videos (see section 3.2). The red cross- mosss- transformer
is trained using cross-modal loss on HowTo with ASR (see section 3.3 'The components below the
line are trained supervised way on various tasks. The purple block is trained for next action prediction on ActivityNet, Breakfast, and 50Salads (see section
4.2). The blue block is trained for
video classification on UCF101 and HMDB501 (see section 4.1).. The green blocks are trained captioning and video segmentation tasks, which are described in the supplementary material (section 6. and section 6.2). Lseq refers to cross-entropy sequence loss.
unlabeled Kinetics dataset (Kay et al.) 2017). We then fine-tune linear classifier for video classifi- cation on UCF101 and HMDB51. We show that our method outperforms previous state-of-the-ar self-supervised methods by large margins (UCF101 from 75.7% 79.5% and HMDB51 from 35.7% to 44.6%).
For temporal representations, encode each window of K frames using S3D network that is pretrained on Kinetics, and then "frozen". We then pass this sequence of features to the CBT model for self-supervised pretraining with the NCE loss on the unlabeled HowTo100M dataset (Miech) et al. 2019b). We also evaluate the effects of running ASR on NowTo1000M, and passing this to our moosl transformer as an additional signal. We then fine-tune various shallow decoders for variety of tasks, including video classification, segmentation and captioning. We show large gains compared to previous methods, especially when we use mross-modal pretraining.
See fig. for summary of our training method and evaluation protocol.
D
2 RELATED WORK
Video representations. Most existing work on learning video representations, such as (Simonyan
& Zisserman 2014 Varol et al.] 2018 Carreira & Zisserman, 2017 Xie et al., 2018 Tran et al.
2014), only captures few seconds of video. Long-term context can be encoded by recurrent neural networks (Abu Farha al. 2018; [Sun et al.] 2019b), graph convolutional networks (Zhang et al.
2019), or long-term feature banks Wu et al. 2019)), but these are all supervised methods. Some recent
work have been done on learning self-supervised video representation (Vondrick et al.)
2016
Wang
& Gupta 2015; Misra et al. 2016 Sermanet et al. 2018 Han et al. 2019 Xu et al.
2019 Wang
[ 2019a) by defining pretext tasks such as ordering (Lee al. 2017), rotation (Jing et al.
2018
temporal cycle consistency (Wang et al.) 2
Dwibedi et al. 2019) or colorization VVondric et al.
2018) but similar their supervised counterparts they capture only few seconds.
2

---

Under review as a conference paper at ICLR 2020
Self-supervised context modeling. Recently, there has between lot of work on self sulf-supe context modeling for language representations (Peters et al. 2018: Radford et al., 2019: Devlin et al. 2018). In particular, the BERT model, which stands for Bidirectional Encoder Representations from Transformers (Devlin et al.) 2018), pre-trains deep bidirectional representations by jointly conditioning on both left and right context in all layers. The pre-trained BERT representations be fine-tuned with just one additional output layer to create state-of-the-ar models for a wide range of NLP tasks, such question answering and linguistic entailment. Our representation builds on this approach and adapts it to continuous video data by using a contrastive loss.
Mutual information estimation and maximization. For representation learning, signal encoder can be trained to maximize the mutual information (MI) between the input signal and its encoded outputs, or the encoded outputs of the signal and its context (see e.g., (Belghazi et al. 2018 Hjelm et al.] 2019 Oord et al., 2018 Tian et al., (019). In particular, contrastive predictive coding (CPC) (Ordd et al. 2018) uses noise contrastive estimation (Gutmann & Hyvärinen, 2010) to maximize the MI lower bound. Unlike CPC, which relies on auto regressive models to encode context, we use BERT to encode bidirectional context within each sequence, and across different modalities.
Cross-modal learning. The multi-modality of video is a rich source of information for self- supervised learning of video representations. Since videos contain both visual and audio signals that are roughly synchronized, the two modalities can supervised each other, as explored in prior work such (Aytar al. 2016 Owens al. 2016b a Zhao al. 2018). Another common form of weak supervision is based on video and language, where language is either obtained by automatic speech recognition (ASR) or from additional textual description. Language can be leveraged by finding a joint embedding space for both visual and textual modalities or by learning alignment between the two modalities (Miech al. 2018 Alayrac al.,
2016:
Zhou et 2018a).
Recently, several concurrent approaches (Sun et al. 2019a Lu et al. 2019 Li et al. 2019a Su et al. 2019; Tan & Bansal 2019; Li et al., 2019b) generalize the BERT architecture and MLM objective to learn visual-linguittc representations. They assume the visual representations to be fixed and given by supervised pre-trained visual encoders, and define the visual MLM objective in terms visual similarities (e.g. via vector quantization measuring L2 distance) between the original and predicted visual representations. To the best of our knowledge, our proposed CBT the first to demonstrate the effectiveness of BERT-style pre-training in fully self-supervised way for video.
3 METHOD
We first give an overview of the BERT model for learning from sequences of words, V1.r". We then discuss an extension to the case of sequences of video frames, F1.r. Finally, we discuss how to learn from both kinds of data, even when not perfectly aligned.
3.1 THE BERT MODEL
The BERT model (Devlin et al. 2018) takes in a sequence of discrete tokens, y1.7*, where ye e {I...., K}. embeds each one into a dense vector, ei eD, and then emits sequence of dense output vectors, hy E D", which are computed using a transformer (Vaswani et al., (017]) The output sequence captures local and global semantic information about the input sequence.
The main training objective for BERT is to minimize the pseudo negative log likelihood, defined by
Loan - EynD logp(ull--)
where y-s is the sequence of all words except the t'th, and
p( = exp(e éx)
(2)
Σ (k)
Here fenc(k) is an embedding lookup table for token k. e, = fene(y)) is the embedding for the token 1, e [yen < I, 1,0, fene (vr) :1 1] is the embedding sequence for the context, and
3

---

Under review as a conference paper at ICLR 2020
é, Ocontexx (e-r) is multi- multi-headed transformer network that takes a T X D feature matrix as input (masked at location t) and returns a matrix of the same size.
3.2 THE CBT MODEL
The BERT model requires fixed discrete vocabulary. However, for images and videos, the inputs are real-valued vectors. We propose to use the softmax version of the noise contrastive estimation (NCE) loss (Jozefowiez et al. 2016), which has the form
Lvisual -xxDD
Σ log ()
(3)
where
exp(e éx)
NCE(X, xx+)
exp(e 0₁) + Σ) EEegett exple
where e, Seno (x,) is the output of 3D CNN applied small window around frame f (we the S3D model from (Xie et al., 2018), è; (oomer((e_)) is the output of a visual transformer, and neg(t) set of (indices of) "negative examples" (in practice we use all the other frames from the same minibatch as frame t).
Intuitively the NCE loss encourages the model to learn to identify the correct frame (given the context) compared to set of negative distractors. More formally, it can be shown that the NCE loss maximizes (a lower bound on) the mutual information (MI) between x, and x- (see e-g., (Oord et al., 2018 Poolle et al. 2019)). This loss has been used in other work on self-supervised visual representation learning, e.g., in the deep infomax (DIM) (Hjelm et al., 2019) and contrastive predictive coding (CPC) (Oord et al.) 2018) papers. In DIM, the context predictor uses CNN applied to neighboring patches in the same image, and in CPC, the context predictor uses a causal autoregressive model applied to "earlier" patches in the same image. In our CBT method, the context predictor is bidirectional transformer applied to video frames.
3.3 THE CROSS- MODAL CBT MODEL
In this section we show how to learn useful representations from sequences of continuous visual features (from video) and sequences of discrete words (from ASR). More precisely, assume we have two sequences, x = X1:7 representing video, and y = yI.T", representing ASR. Note that the sequences may not be perfectly aligned, since person may speak about things time that not visible in the frame at time f. Therefore it does not make sense to try to maximize the MI between X and yt at the frame level. Instead, we try to maximize MI between and y at the sequence level.
To do this, we first encode each sequence using CBT and BERT to get h;rr = CBT(x1:7) and h. BERT ). shown in fig. 1 We then concatenate these sequences and pass them to a shallow cross-modal transformer to produce 22 h Finally, we pass this to a shallow MLP to compute an MI-like score MI(x, y) f fh"""+)) (Here f() extracts the features from h zy but it could also use average pooling.) This model is trained using Leross = - E log NCE x). where
Ml(x,y)
NCE(y MI(x, y) + Ey'eNo(ty) MI(x, y")
(5)
where Neg(y) is set of ASR sequences associated with video
Note that our cross-modal training assumes there is something in common between the video and text streams. In practice this means we have focus certain kinds of videos, such instructional videos, in which the spoken words and the visual content are "about" the same thing. By contrast, arbitrary videos may contain speech content that is unrelated the visual frames (e.g., imagine conversation between two characters in drama or soap opera).
3.4
OVERALL MODEL
Our overall model has three components: one transformer (BERT) that takes discrete ASR tokens, transformer that takes continuous video features, and third transformer to estimate mutual
4

---

Under review as a conference paper at ICLR 2020
information between the two modalities. We jointly train the model by optimizing:
Lebt
Wben + Wvisual visual
Weross Ceross
(6)
We fix Wbent = 0, since we use a pre-trained (frozen) BERT model for ASR. We set Wisuual = 1, and either set Weross 1 or Wcross 0, depending whether we use cross-modal training not.
4 EXPERIMENTS
In this section conduct experiments study the usefulness of the representations learned by CBT model for various downstream tasks, including action anticipation, video captioning and action segmentation. We also consider ablations model, such turning cross-modal training off, varying the size of the visual transformer, and varying the amount of unlabeled pre-training data.
4.1 LEARNING SELF-SUPERVISED VISUAL REPRESENTATION
In this section we evaluate self-supervised visual representation learning on the downstream task of action recognition. Existing methods use various proxy tasks to pre-train feature extractors in self-supervised way, and then use supervised learning train linear classifiers on top of these frozen representations, fine-tune the classifier plus feature extractor end-to-end. We follow the same protocol.
Experimental setup. We follow the standard practice from recent works (Han al. 2019: Jing ct al. 2018; Wang et al. (2019)) by pre- trainig our model (S3D feature extractor followed by CBT) unlabeled RGB- only Kinetics (Kay et al. 2017) videos. Kinetics is the largest action recognition dataset containing 500k short clip videos (about 10 seconds long) for 600 human actions classes. We take minute sliding windows from the original You'Tube videos they selected from. We then use the (average pooled) S3D features as input to linear classifier, and train the classifier on various datasets. For evaluation, use UCF101 (Soomro al. (2012), which contains 13,320 videos from 101 human actions, and HMDB51 (Kuehne et al., 22011), which contains 7,000 videos from 51 classes. For both dataset we report the action recognition test accuracy averaged over the 3 standard train/test splits.
To pre-train our CBT model, use curriculum learning strategy, by first pre-training the S3D feature extractor on unsupervised clips using the loss proposed in the 3DRotNet paper (Jing et al. 2018) on 16 consecutive frames. We then jointly fine-tune the last blocks of S3D (Mi xed5' and Mixed5c) with the visual transformer using the CBT visual loss. We observed that this strategy gave us better results on downstream tasks compared to pre-training from scratch using CBT; also saves memory and computation, which allows us use longer sequences.
During pre-training we set the number of visual transformer layers to be 2, number of attention heads to be 4, and hidden unit size to be 768. We randomly take 60-second sliding windows from the Kinetics videos, and break them into sequences of 1.5-second clips. We randomly mask out out of the 40 possible locations. We resize the video frames 112 by 112 before encoding them with S3D to save memory. The model is trained for 500K iterations with batch size of 128 and learning rate of le-5.
Method
Dataset
UCFF10
IMMBS51
Fine-tuned
Frozen
Method
UCF101 HMDB51
UCF101
HMDB51
Shuffle& Misra et
2016
UCF101
18.1
OPN [ [Tee 2017
DCF101
59.6
23.8
Random
63.3
29.7
ClipOrder (Xu 2019
UCF101
30.9
Shuffc&Leaan"
68.7
35.8
26.5
12.6
Wang el 20119
Kinetics
61.2
33.4
3DRotNet" 75.3
40.0
47.7
24.8
3DKoINer
Kinetics
CBT
79.5
44.5
29.5
DPC
(Hhi of al 7919
Kinetics
75.7
35.7
Kinetics
Table 1: Self-supervised action recognition accuracy. (Left) We show the effect of different pre- training strategies on our model. are our reinplementations. (Right) Comparison to the state of the art UCF101 and HMDB51. (We report the performances from the original papers.)
Comparison of pre-training strategies. In Table Left) we compare our way pre-training the S3D model (i.e., using CBT visual loss) to existing approaches. In particular, consider the
5

---

Under review as a conference paper at ICLR 2020
Shuffle&Laam (Misra al. 2016) and 3DRotNet (Jing et al., 2018) proxy tasks. We reimplement the two methods using S3D CNN, and pre-train them on the same Kinetics data. We also consider random initialization. We report classification results UCF101 and HMDB51 using frozen features and fine-tuned features passed to a linear classifier. We see that our method outperforms existing training methods by large margin.
Comparison to existing methods. T compares the results of our method to various state-of-the art self-supervised methods. (We only report the results of fine-tuning, which are better for all methods than using frozen features.) Note that the methods differ both in architecture and training objective. First we compare against 2DCNN approaches Shuffle&eaan (Misra al., 2016) and OPN (Lee et al.] (017). Our method outperforms both by a very large margin. This can be explained by the fact that our backbone is a 3DCNN architecture, which much more powerful than 21 CNNs for video action recognition. Next we compare against approaches using 3DCNN architectures similar to our S3D. We also outperform all of these methods by a very large margin, and even beat the most recent approach, DPC (Han et al. 2019), by 3.8 points UCF101 and 8.9 points on HMDB51. We believe this is due to the better contextual features that we are able to learn by using the transformer model and NCE loss.
4.2 LEARNING SELF-SUPERVISED TEMPORAL REPRESENTATIONS
In this section, we consider self-supervised training of representations from long videos, followed by supervised fine-tuning on various downstream tasks. To avoid running out of memory, pre-train the S3D model on the task of classifying (short) Kinetics videos. We then freeze this feature extractor, and focus learning longer-term temporal representations using the self-supervised CBT model. That is, we precompute short term representations ef = (X,) for all videos using S3D, and focus on learning global representations hf (B.(e).r).
For the self-supervised pre-training, we use unlabeled videos from the HowTo 100M dataset (Miech ctt al. (2019b). This contains 1M instructional videos (details below), so the speech informative about the vision. Therefore, we also run ASR on this dataset and use cross-modal training to compute h = Trosssraasfrrmr(hh.... h² where h BERT(Y):T)) is the output of pretrained BERT model. To evaluate the performance of the pre-trained features (both visual and cross- cross-modal), we consider three tasks: "action anticipation" (i.e, predicting the next action label to occur given some temporal prefix); video captioning (see section 6.1 in supplementary), and video segmentation (see section 6.2 in supplementary).
Details self-supervised pre-training. We pre-train model on HowTo100M, (Miech al., 2019b). This a new large-scale dataset 1.22M narrated instructional videos available on YouTube and covers among 23k different visual tasks. The average duration of a video is 6.5 minutes and there are on average 110 clips per video.
To extract visual features, resize the videos to be 224 by 224, and compute visual features over sliding windows of 1.5 seconds (30 frames at 20 FPS) using an S3D network (Xie et al., 2018) pre-trained on the Kinetics dataset (Kay al.. 2017). We take the feature embeddings from the final Mixed5c block of the S3D network before the classification layer, and average pool the features spatio-temporally to get vectors of size 1024. We follow the same strategy for extracting visual features on the downstream tasks. The visual features are not fine-tuned during pre-training or when applied to downstream tasks.
To extract text features, we convert the audio track into sentences by calling the YouTube ASR API, followed by an off- the-shelf LSTM-based language model to add punctuation, thus converting the stream of words into a stream of sentences. We then follow the standard preprocessing steps from BERT (Devlin et al., 2018), and use WordPieces tokenization with the same vocabulary of 30,000 tokens. To encode ASR tokens, we take the pre-trained BERT-base architecture, which has 12 layers of Transformers, each of which has 768 hidden units and 12 attention heads.
To construct paired inputs to pre-train CBT with the cross-modal objective, we iterate over all the sentence segments in the NowTo100M dataset, and concatenate short segments until they reach the maximal length of 48 tokens. We then retrieve up to 48 visual features (72 seconds) starting at the same locations in videos. We mask out out of 48 features randomly. For both the video and cross-modal transformers, set the total hidden units per layer to 768. We fix the number of layers

---

Under review as a conference paper at ICLR 2020
to for the cross-modal transformer and explore the optimal number of layers and attention heads for the video transformer. Their weights are randomly initialized.
For pre-training the CBT model on 128. We use the Adam optimizer with an initial learning rate of 1e-5 and a linear decay learning rate schedule. The model is trained for 2 million iterations, which takes around 2 days.
use 32 Cloud TPUs and total batch size of
Details on supervised evaluation. We evaluate the pre-trained temporal representations by transfer learning to downstream tasks. We first focus action anticipation, whose goal is to predict the future actions by observing video sequences preceding them. In the supplementary, we also present results on video captioning and action segmentation.
For the action anticipation task, we follow the standard setup described from recent work (Abu Farha al. 2018; Miech et al. 2019a). We consider three datasets: the Breakfast dataset (Kuehne et al. 2014) is composed of 1712 cooking videos and contains 48 fine-grained action classes; the 50Salads dataset (Stein& & McKennal 2013) contains 50 cooking videos with 17 fine- fine-grained action classes; and the ActivityNet 200 dataset (Heilbron et al., 2015) contains 19994 YouTube videos with 200 human action classes (beyond the cooking and instructional domains). The inputs are video segments up to Te seconds before the actual action starts, and the outputs are categorical labels. For comparison with previous approaches, we set T for Breakfast and 50Salads, and T. = for ActivityNet.
For all experiments except for ablation on sequence lengths, we fix the input sequence length to be 72 seconds (corresponding to 48 sliding windows), features for videos shorter than 72 seconds are zero-padded. The outputs of CBT are transformed features with the same length, we take the output feature at the last non- non-pade padded position to represent the whole sequence, and put a linear classifier on top to predict the future actions. We jointly fine-tune the weights of visual transformers with the linear classifier. The text transformers and cross-modal transformers are not used during fine-tuning since only visual inputs available. We train our model for 5 epochs using a batch size of 32 with the Adam optimizer and an initial learning rate of 1c-3, We report the top-1 accuracy on the test sets for Breakfast and 50Salads, and on the validation set for ActivityNet.
Comparison to existing methods. Table 2 (Left) compares to existing methods. First we compare to two existing self- superviedd approaches, namely Vondrick et al. (0166) and [unn et al. (2119)). Our approach outperforms both by a very large margin. The difference with VideoBERT (Sun et al. (2019a), which also relies on BERT model, can be explained by the fact that it quantizes the visual features into tokens and, hence loses discriminative power. Next we compare to some recent methods that train deep classifiers end-to-end, namely (Abu Farha et al., 2018) and (Miech et al., 2019a). We outperform both by a large margin.
Method
Self-super
Bifst
Sallads ActNet
Window (in sec.)
Blisst AvgPool
Salads AvgPool ISTM CBT
ActNet AvgPool LSTM
LST CBT
CRT
Nondrick all 2016
8.1
6.2
Siyr Parta et al 2015
N
30.1
30.1
20.3
30.2
20193
9.1
5.5
23.5
24.2
26.3
36.8
29.7
Sueet 2019
30
24.2
25.6
29.8
34.3
33.9
37.8
52.9
57.1
57.5
45
22.7
26.2
30.8
32.2
35.3
38.5
50.7
57.3
58.8
CBT
32.7
39.1
59.8
21.9
26.7
35.6
59.8
Table 2: Action anticipation accuracy. (Left) Comparison to the state of the art on Breakfast, 50 Saldads, ActivityNet. Self-super = means the model was pre-trained in self-supervised way, and then fine-tuned using a linear classifier. Self-super N means the model is trained end-to-end on the specific task. (Right) Comparison with the average pooling and LSTM basclines on 50Salads Breakfast, 50Salads and ActivityNet. We vary the observation window lengths (sec.)
Effect of video length. In Table (Right) we show the impact of the length of the training videos on the performance. We compare with two baselines, average pooling (AvgPool) and LSTM (Hochreiter Schmidhuber 1997). The AvgPool baseline simply computes the average of all input visual features over time. The LSTM baseline takes the same sequence of S3D features but recurrently updates its hidden states over time. The final hidden state is used for classification. We adjust the hidden unit size of LSTM to make its number of parameters comparable to CBT. We can see that CBT significantly outperforms the two baselines on all three datasets. Moreover, we can see that as the observed video length increases, the performance of CBT monotonically increases, while LSTM and AvgPool either platcaus or decreases. These results indicate that CBT is better at modeling long-term temporal context.
7

---

Under review as a conference paper at ICLR 2020
Data size (%)
Cross modal
Bkfst
Salads
ActNet
A
Bkfst
Salads
ActNet
A
Bkfst
Salads ActNet
28.8
35.1
57.4
10
29.9
37.3
58.3
29.6
34,89
59.3
1
30.9
58.2
25
30.2
37.3
58.4
4
32.7
39.1
59.8
2
2
31.8
36.3
58.3
50
30.0
37.4
58.5
32.7
38,9
59.2
2
4
32.7
38.9
59,4
75
31.4
38.5
59.3
23.2
58.0
8
30.9
39,9
100
24.9
37.6
57.6
16
9.17
5.8
57.5
16
31.8
39.8
57.8
100
32.7
39.1
59.8
Table 3: Ablation study on the action anticipation task. We show accuracy on Breakfast, 50Salads and ActivityNet. (Left) Impact of the percentage HowTol00M videos used, and the cross-modal objective during pre-training. 0% corresponds to pretraining, ie. using random weights. (Middle, Right) Impact the number layers (I.) and attention heads (A) for the visual transformers.
Effect of dataset size and cross-modal training. In Table (Left), we study the impact of pre- training data set. As expected, pre-training with more examples leads to higher performance on all three benchmarks. We also study the impact of cross-modal training. We see this helps signficantly, especially on the smaller datasets (Breakfast and Salads).
Effect of model size. Table (Middle) and (Right), we study the impact of the number of layers (L) and the number of attention heads (A) for the visual transformer. Not surprisingly. model performance initially increases, but surprisingly, it then starts to decrease, in contrast the case of NLP-BERT. We conjecture that this is because our unlabeled pre-training set is much smaller than used by the NLP-BERT model. Fortunately, our technique quite scalable, since we can train the video representations on top of S3D features using relatively shallow transformers transformer only has 15M parameters, whereas the BERT NLP transformer has 110M parameters.
visual
Applications to other tasks. In Table 4 we show the results of using of our learned temporal representation for video captioning and action segmentation. See section 6.1 and section 6.2 in the supplementary for details.
Method
BLEU-4
METEOR
ROUGE-L
CIDEr
Method
Frame Acc.
et 20188
4.38
11.55
Tang al. 2019
S3D
3,24
9.52
26.09
0.31
Krbhard
2018
21.2
VideoBERT
4.33
11.94
28.80
0.55
Ding & Xu
2018
34.3
CBT
5.12 (+: 0.02)
12.97 (:: 0.05)
30.44 (t: 0.08)
0.64 (:: 0.00)
CBT
53.9
Table (Left) Video captioning results on the YouCook2 dataset (Zhou al. 2018b) We compare with previous cate-offtthe-an methods by Zhou et al. (2018c) and [unn et al. (2019a). the caption decoder of all methods share the same architecture, the main difference comes from the visual encoder. (Right) Action segmentation results on the COIN dataset (Tang et al. 2019). A linear classifier is applied on the sequence of CBT output features for dense frame labeling. We compare with previous state-of-ther -frth-rrr--- methods using the standard frame accuracy metric.
CONCLUSION
We have shown how extend the BERT model learn representations from video in a self-supervised way, without needing vector quantization or pre-trained visual features. We have also shown how extend this to the cross-modat setting, when ASR available. Finally. we demonstrated that our method learns features that are far more useful than existing self-supervised methods for a variety of downstream video tasks, such as classification, captioning and segmentation. We believe that the simplicity and modularity of our method will let us scale to much larger unlabeled video datasets, which we hope will let us finally surpass supervised video pretraining (c.g., on Kinetics), just as other methods (e.g., CPC++ (Hénaff al. 2019)) have recently surpassed supervised image pretraining (on ImageNet).
8

---

Under review as a conference paper at ICLR 2020
REFERENCES
Yazan Abu Farha, Alexander Richard, and Juergen Gall. When will you do what? anticipating temporal occurrences activities. In CVPR, 2018.
Jean-Baptiste Alayrac, Piotr Bojanowski, Nishant Agrawal, Josef Sivic, Ivan Laptey, and Simon Lacoste-Julien. Unsupervised learning from narrated instruction videos. In CVPR, 2016.
Yusuf Aytar, Carl Vondrick, and Antonio Torralba. Soundnet: Learning sound representations from unlabeled video. In NeurlPS, 2016.
Mohamed Ishmael Belghazi, Aristide Baratin, Sai Rajeswar, Sherjil Ozair, Yoshua Bengio, Aaron Courville, and Devon Hjelm. Mine: mutual information neural estimation. arXiv preprint arXiv:1801.04002 2018.
J. Carreira and A. Zisserman. Quo vadis, action recognition? A new model and the Kinetics dataset. In CVPR, 2017.
Jacob Devlin, Ming- Wei Chang, Kenton Lee, and Kristina Toutanova, BERT; Pre-training of deep bidirectional transformers for language understanding, arXiv preprint arXiv:1810.04800 04805, 2018.
Li Ding and Chenliang Xu. Weakly-supervised action segmentation with iterative soft boundary assignment. In CVPR, 2018.
Debidatta Dwibedi, Yusuf Aytar, Jonathan Tompson, Pierre Sermanet, and Andrew Zisserman. Temporal cycle-consistency learning. CVPR, 2019.
Michael Gutmann and Aapo Hyvärinen. Noise-contrastive estimation: estimation principle for unnormal- ized statistical models. In AISTATS, 2010.
Tengda Han, Weidi Xie, and Andrew Zisserman. Video representation learning by dense predictive coding.
arXiv preprint www.v:vv99.000m 000604466. 2019.
Fabian Caba Heilbron, Victor Escorcia, Beraadd Ghanem, and Juan Carlos Niebles. Activitynet: A large-scale video benchmark for human activity understanding. In CVPR, 2015.
Olivier Hénaff, Ali Razavi, Carl Doersch, SM Eslami, and Aaron den Oord. Data-efficient image recognition with contrastive predictive coding, arXiv preprint wwXivvI905.092111 2019.
R Devon Hjelm, Alex Fedorov, Samuel Lavoie-Marchilee Marchildon, Karan Grewal, Adam Trischler, and Yoshua Bengio. Learning deep representations by mutual information estimation and maximization. ICLR, 2019.
Sepp Hochreiter and Jürgen Schmidhuber. Long short-term memory. Neural computation, 1997.
Longlong Jing, Xiaodong Yang. Jingen Liu, and Yingli Tian. sule-super spatiotemporal feature learning via video rotation prediction, 2018
Rafal Jozefowicz, Oriol Vinyals, Mike Schuster, Noam Shazeer, and Yonghui Wu. Exploring the limits of language modeling, arXiv preprint 1r0iv.1602.024/1 2016.
Will Kay. Joao Carreira. Karen Simonyan, Brian Zhang, Chloe Hillier, Sudheendra Vijayanarasimhan, Fabio Viola, Tim Green, Trevor Back, Paul Natsev, al. The kinetics human action video dataset. arXiv preprint (7050069050655/ 2017.
H. Kuehne, H. Jhuang, E. Garrote, T. Poggio, and T. Serre. HMDB: A large video database for human motion recognition. In ICCV, 2011.
Hilde Kuchne, Ali Arslan, and Thomas Serre. The language of actions: Recovering the syntax and semantics of goal-directed human activities. CVPR, 2014.
Hsin-Ying Lee, Jia-Bin Huang, Maneesh Kumar Singh, and Ming-Hsuan Yang. Unsupervised representation learning by sorting sequenees. In ICCV, 2017.
Gen Li, Nan Duan, Yuejian Fang, Daxin Jiang, and Ming Zhou. Unicoder-vl: A universal encoder for vision and language by cross-modal pre-training. arXiv preprint arXiv:1908.06066, 2019a.
Liunian Harold Li, Mark Yatskar, Da Yin, Cho-Jui Hsich, and Kai- Wei Chang. Visualbert: A simple and performant baseline for vision and language. arXiv preprint arXiv:1908.00 03557, 2019b.
Jiasen Lu, Dhruv Batra, Devi Parikh, and Stefan Lee. Vilbert: Pretraining task-agnostic visiolinguistic represen- tations for rision-and-lane lan-aauuee tasks. In NeurlPS, 2019.

---

Under review as a conference paper at ICLR 2020
Antoine Miech, Ivan Laptev, and Josef Sivic. Learning text-video embedding from incomplete and heteroge- neous data, arXiv preprint arXiv:/804.025// 2018.
Antoine Miech, Ivan Laptev, Josef Sivic, Heng Wang. Lorenzo Torresani, and Du Tran. Leveraging the present to anticipate future videos. In CVPRW, June 2019a.
Antoine Miech, Dimitri Zhukov, Jean-Baptiste Alayrac, Makarand Tapaswi, Ivan Laptev, and Josef Sivie. HowTo100M: Learning a Text- Video Embedding by Watching Hundred Million Narrated Video Clips. In ICCV, 2019b.
Ishan Misra, C. Lawrence Zitmick, and Martial Hebert. Shuffle and learn: Unsupervised learning using temporal order verification. In ECCV, 2016.
Aaron van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predictive coding. In NewrlPS, 2018
Andrew Owens, Phillip Isola, Josh McDermott, Antonio Torralba, Edward I Adelson, and William Freeman. Visually indicated sounds. In CVPR, 2016a.
Andrew Owens, Jiajun Wu, Josh H McDermott, William T Freeman, and Antonio Torralba. Ambient sound provides supervision for visual learning. In ECCV, 2016b.
Matthew E Peters, Mark Neumann, Mohit lyyer, Matt Gardner. Christopher Clark, Kenton Lee, and Luke Zettlemoyer. Deep contextualized word representations. In NAACL, 2018.
Ben Poole, Sherjil Ozair, Aaron van den Oord, Alexander A Alemi, and George Tucker. On variational bounds of mutual information. In JCML, 2019.
Alee Radford, Jeff Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. Language models unsupervised multitask learners. 2019.
Alexander Richard, Hilde Kuchne, Ahsan Igbal, and Juergen Gall. Neuralnetwork-vieerbb:: framework for wcakly supervised video lcarning. CVPR, 2018.
Pierre Sermanet, Corey Lynch, Yevgen Chebotar, Jasmine Hsu, Eric Jang, Stefan Schaal, Sergey Levine, and Google Brain. time-contrastive networks: Self-supervised learning from video. ICRA, 2018.
Karen Simonyan and Andrew Zisserman. Two-stream convolutional networks for action recognition in videos. In NewrlPS, 2014.
Soomro, A. Zamir, and M. Shah. UCF101: A dataset of 101 human actions classes from videos in wild. Technical Report CRCV-TR-12-01 2012.
Sebastian Stein and Stephen McKenna. Combining embedded accelerometers with computer vision for recognizing food preparation activities. In UbiComp, 2013.
Weijie Su, Xizhou Zhu, Yue Cao, Bin Li, Lewei Lu, Furu Wei, Jifeng Dai. VI-hert: Pre-training of generic visual-linguistic representations. arXiv preprint 1908.0890000830 2019.
Chen Sun, Austin Myers, Carl Vondrick, Kevin Murphy, and Cordelia Schmid. VidcoBERT: A joint model for video and language representation learning. In ICCV, 2019a.
Chen Sun, Abhinav Shrivastava, Carl Vondrick, Rahul Sukthankar, Kevin Murphy, and Cordelia Schmid. Relational action forecasting. In CVPR, 2019b.
Hao Tan and Mohit Bansal. Lxmert: Learning cross-modality encoder representations from transformers. arXiv preprint 190.:::900..044000 2019.
Yansong Tang, Dajun Ding, Yongming Rao, Yu Zheng, Danyang Zhang, Lili Zhao, Jiwen Lu, and Jie Zhou. COIN: A large-scale dataset for comprchensive instructional video analysis. In CVPR, 2019.
Yonglong Tian, Dilip Krishnan, and Phillip Isola. arXiv:1906.05844 05849, 2019.
Contrastive multiview coding.
arXiv preprint
Du Tran, Lubomir D Bourdev, Rob Fergus. Lorenzo Torresani, and Manohar Paluri. C3d: generic features for video analysis. arXiv preprint arXiv:1412.0767, 2014.
Güi Varol, Ivan Laptev, and Cordelia Schmid. Long-term temporal convolutions for action recognition. TPAMI, 2018.
10

---

Under review as a conference paper at ICLR 2020
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention you need. In NeurlPS, 2017.
Carl Vondrick. Hamed Pirsiavash, and Antonio Torralb. Anticipating visual representations from unlabeled video. In CVPR, 2016.
Carl Martin Vondrick, Abhinav Shrivastava, Alireza Fathi, Sergio Guadarrama, and Kevin Murphy. Tracking emerges by colorizing videos. ECCV, 2018.
Jiangliu Wang. Jianbo Jiao, Linchao Bao, Shengfeng He, Yunhui Liu, and Wei Liu. Self-supervised spatio- temporal representation learning videos by predicting motion and appearance statistics. CVPR, 2019a.
Xiaolong Wang and Abhinav Gupta. Unsupervised learning of visual representations using videos. In ICCV, 2015.
Xiaolong Wang. Allan Jabri, and Alexei Efros. Learning correspondence from the cycle-consistency of time. CVPR, 2019b.
Chao-Yuan Wu, Christoph Feichtenhofer, Haoqi Fan, Kaiming He. Philipp Krähenbühhl, and Ross B. Girshick. Long-term feature banks for detailed video understanding. CVPR, 2019.
Saining Xie, Chen Sun, Jonathan Huang, Zhuowen Tu, and Kevin Murphy. Rethinking spatiotemporal feature learning video understanding. ECCV, 2018.
Dejing Xu, Jun Xiao, Zhou Zhao, Jian Shao, Di Xie, and Yueting Zhuang. Self-supervised spatiotemporal learning via video clip order prediction. In CVPR, 2019.
Yubo Zhang, Pavel Tokmakov, Cordelia Schmid, and Martial Hebert. A structured model for action detection. In CVPR, 2019.
Hang Zhao, Chuang Gan, Andrew Rouditchenko, Carl Vondrick, Josh McDermott, and Antonio Torralba. The sound pixels. In ECCV, 2018.
Luowei Zhou, Nathan Louis, and Jason Corso. Weakly-supervised video object grounding from text by loss weighting and ohject interaction. In BMVC, 2018a.
Luowei Zhou, Chenliang Xu, and Jason J Corso. Towards automatic learning of procedures from web instruc- tional videos. AAAI, 2018h.
Luowei Zhou, Yingbo Zhou, Jason J. Corso, Richard Socher, and Caiming Xiong. End-to-end dense vidco captioning with masked transformer. CVPR, 2018c.

---

Under review as a conference paper at ICLR 2020
6 SUPPLEMENTARY MATERIALS
6.1 VIDEO CAPTIONING
In this section, we apply our model to video captioning.
Dataset. We pretrain our model on How TOO0MM, and then use its features as input to a captioning model (details below) which is trained on the YouCook2 dataset (Zhou al. 2018b). This contains 2000 Youtube videos of an average length of 5.26 minutes for total of 176 hours. The annotations consist of segmentation boundaries and captions with on average 7.7 segments per video and 8.8 words per caption. We made sure that there is no overlap between the videos from our pre-training datasets and YouCook2.
Model. We follow the experimental setup from (Zouu et al. 2018c), where the ground truth video segmentations from YouCook2 are used to train a supervised model mapping video segments to captions. Our captioning model is a transformer with 2 layers and hidden layer of size 128. During training we set the dropout probability to 0.4. We train our model for 10K iterations using batch size of 128 with the Adam optimizer and an initial learning rate of le-4. We report BLEU, METEOR and ROUGE metrics on the validation set.
Comparison to other methods. computed using average-pooled S3D features. We also outperform the approach of [hoou et al. (2018c) and VideoBERT (Sun et al. (2019a) all reported metrics. The comparison VideoBERT is particularly interesting. The gains suggest that removing the quantization of video features is important for obtaining fine-gra geniigr video representation. We also observe that the difference between CBT and VideoBERT is smaller for YouCook2 than for Breakfast and 50Salads action anticipation task, possibly because the YouCook2 dataset is more similar to the cooking videos used for pre-training by VideoBERT.
Table 4 shows results. We outperform simple baseline
6.2 ACTION SEGMENTATION
In this section, we apply our model to the task of temporal action segmentation.
Dataset. We pretrain our model on NowTo100M and then use its features input to lincar classifier (details below) which is trained on the COIN dataset Tang et al.) (2019). This contains 11827 instructional Youtube videos of average length of 2.36 minutes. The annotations consist of segment boundaries and class label. On average there are 3.91 segments per video each of which lasts 14.9 seconds. There are in total 779 classes.
Model. We extract video features using S3D and feed the sequence to the visual transformer. We use fixed size of 72 seconds and use zero-padding for shorter sequences. The overall clip is represented by its associated output embedding of size 768. This preprocessing step is frozen. We feed the features linear classifier, which we train model for 100K iterations using batch size of 32 with the Adam optmizer and initial learning rate of le-3. At test time we operate on a long video with sliding window of 72 seconds.
Comparison to existing approaches. In Table 4 we compare CBT against various state of the art approaches using the frame acuracy as metric, including (Tang al. (001)), (Richard et al. 2018) and (Ding & Xu 2018). We outperform them by large margin state (+19.6 points).
12


Reviews:
[{"review_id": "ryg9-YehtB", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper presents a novel method to extract cross-modal text-visual embeddings on the HowTo 100M corpus. The core idea is to extend previous work on clip-level embeddings (e.g. the max-margin ranking loss proposed for HowTo 100M) to a transformer architecture which takes into account the entire context of a video, which should lead to better learned representations and improved performance in downstream tasks. In addition, the max-margin loss is replaced by noise contrastive estimation. \n\nThe paper is well written and explains the main problem well, however I do have a few questions:\n- I do not understand the sentence \"However, for images and videos, the inputs are real-valued vectors.\" (Section 3.2) - Transformers are being used for speech recognition or speech translation - the input features are not the problem. The outputs are assumed to be discrete (in the original formulation)\n- Why not directly compare your approach to the approach presented in (Miech, 2019c) - it would be interesting to see a direct comparison, but as far as I can tell, there is no overlap in tasks?\n- What is the influence of adding punctuation to the ASR output, how good is it, and how good is the underlying ASR? Why did you not use the original text annotations provided by HowTo 100M, but run the audio through Google ASR (again?) It would be good to know how good the ASR is, and if adding in punctuation post-hoc works well, and how this influences your use with a pre-trained BERT model. My guess is that the BERT model will be happy as long as it sees a \".\" at the end?\n- Also, would it be possible to compare the results of your work with some of the work in (Miech, 2019c) - it almost seems that your work avoids comparing your results to this previous work.\n", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper."}, "extra_text": {}}, {"review_id": "Syl7oex0cB", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper is about a self-supervised video representation with a multi-modal learning process that the authors then use for performance on a variety of tasks. The main contribution of the paper is a successful effort to incorporate BERT-like models into vision tasks. As is detailed in the related work, the field has been inching towards this but without as much success as this paper has.\n\nMy main criticism of the paper is that it feels like there is everything and a bag of chips happening; It's exceptionally hard to tease apart what is the main contribution to its success. I mostly came away from the paper thinking that it was good to see an existence proof of successfully incorporating the result, but not having really understood anything more wrt why or how this works. Other than it being a good idea to have a bigger model and more varied types of gradients, it's unclear what this model does that distinguishes it from other approaches.\n\nOn a more specific critique level, why use COIN? And why compare on a frame accuracy metric? The comparison to Ding & Xu seems a bit odd given that they don't assume access to annotations but rather to video transcripts. There are other datasets that you could make use of here that are more applicable, like Thumos14 or ActivityNet. I understand that this is a small section, but arguably the paper would be stronger if more time was spent on the main result than on this sidebar.\n\nOverall, I'm giving it a weak accept because I do think that the community should be aware of this paper's result.", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_checking_correctness_of_derivations_and_theory": "N/A", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper."}, "extra_text": {}}, {"review_id": "HJeDipXRtr", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This is one of those papers where the number of experiments conducted to produce the results is beyond the capabilities of \"almost all\" research groups. From the paper: \"we use 32 Cloud TPUs. The model is trained for 2 million iterations, which takes around 2 days.\" However, with that being said, it's a good paper of general interest to the community.\n\nThe paper focuses on self-supervised learning in video, and combines two contributions. The first is using a noise contrastive estimation loss (2016) which can be used for any visual dataset. The second is a cross-modal (BERT) model that requires language and vision. A few modifications over other BERT flavours are introduced. The cross-modal BERT is not tested alone, however when added to the NCE loss function, seems to suit a range of downstream tasks from classification to anticipation and captioning. NCE alone seems to clearly produce better results over published results, however these are not compared like-to-like, as published results are used for this comparison.\n\nThe paper is full of technical details to reproduce the results. This makes the main novelty is actually in showing that this approach works. However, the approach is technically sound and up to my knowledge has not been attempted before.", "extra_scores": {"experience_assessment": "I have published in this field for several years.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper studies self-supervised video representations with a multi-modal learning process that the authors then use for performance on a variety of tasks. The main contribution of the paper is a successful effort to incorporate BERT-like models into vision tasks.

Reviewers acknowledged the extensive empirical evaluation and the good performance of the approach. However, they raised some concerns about the lack of clarity and the absence of analysis and interpretation of the results. The AC shares this view, and recommends rejection at this time, encouraging the authors to revise their work addressing these analysis and clarity questions.  
Decision: Reject