Watching Too Much Television is Good: Self-Supervised Audio-Visual Representation Learning from Movies and TV Shows
Authors: Mahdi M. Kalayeh
Nagendra Kamath
Lingyi Liu
Ashok Chandrashekar
Venue: NeurIPS 2021 Submitted (2021)
Keywords: self-supervised learning
contrastive loss
multi-modal
audio
video
movies
TV

Abstract:     The abundance and ease of utilizing sound, along with the fact that auditory clues reveal so much about what happens in the scene, make the audio-visual space a perfectly intuitive choice for self-supervised representation learning. However, the current literature suggests that training on \textit{uncurated} data yields considerably poorer representations compared to the \textit{curated} alternatives collected in supervised manner, and the gap only narrows when the volume of data significantly increases. Furthermore, the quality of learned representations is known to be heavily influenced by the size and taxonomy of the curated datasets used for self-supervised training. This begs the question of whether we are celebrating too early on catching up with supervised learning when our self-supervised efforts still rely almost exclusively on curated data. In this paper, we study the efficacy of learning from Movies and TV Shows as forms of uncurated data for audio-visual self-supervised learning. We demonstrate that a simple model based on contrastive learning, trained on a collection of movies and TV shows, not only dramatically outperforms more complex methods which are trained on orders of magnitudes larger uncurated datasets, but also performs very competitively with the state-of-the-art that learns from large-scale curated data. We identify that audiovisual patterns like the appearance of the main character or prominent scenes and mise-en-scène which frequently occur through the whole duration of a movie, lead to an overabundance of easy negative instances in the contrastive learning formulation. Capitalizing on such observation, we propose a hierarchical sampling policy, which despite its simplicity, effectively improves the performance, particularly when learning from TV shows which naturally face less semantic diversity.

Watching Too Much Television is Good: Self-Supervised Audio- Visual Representation Learning from Movies and TV Shows
Anonymous Author(s)
Affiliation
Address
email
Abstract
The abundance and ease utilizing sound, along with the fact that auditory clues reveal so much about what happens in the scene, make the audio-visual space a perfectly intuitive choice for celf-supervised representation learning. However, the current literature suggests that training on uncurated data yields considerably poorer representations compared the curated alternatives collected in supervised manner, and the gap only narrows when the volume of data significantly increases. Furthermore, the quality of learned representations is known to heavily influ- enced by the size and taxonomy of the curated datasets used for self- supervieee training. This begs the question of whether we are celebrating too early on catching up with supervised learning when self-supervised efforts still rely almost exclu- sively on curated data. this paper, we study the efficacy of learning from Movies and TV Shows forms of uncurated data for audio visual self-supervised learning. We demonstrate that simple model based on contrastive learning, trained on a collection of movies and TV shows, not only dramatically outperforms more com- plex methods which are trained on orders of magnitudes larger uncurated datasets, but also performs very competitively with the state-of-the-art that learns from large-scale curated data. We identify that audiovisual patterns like the appearance of the main character prominent scenes and mise-en-scène which frequently occur through the whole duration of movie, lead to an overabundance of easy negative instances in the contrastive learning formulation. Capitalizing on such observation, we propose a hierarchical sampling policy, which despite its simplicity, effectively improves the performance, particularly when learning from TV shows which naturally face less semantic diversity.
11
13
15
17
19
20
21
22
23
Introduction
24
Recently, there has been tremendous progress in self-supervised learning from still images, where the standard supervised training has been outperformed variety of image-related tasks [7, 8, 15, 29]. The appeal of detaching representation learning from human annotations is rooted not only in the non-trivial challenges of scaling-ur up the labeling process, but also in the ill-defined task of determining proper taxonomy with generalization power and transferability. Both challenges only exacerbate as 30 we move from images to videos, where the notion of time involved and the complexity of visual
26
28
29
31 concepts increases. Simply considering the number of training instances even the cardinality of
32 the label set is not sufficient to conclude if one large-scale supervised dataset is more suitable than
33 another for transfer learning video classification tasks [20]. That is, the abundance of attention
34 which video self-supervised learning has lately received is only to be expected. While many research
35 efforts in this area extend the contributions made initially in the image domain the video domain,
Submitted to 35th Conference on Neural Information Processing Systems (NeurIPS 2021). Do not distribute.

---

36
others, including our work, have explored harnessing additional modalities such as audio or text for multi-modal self-super learning [2, 3,4, 4, 22, 27, 31, 37, 36, 39].
37
From the current state-of-the-art one makes two major conclusions. First, the quality of learned representations, evaluated by fine-tuning on downstream tasks, is heavily influenced by the size and laxonomy of the pretraining datasets [2, 3, 39]. Second, an uncurated pretraining dataset yields considerably poorer representations compared curated one and the gap only narrows when the total amount of pretraining data significantly increases [3]. Curated data refers to likes of supervised large-scale action recognition and audio classification datasets such as Kinetics [6], IG-Kinetics [12], AudioSet [11], and YouTube-8M [1]. While the -annaattd labels are not accessed for self-supervised pretraining, videos being trimmed and from label set of limited cardinality with biased sampling distribution' implicitly acts as a sort of supervision. On the other hand, an uncurated data refers to likes of IG-Random{33, simply body of unlaheled videos collected blindly with none of the aforementioned careful human-invol vements. That being said, we know that something as simple as having access to clean object-centric training data, like Imagenet, can be indirectly exploited by contrastive self--uper learning in image domain to obtain additional performance 51 gain [41] on the downstream tasks which exhibit similar properties. The analogous to it of course are the well trimmed closed-set curated datasets which are being extensively used in the literature for 53 video self- supervuee pretraining, while downstream evaluations focus on benchmarks with similar characteristics. Our work aims comprehensively exploring the efficacy of learning from Movies 55 and TV Shows, as forms of uncurated data, for audio-visual self supervisee learning.
38
39
40
41
42
43
44
45
48
47
48
49
Many of us can relate to an experience in movie theaters when the sound of the engine, first perceived 57 by our left ear, gradually heard more by the right ear as a car moves from the left side of the screen to the right side. Another example is scene in which an object, like helicopter, approaches
56
58
59 the camera from distance and eventually flies over it. In this case, the perceived sound not only changes in loudness but also transitions from front to back, in concert with the visuals, giving the audience a more realistic feeling as if they are indeed positioned behind the camera. Besides, with art being inherently novel, two movies even if they share genres or revolve around similar story lines often deliver quite different experiences and portray distinct visuals, thanks to the extremely artist-driven creative process behind such productions. We hypothesize that the aforementioned high audio fidelity, and inherent semantic diversity characterize long-form content? as potentially a very rich source for self-supervieed multi-modal representation learning. It is worth emphasizing that in spirit of uncurated data, we not only blindly sample from a large collection of movies and TV shows when constructing our pretraining dataset, but also perform ablation studies on the effect of genre distribution, the closest we have to taxonomy in the curated datasets, confirming that the quality of learned representations is agnostic with respect to such statistics.
61
63
64
65
67
69
70
71 To the best of our knowledge, we are the first to solely rely on uncurated data and study the efficacy of
72 self-supervised multi-modal representation learning from movies and TV shows. Despite meaningful
73 domain gap between our pretraining data and the space of downstream tasks, we obtain representations
74 which are very competitive with those learned from curated datasets. This is particularly important as
75 we follow a much simpler modeling approach in comparison with the sfate-of-the art.
78 2 Related Work
Self-supervie learning techniques define pretext tasks, mostly inspired by the natural structures
78 in the data, in order to generate supervisory signals for training. Despite the plethora of proposed
79 pretext tasks in the literature, these approaches can be coarsely divided into two groups, namely
80 pretext learning, and pretext-imvariant methods. Approaches which fall in the former bucket, usually
81 apply a form of transform, randomly drawn from a parametric family, to the input data then optimize
82 for predicting the parameters of the chosen transformation. Predicting the relative position of image
83 patches [9]. solving jigsaw puzzles [33], estimating artificial rotations [13], colorization [50], context
84 encoders learned through inpainting [38], and learning by counting scale and split invariant visual
85 primitives [34], are among many methods which belong to this category. Similar techniques have been extended from images to videos [10, 21, 24, 25, 30, 46, 48, 49], where in addition to the spatial context, the temporal domain, and the arrow of time have been heavily exploited. In contrast,
86
87
1 the associated taxonomy has similarities with those of downstream benchmarks [3] alternatively referring to movies and TV shows
2

---

88
pretext-invariant methods [5, 7, 15, 18, 17, 29, 35, 39, 44] are built on the concept of maximizing mutual information across augmented versions of single instance, and are mostly formulated as contrastive learning. In other words, a pretext is used to generate different views of single input for which the learning algorithm aims to maximize the intra-instance similarity, across variety of transformations. Our work falls within this category, however we function in multi-modal realm employing both audio and video.
89
90
91
92
93
94
Earlier works which harnessed audio and video for representation learning, have leveraged audio- visual temporal synchronization [22, 36]. correspondence [4], and cross-modal clustering [3. 37]. The work by Patrick et [39] proposes a generalized data transformation in order to unify variety of audio-visse vistal self-supervised pretext tasks through a noise contrastive formulation. This work is close to ours in choice of objective function and data type, yet we employ no augmentation (except modality projection in the terminology of [39I), and solely focus on capitalizing the advantages of learning from long-form content. Morgado ef [l][ show that cross- modll discrimination is important for learning good audio and video representations, something which was also pointed out earlier in a clustering framework [3]. Beyond that, [31] generalizes the notion of instance-level positive and negative examples by exploring cross-modal agreement where multiple instances are grouped together as positives by measuring their similarity in both the video and audio feature spaces. While we also adopt a cross modal noise contrastive estimation loss, we stick with the vanilla version, instance-level positive and negatives, and do not use any memory bank feature representations. Finally, Alayrac al. recently proposed a multi-modaa versatile network capable of simultaneously learning from audio, video and text. Building on the intuition that different modalities are of different semantic granularity, audio and video are first compared in a fine-grained space while text is compared with the aforementioned modalities in lower dimensional coarse- saarreegraieed space. In our experiments, compare with a variant of [2] where only audio and video modalities are utilized.
95
96
97
98
99
100
101
102
103
105
107
109
110
111
3 Approach
112
113
Notations and Architecture. Our pretraining dataset is denoted by X' = (Xx []] where X, [m E [I...MM,. contains M,, non-overlapping audiovisual snippets which are temporally segmented from the duration of the nh long-form content in the dataset. Each snippet includes both audio and video modalities, formally In.m = (an, m2 Up.m), where an,m E R RIxPxQ and Un.m E xx T. H, and W denote the number of frames, height and width of the video, while P. and respectively stand for the number of mel filters, and audio frames. Video and audio are processed through 18-layers deep R(2+1 I)D [45] and ResNet [16] architectures, respectively referred
114
115
118
117
118
119
to as f R + Rdd and g R. + Rdd. Inspired by [7], we use projection heads, hy R2d - R and hg 12d + Rd. map corresponding representations into a common d-dimensional space before computing the contrastive loss. The shallow architecture of f and hg consists of two convolution layers, separated by Batch Normalization [19] and ReLU [32], followed by global average pooling. Once self eeff-supervieed pretraining finished, we discard the projection heads and fine-tune f and g for respective downstream tasks.
120
121
122
123
124
125
126
Loss Function. With a slight abuse of notation B = {ax. = (az, vi)li E [1.... B]} represents a minibatch of size B, where video and audio modalities associated with the 7th sample, Tt, are denoted by i2 and a, We use 2 = (v:)) and & = hg (g(a)) to represent the associated embeddings generated by projection heads, and optimize the noise-contrastive loss [14] shown in I in order to maximize the symmetric joint probability between audio and video. For the 7th element in the minibatch, (2) 2 serves as the positive pair, while assuming negative pairs for both modalities,
127
129
130
131
NG {(2 2)), (2g, Zi E [1... B), + j} constitutes the set of negative pairs.
132
(=+))(=)) ((()))(=))
C =
Σ i=1
log
(1)
+
Σ
(EL
133 Most of the previous works [2, 31, 39] normalize the embeddings before computing the contrastive
134 loss and employ a temperature
often denoted by 7 as in [2, 31], to control the
i/ enumerates elements in the minibatch
3

---

135
smoothness for the distribution of pairwise similarities. In contrast, we have chosen to operate in an unnormalized embedding space. Besides the obvious benefit of eliminating the need for tuning т, we empirically show that such decision does not affect the quality of the learned representations.
136
137
138
Sampling Policy. Contrastive loss function shown in Equation 1 is computed over B training instances, each in form of an audiovisual snippet. A naive sampling policy may ignore the fact that snippets comprising the pretraining dataset are in fact temporal segments that were trimmed from longer-form contents, i.e. movies and TV shows. Such an assumption treats our training data as independent and identically distributed random variables from U Xx, which constitutes the default sampling policy that is commonly used in the general deep learning literature. However, in reality, commonalities and correlations do exist along the temporal axis of movie TV show, things like audio mastering artifacts, frequent appearance of the main character's ' face and voice, thematic music, repetitive scenes and misc-en-scène' all of which contribute to breaking the previously discussed ii assumption. This is even more pronounced when we deal with multiple episodes of the same TV show appearing in the pretraining dataset'. Note that, sampling from video data is going to be i..d but in this case the temporal correlations extend for much longer given our entities are movies and TV shows. Thus, it is more accurate think of X' having multiple underlying domains, oriented towards exclusive properties which different long-form contents are characterized by. We hypothesize that during training, model gradually discovers such patterns of commonalities, which are not semantically valuable, and latches onto those to quickly minimize Equation 1 leading to poor generalization', The reason being B < N, hence for n ~ U(1, NN) and m + m'. P( B In,mm B)) is negligible. In other words, the set of negative pairs in Equation 1 mainly includes pairs for which audio and video come from two different movies TV shows, thus due to the aforementioned artifacts behave
139
140
141
142
143
145
147
148
149
150
151
152
159
154
155
156
157
as easy negatives.
In order to quantitatively measure our hypothesis, we define different distributions, shown in Equation 2, over the space of -udio-visual similarity. St indicates the space of correct matches, i.e. where audio and video correspond to the same snippet. S indicates the space where audio and video do
159
160
not correspond yet belong to the same movie or TV show. Finally, S# indicates the space in which audio and video sampled from two distinct long-form content, hence naturally do not correspond.
161
162
if = n' A m = m' ST, if = n' A + m' S8 if + n' (mm, m²)
St
mm
With that, and KL denoting Kullback-Leibler divergence, KL(S- || ST) measures the expected
163
164
difference between positive and negative pairs within the same movie or TV show. Ideally, this should increase as the training progresses, since the model gradually learns audio-video correspondence by
165
minimizing Equation 1. Mcanwwile, the i.dd assumption suggests KL(S- || St) ~ KL(S* St)
186
and KL(S- ) = O, yet as we empirically illustrate later, KL || S+) < KL(S* S+) and
167
168
(L ll S*) is rather large, indicating that, upon convergence and on a held-out set, model has harder time pushing apart negative pairs when audio and video come from the same underlying long-fo form-formmon content. Next, we explain how simple alternative policy which samples k snippets from each long-form content effectively reduces both of the discrepancy measures, referring to KL || S*) and KL(S** || S+) - KL(S || S+), while yielding better generalization on range of downstream tasks.
169
170
171
172
173
174 To ameliorate the aforementioned optimization challenge, we take a hierarchical approach. In particular, we first uniformly sample long-form content, U(1, N), and then draw k distinet snippets from X., creating {zn,m I E M. where Mn C [1..- ... Mn] and MM = k. This ensures that for I; B, Nr always includes 2k 2 pairs sampled from the same movie or TV show to which x; belongs. By putting constraints on Mn. specifically how temporally far from each other the l samples are drawn, we may go one step further and to some extent control the audiovisual similarity between snippets. This serves as an additional nob to tune for hard negative sampling. The intuition is that, the larger narrative of a professionally made movie or TV show is composed of shorter units called scene. Each scene comprises a complete event, action, or block of storytelling and
175
178
177
178
179
180
181
182
collectively referred to as content-exclusive artifacts $ aalternatively think of as a very long movic created by stitching different episodes together 'refer to supplemental material for illustrations of training loss
4

---

183
normally takes place in one location and deals with one action. That is, if our samples are temporally close, it more likely for corresponding snippets to be highly correlated and/or look/sound alike. min[M.,] +1 I < M, defines the bounds on our sampling policy, where w, standing for a sampling window, determines the farthest two out of k samples drawn from X, can be. Accordingly, w = k represents the case where all k samples are temporally adjacent, hence the expected audiovisual similarity is maximized due to temporal continuity in content. We show that having such level of hard negatives, even with a small k. prevents proper training and results in performance degradation. On the other hand, w = M,, indicates random sampling where no temporal constraint is imposed on My,, thus samples are less likely to be drawn from adjacent time-stamps. In this case, expected audiovisual similarity (i.e. hardness of negative pairs) is mainly derived from global content- exclussve artifacts like, color palette, frequent appearance of the main character's face and voice, repetitive scenes, and etc. The rest of the spectrum provides middle grounds where two samples drawn from Xr can at most be + snippets apart, something reminiscent of temporal locality. Our sampling policy can be easily implemented in a few lines of Python. Please refer to supplemental material for further details.
184
k < max[MM,]
185
186
187
188
190
192
193
194
195
196
197
4
Experiments
198
199
4.1
Experimental Setup
Datasets and Reproducibility. We use full-length movies and episodes of TV shows for self-
201
supervised pretraining. Titles are randomly chosen from large collection spanning over a variety of genres, namely Drama, Comedy, Action, Horror, Thriller, Sci-Fi and Romance. All audio is in English language. Our Movie dataset, consists of 3.6K films with an average duration of 105 minutes. Our TV dataset includes 9.2K episodes from a total of 581 shows with an average duration of 42 minutes per episode. Each of our datasets comprises 0.7 years worth of uncurated audiovisual content, which is significantly smaller than IG-Random [3] with variants at 5 and 21 years. Scaling up our pretraining
202
203
205
206
207
datasets to volumes comparable to the IG-Random [3] while possible is non-trivial and demands dramatically larger compute resources for training, something which we currently cannot afford.
208
209
Given that we cannot publicly release our dataset due to copyright reasons, we acknowledge that it is not possible for other research groups to fully reproduce our results. However, we intend to make available the pretrained models and hope that research community finds them, along with the other contributions of this work, of value whether within the context of self-supervised learning or adoption for various downstream tasks. We would like to emphasize that similar limitations have precedents in multiple earlier works including but not limited to [3, 12, 26, 43). To evaluate the efficacy of self- self-super audio-vics visuul representation learning from movies and TV shows, we follow recent works [3, 39, 31, 2] and benchmark UCF101 [42] and HMDB5 I [23] for action recognition, along with ESC50[40] for audio classification. Results for the ablation studies are reported on the split-1 I of the corresponding datasets. Following the standard protocol, we report the average performance over all splits when we comparing with the -ottt-of-thharr...
210
211
212
213
214
215
216
218
220
Pretraining. Unless mentioned otherwise, we use video snippets with 16 frames 5 fps. For data augmentation, we resize the shorter side to 190 pixels, then randomly crop them into 158 x 158 pixels. As for sound, we compute mel spectrogram from the raw audio at 48K sample rate using 96 mel filters and an FFT window of 2048, while the number of samples between successive frames is set to 512. For data augmentation, we randomly drop out up to 25% from either temporal or frequency axis of the 2-D mel spectrogram image. Training uses batch size of 512 and takes on average 42 hours on 8 NVIDIA A100 GPUs The dimension of audio-video joint embedding space, d, is set to 512.
221
222
223
224
226
227
Downstream Evaluation. For training on UCF101 [42] and HMDB51 [23], we use video clips that are 32 frames long at 10 fps. Unless mentioned otherwise, these clips are randomly chosen from the duration of the video instances. A scale jittering range of [181, 226] pixels is used and we randomly
228
229
230
crop the video into 158 x 158 pixels. Furthermore, random horizontal flipping and color jittering are employed. During inference, 10 temporal clips are uniformly sampled where each is spatially cropped in 3 ways (left, center, right) resulting in a total of 30 views. We then average the model predictions across these 30 views and report top-1 classification accuracy. For training on ESC50
231
232
233
234
[40], we use 3-seconds clips which are randomly chosen from the duration of the audio instances and apply time and frequency masking to spectrogram images for data augmentation. The maximum possible length of the mask is 50% of the corresponding axis. We do not use any scale jittering or
296
random cropping on the spectrograms. During inference, 10 temporal clips are uniformly sampled
5

---

238
and we average the model predictions across these 10 views and report top- I classification accuracy. For further implementation details, please refer to the supplemental material.
239
0v0191
016
000000
0.60
0.50
64
8
04
125
391
612
1024
2348
(a)
(b)
0.06
0.7
2.07
0.01
TO
2.00
KLUS
0.1
0.02
(9L
(96, (6)
(4-1
(64.MM
(1S, MG)
(4.M)
(1-)
(8. M,)
M,)
64)
120)
1.6, (M)
(At m))
(R.w)
(c)
(d)
Figure Ablation study of the proposed sampling policy on reducing the discrepancy measures.
240
4.2
Ablation Study
In the following, we discuss multiple ablation studies to assess our main hypothesis that, a hierarchical sampling policy, as described in Section 3, enables better repre- sentations to be learned by increasing the portion of hard
242
243
0.00
244
018
245
negative pairs which the contrastive loss function observes.
0.80
246
Here, pretraining uses 90% of either Movie or TV dataset, while the remaining 10% constitute held-out validation set" on which we report the discrepancy measures.
0.6
247
0.70
248
0.70
Sample size (k) Figure 1a illustrates that compared to the baseline sampling denoted by A: = 1, our approach (k 1) effectively shrinks the gap between S and S
K
0.60
250
251
0.00
0.6
252
when measured either directly or against S+, Its pattern of behavior also perfectly follows our earlier intuition (ref. Section 3). In particular, given a fixed minibatch budget,
253
254
(At.r)
255
a larger favors more training instances to be sampled from fewer number of long-form contents. That increases the portion of hard negative pairs, thus pushes the con- trastive loss to more aggressively separate mismatched
Figure 2: Effect of color jitter on the dis- crepancy measures.
256
257
259
audio-video pairs from same movie, which leads model to maintain less of the content-exclusive artifacts in the embedding space. In the most extreme case, & = 64, all the training instances are
260
Given a TV show, either all none of its episodes included in the held-out

---

261
sampled from the same movie. From Table 1, we observe that different variants of our sampling policy, with no imposed temporal constraint, i.e. w Mn, outperform the baseline on all three downstream tasks
262
Sampling window (w) Smaller forces samples
285
that belong to same movie to be drawn from shorter temporal window, hence growing the probability that
Table I: Ablation study of the proposed sam- pling policy different downstream tasks, measured by top-1 classification accuracy.
268
257
they look/sound very much alike (i.e. harder neg- ative pairs). That is, it should further diminish the discrepancy measures, Figure 1b illustrates this be- havior where we gradually increase wo while & 16. However, from Table 1, it does not seem that tuning for w, ie w * Mn. provides a meaningful gain on downstream tasks. This implies that commonalities which persist throughout the duration of movie are sufficiently powerful signals to be exploited for gen-
268
289
pretraining dataset: Movie
270
k
10
HMDB51
ESC50
UCF101
271
272
60.32
86.50
85.69
M,
61.37
89.91
85,38
8
Ma
62.09
88.75
86.06
274
16
Mn
62.92
88.33
86.30
32
Ma
61.04
88.00
85.98
276
crating hard negatives. We hypothesize that different
Mn
61.30
86.83
85.43
scenes both within and across different movies and TV shows are of variety of length, thus a fixed w is sub-optimal. Ideally, we should identify scene
278
16
64
60.26
87.00
83.61
128
60.58
86.50
85,30
280
boundaries and and dynamically modify w during sampling. something which we leave for future iter-
16
256
62.02
87.75
84.85
16
512
61.30
87.08
85.38
16
1024
60.65
86.16
84.61
282
ations of this work.
16
2048
61.83
87.66
85.11
Temporally adjacent samples. Along the lines of previous observations, Figure Ic shows that in- deed drawing temporally adjacent snippets from the same long-form content, i.e. w = k; results in aggressively reducing the discrepancy measures. This behavior is agnostic with respect to h yet ex- acerbates k grows. Note that, the contrastive loss is an iastane-discrimination objective func-
4
60.19
88.00
84.66
284
56.86
88.75
82.71
64
64
57.45
84.58
82.68
286
pretraining dataset: TV
4:
HMDB51
ESC50
UCF101
288
1
56.40
85.50
84.37
290
8
Mn
61.50
87.50
85.96
tion. Therefore, foreing it distinguish between temporally adjacent snippets, that naturally sound and look extremely similar, leaves no choice for the model but to discard valuable semantic notions, which predictably leads to poor representations, also confirmed by result reported in Table 1.
16
Ma
61.69
89.00
85.64
292
8
64
60.58
88.00
85.96
128
60.00
85.66
85.77
284
16
256
61.30
86.41
85,01
295
296
Movies vs. TV Shows. To confirm that sampling policy behaves consistently across both movies and TV shows, Figure 1d illustrates the discrepancy measures computed on TV dataset. We observe similar effectiveness when using k and as tuning nobs for reducing either KL(S S²) or the gap between KL(S- || S+) and (L || St). Table I demonstrates that different variants of our approach significantly outperform the baseline, i.e. A = 1. We attribute the larger gains achieved when using TV instead of Movie dataset to the fact that content diversity is naturally lower when pretraining on TV shows since each one includes many episodes that all are characterized with the same exxllnveexclusive artifacts.
298
301
303
305
Color jitter. We have established far that commonalities which persist throughout the duration of a long-form content, things likely associated with color pallet, frequent appearance of the main character's face and voice, and repetitive scenes can be exploited for learning better representations.
306
307
308
That is, one may naturally assume that employing data augmentation techniques like color jitter should be helpful since by distorting content-exclusic
309
visual artifacts, color jitter is expected to reduce KL(SS- SF). Figure 2 illustrates the effect of color jitter, where brightness, contrast, and saturation jitter values are chosen uniformly from [max(0, σ), + a.. We observe that color
311
jitter reduces the discrepancy measures for the baseline but not as much it can be obtained by our proposed sampling policy (k: > 1), and even then according to Table only yields a slight gain on downstream tasks.
313
315
lz normalizee feature space. The common practice [2, 31, 39, 7] is to compute contrastive loss in an fz-normalized feature space, where according to [47] the temperature hyper-parameter, т, controls
316
7

---

Table 4: Effect of self-supervised learning from curated versus uncurated data different downstream tasks. The "years" column indicates the duration of the pretraining datasets in years.
method
pretraining dataset
uncurated
years
HMDB51
ESC50
UCF101
Ours
Movie
0.7
62,9
88.3
86.3
Ours
TV
/
0.7
61.7
89.0
85.6
XDC
IG Ne-Randoml66N
/
5
55,2
84.3
84.1
XDC[3]
IG-Random65M
/
21
61.2
86.3
88.8
XDC IG Kinetiics 16M
x
5
57.3
82.5
87.6
XDC IG IG-Kinetics65M
X
21
63.1
84.8
91.5
317
the strength of penalties on hard negative samples. We explored this with two widely-used values. From Table 2, we observe that compared to operating in unnormalized embedding space, adopting such design choice results in a large performance drop on HMDB51[23) while other downstream benchmarks see only negligible gains.
318
319
320
321
Curated vs. Uncurated data. To the best of knowledge, the only other uncurated dataset used for audio-visual self-supervised learning is IG- [3 Table 4 confirms that learning from curated movies and TV shows is extremely effective. Our results significantly exceed those of XDC[3] obtained on IG-Random 16M despite using sim- pler model and times smaller volume of pretrain- ing data. Even in comparison to (G-Random65M with 30 times larger data, we obtain better perfor-
Table 2: Effect of color jitter (Ø) and comput- ing contrastive loss in Ez-normalized embed- ding space with temperature hyper-paranen (T) on different downstream tasks.
323
325
327
k
HMDB51
ESC50
UCF101
328
0.0
60.32
86.50
85.69
329
0.0
62.92
88.33
86.30
330
mances on 2 out of 3 benchmarks. The most promis-
60.45
87.66
84.82
332
ing of our findings though is how competitive our
0.5
60.13
87.75
85,9
results are against XDC[3] when it is trained on variants of IG-Kineties which are not only curated
16
1.0
61.11
88.33
85.93
334
k:
HMDB51
ESC50
UCF101
but also orders of magnitude larger. With all that, we confidently reject the notion that mudio-visual self-supervised learning from uncurated data con- siderably lags behind utilizing large-scale curated datasets.
16
0.07
60.78
87.08
86.86
336
0.30
60.78
89.25
85.72
338
Table 3: Effect of genre distribution in Movie dataset different downstream tasks. Experi- ments are conducted with input spatial resolu- tion of 112 112 pixels.
340
Effect of genre. The distribution of genre among movies used in our pretraining is the closest we have to taxonomy in the curated datasets. So, is worth examining the quality of our learned repre-
342
setting
HMDB51
ESC50
UCF101
344
sentations under various genre distributions. To do
57.58
86.50
82.44
50, given fixed pretraining budget (N =1.6K), we compare four different scenarios where movies used in the pretraining are distributed i) non-uniformly over all genres except Drama, and Comedies, ii) non-urifymlly over Drama, and Comedies. iii) uni-
ii 56.99
85.50
82.39
346
in 56.27
85.25
82.87
iv 56.40
86.75
83.24
348
350
formly over all genres, and iv) non- con-uniformly over all genres. Table 3 confirms that indeed there is very little difference between the aforementioned setups when it comes to transfer learning to the downstream tasks.
352
353
4.3
Comparison with state-of-the-art
354
Table 5 compares our proposed approach of learning from Movies and TV shows against the best performing audio-visus self-supervised learning methods. In general, our numbers are comparable with the best existing results reported in the literature, even with much less data and considerably simpler model/training procedure". It is interesting that training on Movie dataset alone obtains
355
356
$ the data not publicly available, and similarly the implementation to train XDC[3]
supplemental material includes comparison of training costs

---

Table 5: Comparison with state-of-the-art. Dataset abbreviations: AudioSer[11] HowToo00MN IG- Ki-etiieiis6MM [12]; their length in years is given in the "years" column. "Arch." denotes the architecture of video backbone (f). [2] indicates when the corresponding model only audio and video, and not text modality. For a fair comparison, when using only Movie dataset, we train for twice many epochs our other variants in order match their total number of gradient updates.
Method
Arch.
pretraining dataset
curated
years
HMDB51
UCF101
ESC50
GDT[39]
R(2+1)D-18
AS IG65M
1
66.1
92.5
88.5
GDT[39]
R(2+1JD-18
21
72.8
95.2
XDC[3]
R(2+1)D-18
AS
91.2
84.8
XDC[3]
R(2+1)D-18
IG65M
21
67.4
94.2
2
MC3
AS
1
61.6
89.0
82.3
AVID[31]
R(2+1)D-18
AS
1
64.7
91.5
89.1
MMV
R(2+1)D-18
AS
1
70.1
91.5
85.6
M
S3D-G
AS
1
68.2
90.1
86.1
MMV2]
S3D-G
AS+HIT
16
68.3
91.1
87.2
Ours (k=16)
R(2+1)D-18
Movie
0.7
64.5
87.9
88.8
Ours (k=8)
R(2+1)D-18
Movie+TV
X
1.4
65.0
87.7
89.1
Ours (A=16)
R(2+1)D-18
Movie+TV
X
1.4
65.1
88.5
89.1
Ours (k=32)
R(2+1)D-11
Movie+TV
1.4
65.6
88.7
88.2
358
comparable performance to the cases where both TV and Movie datasets are used for pretraining.
This further confirms the richness of the training data which movies and TV shows can provide to self-supervised learning problems. also see that increasing even beyond gives further incremental gains action recognition benchmarks.
360
Conclusion
362
Despite its amazing recent progress, state-of-the-art self- self-supervised learning still heavily relies supervised, i.e. curated, large-scale datasets for pretraining. In this work, we have shown that pretraining solely uncurated data in forms of movies and TV shows, even comparatively small scale, can give rise to representations which are capable of competing with the state-of-the- of more complex architectures trained on larger curated datasets. This comes contrary to the current literature which tends to suggest that learning from uncurated data largely falls behind
364
366
387
388
389
the use of curated alternatives. We intentionally made design decisions keep our approach and
training strategy as simple as possible to demonstrate that learning decently powerful audio-visual
370
representations does not necessarily require gigantic data and compute resources. Through extensive set of experiments, our work establishes for the first time the efficacy of self-supervised learning of
372
373
audio-visual representations from movies and TV shows.
Broader impact
374
375
Potential benefits. Our work shows that competitive multimodal representations can be learned from comparatively small volume of uncurated data in the form of movies and TV shows. Besides minimizing any sort of human-involvement, which we believe must have already been paid an extra attention to in the literature, compute resources for effective self-supervised pretraining. Such results promise a more democratized research arena where smaller groups are not alienated due lack of sufficient compute resources. More importantly, lowering the compute requirements naturally reduces any environmental effects which training these models can potentially have.
377
work demonstrates that one does not require gigantic data and
379
381
385
Potential risks. Any machine learning method is susceptible to the potential underlying biases in the data. This is more important for self-supervised methods that deal with huge volumes, often not evaluated by diverse group of humans for any fairness concerns. The same is generally true in our
385
case which requires to make sure that titles that are included in training are diverse and inclusive.

---

References
387
[1] S. Abu-EI-Haija, N. Kothari. J. Lee. P. Natsev. G. Toderici. B. Varadarajan, and S. Vijayanraaimmhnn. Youtube-8m: A large-scale video classification benchmark. arXiv preprint wwt.i::600.000000 2016.
389
[2] J.-B. Alayrac, A. Recasens, R. Schneider, R. Arandjelovié, J. Ramapuram, J. De Fauw, L. Smaira,
391
S. Dieleman, and A. Zisserman. Superruuee MultiModal Versatile Networks. In NeurIPS, 2020.
[3] H. Alwassel, D. Mahajan, B. Korbar, L. Torresani, B. Ghanem, and D. Tran. Self-supervised learning by
393
cross-modal audio-video clustering. In Advances in Neural Information Processing Systems, volume 33, pages 9758-9770, 2020.
394
385
[4] R. Arandjelovic and A. 7isserman. Look, listen and learn. In Proceedings of the IEFE International Conference on Computer Vision, pages 609-617, 2017.
396
397
[5] P. Bachman, R. D. Hjelm, and W. Buchwalter. Learning representations by maximizing mutual information across views. In Advances in Neural Information Processing Systems, pages 15535-15545, 2019.
398
399
[6] J. Carreira and A. Zisserman. Quo vadis, action recognition? a new model and the kinetics dataset. In proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 6299-6308, 2017.
400
402
[7] T. Chen, S. Kornblith, M. Norouzi, and G. Hinton. A simple framework for contrastive learning of visual representations. arXiv preprint arXiv:2002.057700 000009 2020.
404
[8] T. Chen, S. Kornblith, K. Swersky, M. Norouzi, and G. Hinton. Big self-supervised models are strong semi-supervised learners. arXiv preprint 2rXiv:2006.1000 2020.
406
[9] C. Doersch, A. Gupta, and A. A. Efros. Unsupervised visual representation learning by context prediction.
In Proceedings of the IEEE international conference on computer vision. pages 1422-1430. 2015.
[10] B. Fernando, H. Bilen, E. Gavves, and S. Gould. supe-supe video representation learning with odd-one-out networks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 3636-3645, 2017.
410
[11] J. Gemmeke, D. P. Ellis, D. Freedman, Jansen, W. Lawrenee, C. Moore, M. Plakal, and M. Ritter. Audio set: An ontology and human-labclees dataset for audio events. In 2017 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 776-780. IEEE, 2017.
412
413
414
[12] D. Ghadiyaram, D. Tran, and D. Mahajan. Large-scale weakly-supervised pre- pre-training for video action recognition. In Proceedings of the IREE Conference on Computer Vision and Pattern Recognition, pages 12046-12055. 2019.
415
417
[13] S. Gidaris, P. Singh, and N. Komodakis. Unsupervised representation learning by predicting image rotations. arXiv preprint arXiv:1803.07728, 2018
419
[14] M. Gutmann and A. Hyvärinen. Noise-contrastive estimation: A new estimation principle for unnormalized statistical models. In Proceedings the Thirteenth International Conference on Artificial Intelligence and Statistics, pages 297-304, 2010.
421
[15] K. He, H. Fan, Y. Wu, S. Xie, and R. Girshick. Momentum contrast for unsupervised visual representation learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 9729-9738, 2020.
423
424
425
[16] K. He, Zhang, S. Ren, and Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770-778, 2016.
426
427
[17] O. J. Hénaff, A. Srinivas, De Fauw, A. Razavi, C. Doersch, S. Eslami, and A. v. d. Oord. Data-efficient image recognition with contrastive predictive coding. arXiv preprint M.X.vv9995.09777 2019.
428
429
[18] R. D. Hjelm, A. Fedorov, S. Lavoic-Marchildoo K. Grewal, P. Bachman, A. Trischler, and Y. Bengio. Learning deep representations by mutual information estimation and maximization. arXiv preprint arXiv: 06670, 2018
430
432
[19] S. loffe and C. Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. arXiv preprint arXiv:1502.0316; 2015.
434 [20] H. Kataoka, T. Wakamiya, K. Hara, and Y. Satoh. Would sega-scale datasets further enhance spatiotempo- ral 3d cans? arXiv preprint 200:2000400498.. 2020.
10

---

436
[21] D. Kim, D. Cho, and I. S. Kweon. Self-supervised video representation learning with space-time cubic puzzles. In Proceedings of the AAAI Conference Artificial Intelligence, volume 33, pages 8545-8552, 2019.
438
439
[22] B. Korbar, D. Tran, Torresani. Cooperative learning audio and video models from self- supervise
440
synchronization. In Advances in Neural Information Processing Systems, pages 7763-7774, 2018.
441
[23] H. Kuchne, H. Jhuang, E. Garrote, T. Poggio, and T. Serre. Hmdb: a large video database for human motion recognition. In 2011 International Conference on Computer Vision, pages 2556-2563. IEEE, 2011.
442
443
[24] Z. Lai and W. Xie. Self-supervised learning for video correspondence flow. arXiv preprint arXiv: 1905000055,05 2019.
444
445
[25] H.-Y. Lee, J.-B. Huang, M. Singh, and M.-H. Yang. Unsupervised representation learning by sorting sequences. In Proceedings of the IEEE International Conference Computer Vision, pages 667-676, 2017.
447
[26] D. Mahajan, Girshick, V. Ramanathan, K. He, M. Paluri, Y.I., A. Bharambe, and L., van der Maaten. Exploring the limits of weakly supervised pretraining. In Proceedings of the European Conference on Computer Vision (ECCV), pages 181-196, 2018.
449
450
451
[27] A. Miech, J.-B. Alayrac, L. Smaira, I. Laptev, J. Sivic, and A. Zisserman. End-to-end learning of visual representations from uncurated instructional videos. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 9879 9889, 2020.
452
[28] A. Miech, D. Zhukov, J.-B. Alayrac, M. Tapaswi, I. Laptev, and J. Sivic. Howtol00m: Learning text-video embedding by watching hundred million narrated vidco clips. In Proceedings of the IEFE international conference on computer vision, pages 2630-2640, 2019.
455
456
457
[29]
Misra and L. v. Maaten. Self-supervised learning pretext-invariant representations. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 6707-6717, 2020.
458
459
[30] I. Misra, C. L. Zitnick, and M. Hebert. Shuffle and learn: unsupervised learning using temporal order verification. In European Conference Computer Vision, pages 527-544. Springer. 2016.
480
461
[31] P. Morgado, N. Vasconcelos, and I. Misra. Audio-visual instance discrimination with cross-modal agreement. arXiv preprint arXiv:2004. 12943, 2020.
[32] V. Nair and G. F. Hinton. Rectified lincar units improve restricted boltzmann machines. In ICML, 2010.
464
[33] M. Noroozi and P. Favaro. Unsupervised learning of visual representations by solving jigsaw puzzles. In European Conference on Computer Vision, pages 69-84. Springer, 2016.
[34] M. Noroozi, H. Pirsiavash, and P. Favaro. Representation learning by learning count. In Proceedings of the IEEE International Conference on Computer Vision, pages 5898-5906, 2017.
487
468
[35] A. Oord, Y. Li, and O. Vinyals. Representation learning with contrastive predictive coding. arXiv preprint Xiv: 1xX::1887.03744 807.03748, 2018.
469
470
[36] A. Owens and A. Efros. Audio-visual scene analysis with self-supervised multisensory features. In
471
Proceedings of the European Conference on Computer Vision (ECCV), pages 631-648, 2018.
472
[37] A. Owens, J. Wu, J. H. McDermott, W. T. Freeman, and A. Torralba. Ambient sound provides supervision for visual learning. In European conference on computer vision, pages 801-816. Springer, 2016.
473
474
[38] D. Pathak, P. Krahenbuhl, J. Donahue. T. Darrell, and A. A. Efros. Context encoders: Feature learning by inpainting. In Proceedings of the IEEE conference computer vision and pattern recognition, pages 1336-2544, 2016.
476
[39] M. Patrick, Y. M. Asano, R. Fong, J. F. Henriques, G. Zweig, and A. Vedaldi. Multi-modal self-supervision from generalized data transformations. arXiv preprint arXiv:2003.0429. 2020.
478
479
[40] Piczak. Esc: Dataset for environmental sound classification. In Proceedings of the 23rd ACM
480
international conference on Multimedia, pages 1015-1018, 2015.
481
[41] S. Purushwalkam and A. Gupta. Demystifying contrastive self-supervised learning: Invariances. augmen- tations and dataset biases. In H. Larochelle, M. Ranzato, R. Hadsell, M. F. Balcan, and H. Lin, editors. Advances in Neural Information Processing Systems, volume 33, pages 3407-3418, 2020.
482
11

---

484
[42] K. Soomro, A. R. Zamir, and M. Shah. Ucf101: A dataset of 101 human actions classes from videos in the wild. arXiv preprint arXiv:1212.0402 2012.
[43] C. Sun, A. Shrivastava, S. Singh, and A. Gupta. Revisiting unreasonable effectiveness of data in deep learning era. In Proceedings of the IEFE international conference on computer vision, pages 843-852, 2017.
486
489
[44] Y. Tian, D. Krishnan, and P. Isola. Contrastive multiview coding, arXiv preprint arXiv:1906.0584 2019.
[45] D. Tran, H. Wang, I.. Torresani, J. Ray, Y. LeCun, and M. Paluri. A closer look spatiotemporal convolutions for action recognition. Proceedings of the IEFE conference on Computer Vision and Pattern Recognition, pages 6450-6459, 2018.
491
482
493
[46] C. Vondrick, A. Shrivastava, A. Fathi, S. Guadarrama, and K. Murphy, Tracking emerges by colorizing videos. In Proceedings of the European conference on computer vision (ECCV), pages 391-400 2018.
494
495
[47] F. Wang and H. Liu. Understanding the behaviour of contrastive loss. arXiv preprint arXiv:0012.097400 2020.
497
[48] J. Wang. J. Jiao, L. Bao, S. He, Y. Liu, and W. Liu, Self-supervised spatio-temporal representation learning for videos by predicting motion and appearance statistics. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 4006 4015, 2019.
499
[49] D. Xu, J. Xiao, Z. Zhao, J. Shao, D. Xie, and Y. Zhuang. Sele-super spatiotemporal learning via video clip order prediction. In Pruceedings of the IEEE Conference on Computer Vision and Pattern Recognition. pages 10334-10343, 2019.
501
502
503
[50] R. Zhang, Isola, and A. Efros. Colorful image colorization, In European conference on computer vision, pages 649-666. Springer, 2016.
504
Checklist
505
1. For all authors...
507
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope? [Yes]
508
509
(b) Did you describe the limitations of your work? [Yes] Specifically, training data being proprietary creates concerns around reproducibility. which has precedence in the literature as mentioned in the paper. We address that partially by planning to publicly release pretrained models.
511
513
(c) Did you discuss any potential negative societal impacts of your work? [Yes]
514
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you are including theoretical results... 516
517
(a) Did you state the full set of assumptions of all theoretical results? [N/A]
(b) Did you include complete proofs of all theoretical results? [N/A]
519
3. If you ran experiments..
520
(a) Did you include the code, data, and instructions needed to reproduce the main exper- imental results (either in the supplemental material or as a URL)? [No] The data is proprietary. However, we have provided implementation of the proposed method in supplemental material and aim to publicly release the pretrained models.
522
523
524
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [Yes] They are all discussed in detail either in the main submission or in supplemental material.
526
527
(c) Did you report error bars (e.g., with respect to the random seed after running experi- ments multiple times)? [Yes]] We only observed meaningful differences after running experiments multiple times, for ESC50[40] downstream experiments. Corresponding standard errors are reported in supplemental material.
528
529
530
531
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, cloud provider)? [Yes] Please refer to Section 4.1.
12

---

4. If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
534
(a) If your work uses existing assets, did you cite the creators? [N/A]
535
(b) Did you mention the license of the assets? [N/A]
(c) Did you include any new assets either in the supplemental material or as URL? [N/A]
537
538
(d) Did you discuss whether and how consent was obtained from people whose data you're using/curating? [N/A]
539
540
(e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content? [N/A]
542
5.
If you used crowdsourcing or conducted research with human subjects...
543
(a) Did you include the full text of instructions given to participants and screenshots, applicable? [N/A]
545
(b) Did you describe any potential participant risks, with links Institutional Review Board (IRB) approvals, if applicable? [N/A]
546
(c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation? [N/A]
548
13


Reviews:
[{"review_id": "q0UvemSjvaF", "reviewer": "Reviewer_Lidi", "summary": "The task is to learn audio-visual representations from uncurated data such as movies and TV videos. Unlike previous works that show learning from uncurated data is ineffective, the authors show good performance from learning from these videos. They propose hierachical sampling policy, with negatives being sampled from both within and across different sources. The model is trained on the dataset that the authors collected, which consists of thousands of hours. The performance on various downstream tasks (action recognition, sound classification) is on par compared to using much larger uncurated datasets such as IG-Random.", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The authors tackle an important problem of learning uncurated datasets. The effective use of uncurated data is an important question and a future direction for the ML community. \n\nThe dataset used in the paper is of significant size -- 1.4 years (12k hours). The sources are from TV/ movies, which is already somewhat refined and curated compared to user-created content such as IG-Random. This might explain the performance advantage over IG-Random. The data has not been released due to copyright.\n\nThe method used is fairly standard contrastive learning. The authors propose sampling strategies to adjust the difficulty of negatives from same and different source videos. The taking of negatives from same/different videos has been shown in previous works, but the adjustment of temporal windows based on the intuition on formulation on scenes is original. However, the effect of sampling strategies shown in Table 1 is marginal, which leads me to think that the good performance comes from the quality of inputs rather than the training strategy.\n\nOverall, it is notable that the authors highlighted the TV/ movies data as a good source of self-supervised learning, but the technical contributions are not sufficient to merit presentation at the conference.\n\nThe writing is clear.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The explanation in Sec 6 is adequate.", "review_text": "", "extra_scores": {"ethical_concerns": "No concerns", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "2hr"}, "extra_text": {}}, {"review_id": "nN_0iNnV9uk", "reviewer": "Reviewer_cgAj", "summary": "The paper proposes a method to learn self-supervised audio-visual representations from long (order of half to couple hours) uncurated videos namely from movies and TV shows. A new sampling approach is presented to ensure batches are created while taking into consideration the effect of re-occurring audio-visual patterns (main character or background scenes). Picking multiple clips from the same long video that are spaced by some minimum distance is found to be effective as compared against random sampling. Pretrained models are evaluated on standard action recognition tasks: UCF101 and HMDB, and audio classification on ESC50.", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "*Strengths:*\n1. The paper makes a strong case for using uncurated datasets used for pretraining.\n2. Using KL divergence to measure the quality of batches created for contrastive learning is interesting. In particular, desirable features are observed by the proposed sampling scheme - a low KLD between negative clips from the same movie ($S^-$) and other movies ($S^\\neq$), and a small difference between $S^\\neq$ and $S^+$ and $S^-$ and $S^+$ (see Fig. 1).\n\n*Weaknesses:*\n1. Learning from uncurated data. Firstly, I'm not convinced that movies and TV shows are a uncurated dataset. In fact, as compared to most user generated content on Youtube, one could argue that they are highly curated with a large amount of editing and production value. Please also see point 2, which clarifies how movie data is not uncurated with respect to downstream tasks either.\n\n*Discussion for point 1, not influencing the rating*. I think the paper raises some interesting questions on whether it is possible to truly acquire random uncurated audio-visual data. Randomly picking Instagram/Youtube ids too has a bias of the video being on Instagram/Youtube in the first place, so it's hard to say that IG-Random [3] is in fact uncurated. In fact, I'm not even convinced that we need to train machines on uncurated videos - humans don't simply listen and watch *everything* to learn about the world, we go study at schools and universities and experience curated learning.\n\n2. One of the downstream evaluation datasets, HMDB51, has a majority of clips sourced from movies. HMDB51 also tends to show highest performance improvements for the video representation in the experiments when training with movies (+2.6% in Table 1 over k=1), while UCF101 which features the sports categories, not covered commonly in movies and TV shows, does not improve much (+0.6% in Table 1). This raises the question of whether we should be training on uncurated data. More importantly it goes against one of the claims on lines 72-73: that there is a \"meaningful domain gap between our pretraining data and space of downstream tasks\". In fact, ESC50 categories may also benefit more by sound effects inserted in TV/movies as compared to speech or background-music heavy vlogs, messages, or instructional videos found on youtube/instagram, but this would require further investigation.\n\n3. While the core idea presented here has merit - how to sample when given long videos - I don't see anything special catered towards movies/TV, for example, by actually analyzing their content. I see two options:    \na) Either position the paper on how to create batches from long videos and also demonstrate the effectiveness of the sampling algorithm by pretraining on other datasets with long(ish) videos such as HowTo100M (million videos of 6 minutes average duration, admittedly not hours); or  \nb) it would be useful to incorporate some movie/TV content analysis if the core message is about the benefits of training on such data.\n\n4. (not influencing rating) The argument about such data democratizing research for smaller groups with limited compute resources should be removed. Firstly, such data is not commonly available: 3600 films and 9200 TV episodes. Secondly, training required almost 2 days with 8 A100 GPUs - 320 GB memory could often be 50-100% of the GPU memory of an academic lab, 32 GPUs x 10GB. In such a case, even creating just the final version of Fig. 1 (20+ experiments), will require all GPUs used for 40 days!\n\n5. Minor:\na) Fig. 1b, k is said to be 16. Similarly, what is w for Fig. 1a?  \nb) Is the batch size 64 as indicated on L260?  \nc) Making highest performance numbers bold in the tables would improve readability.  \nd) While Fig. 1 is very interesting, it would also be useful to include the actual values of KL(S^- || S^+) as an indicator of training performance, since the desired low scores can be achieved for the other metrics with a failure mode in training where all S+, S-, S^\\neq lead to similar scores.\n\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Some worry of using the models trained on movies for downstream tasks as movies may often be racially insensitive and surface social problems. This is however indicated in the paper.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "4"}, "extra_text": {"ethical_concerns": "Perhaps some, related to the use of copyrighted materials -- movies and TV series -- for training. The paper does talk about it and indicates that there is precedence for it, however, does not mean that previous works had obtained permission."}}, {"review_id": "WZmEtu1GODr", "reviewer": "Reviewer_8Arr", "summary": "The paper works on self-supervised audio-visual representation learning. It shows uncurated movies and TV shows are good sources for learning audio-visual representation. The proposed model targets on short video clips like 3-second clips, and learned representation is evaluated on action classification and audio classification. ", "questions": "", "limitations": "", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The paper trains models on a large-scale TV and movie dataset for representation learning, and it shows uncurated TVs and movies are good for audio-visual representation learning with a simple contrastive method. This message is valuable because there are not many large-scale TV/movie datasets available for representation learning. But the technical contribution of the paper is limited. The key method is a basic contrastive loss, and additionally, the paper explores sampling strategy, which is not very new (see limitation). \n\nThe paper provides detailed experimental studies on sampling strategy, color jittering and the effect of genre. The quality of submission is good.\n\nOverall, the paper is clearly written. But figure 1 is hard to read. I suggest giving short names for different KL divergence values and adding more captions. \n\nThe key message from the paper is encouraging, showing that uncurated TV shows and movies are a good data source for audio-visual representation learning. However, public access to these copy-right datasets is a long-standing problem. The result of the paper will be more significant if these datasets can be publicly released in some way.  \n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Given the fact that the TV and movie datasets cannot be released and only the pretrained models will be publicly shared, I think the representation quality is not strong enough (measured by downstream performance). If the TV and movie dataset can be released it would be an impactful dataset for the community, but without that the technical contribution of the paper is limited.\n\nThere are some properties of the TV and movie datasets that are neglected. I think the audio accompanying TVs and movies could be background music (not much relevant to visual information), human’s speech (may correspond to the human's face/body), and the sound of objects/scenes that the model aims to learn. In other words, some portion of the audio is noise for audio-visual representation learning. But the proposed basic contrastive loss does not explicitly handle this.\n\nThe tech details about sampling policy (e.g. sampling clips from the same source video as hard negative, and the sampling windows affects difficulties) was studied in previous audio-visual paper [1] and video paper [2]. Thus the novelty is limited. \n\nBesides, I think the experiments do not match the claimed contribution very well. It feels the claimed main contribution is the power of large-scale TV and movie datasets, then maybe the paper can consider comparing the same method trained on other TV/movie datasets (like [3]). \n\n* [1] Cooperative Learning of Audio and Video Models from Self-Supervised Synchronization, Korbar et al. \n* [2] Video Representation Learning by Dense Predictive Coding, Han et al.  \n* [3] Condensed Movies: Story Based Retrieval with Contextual Embeddings, Bain et al. \n", "review_text": "", "extra_scores": {"ethical_concerns": "There is no major ethical concern. ", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "4"}, "extra_text": {}}, {"review_id": "C9f4QaQ-l9C", "reviewer": "Reviewer_zZqb", "summary": "This paper performs self-supervised, audio-visual pretraining of deep CNNs on movies and TV shows. The downstream tasks investigated are action classification in videos (HMDB51 and UCF101 datasets) and environmental audio classification (ESC50 dataset). The paper shows that pre-training on 0.7-1.4 years worth of TV and movies results in downstream task performance that is competitive with, but lags slightly behind, the current state-of-the-art approaches that pre-train on datasets like HowTo100M, AudioSet, and IG-Kinetics65M. ", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n\n- The paper studies a timely problem that is of broad and growing interest to the community\n\n- The paper is clear and easy to read\n\n- The paper demonstrates that Hollywood Movies and TV shows are an effective data source for learning audio-visual representations\n\n\nWeaknesses:\n\n- Aside from the novel dataset used, the paper offers little in the way of modeling or algorithmic novelty over the existing state of the art. \n\n- The downstream results slightly lag current SotA approaches trained on other widely-used datasets\n\n- It would have been better to also run baseline models on the movie dataset, as current experiments tangle up architecture with training data\n\n- One of the paper's main arguments is that uncurated data is preferable over curated datasets, but the licensing and distribution restrictions surrounding the data used here (movies, TV shows) make it even more difficult for the research community to use than existing datasets based on e.g. YouTube. It’s also not clear to me why a collection of Hollywood movies and TV shows that are categorized by genre qualifies as “uncurated” when a dataset of how-to videos scraped from youtube (e.g. HowTo100M) does not.\n\n\nSuggestions to authors:\n\nIt seems to me that the experiments here show that long-form content like movies offer no advantage over scraped youtube videos when incorporated into architectures that make i.i.d. assumptions about the relationship of clips drawn from the same video. It seems like a key difference between long-form content and youtube content is the presence of an extensive overarching narrative structure, which I can see being useful in future work but wasn’t leveraged here.\n\nI would have liked to seen a comparison between the TV+Movie dataset and a subset of HT100M or IG-Kinetics65M that was sized to match the duration of the TV+Movie dataset; the current results don’t establish why pre-training on TV+Movies lags behind the other datasets. Is it simply because the other datasets are larger, or is one video domain naturally more informative than others? \n\nFollowing from the previous point, it would have been even more informative if the paper included experiments that held the modeling architecture and learning algorithm fixed, and varied the dataset used for pre-training.\n\n-----Post rebuttal update:\nAfter reading the authors' feedback and the thoughts of the other reviewers, my assessment of the paper remains unchanged. \n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The authors do acknowledge the copyright restrictions which would make their dataset generally inaccessible to most of the research community. ", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "2.5"}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper has a good message about using uncurated data for audio-visual self-supervised learning. However, the reviewers were not convinced that this paper carried through on the message, since TV material and movies are curated to an extent (though not class balanced). 

The reviewers identified a number of weaknesses, including:

- insufficient novelty

- algorithm not applied to other (less curated) datasets

The rebuttal did not change their opinion. 

The authors should take note of the suggestions for how to improve the paper,  particularly from Reviewer cgAj

Decision: Reject