Watching Too Much Television is Good: Self-Supervised Audio-Visual Representation Learning from Movies and TV Shows
Year: 2021

Abstract:
The abundance and ease of utilizing sound, along with the fact that auditory clues 1 reveal so much about what happens in the scene, make the audio-visual space a 2 perfectly intuitive choice for self-supervised representation learning. However, 3 the current literature suggests that training on uncurated data yields considerably 4 poorer representations compared to the curated alternatives collected in supervised 5 manner, and the gap only narrows when the volume of data significantly increases. 6 Furthermore, the quality of learned representations is known to be heavily influ7 enced by the size and taxonomy of the curated datasets used for self-supervised 8 training. This begs the question of whether we are celebrating too early on catching 9 up with supervised learning when our self-supervised efforts still rely almost exclu10 sively on curated data. In this paper, we study the efficacy of learning from Movies 11 and TV Shows as forms of uncurated data for audio-visual self-supervised learning. 12 We demonstrate that a simple model based on contrastive learning, trained on a 13 collection of movies and TV shows, not only dramatically outperforms more com14 plex methods which are trained on orders of magnitudes larger uncurated datasets, 15 but also performs very competitively with the state-of-the-art that learns from 16 large-scale curated data. We identify that audiovisual patterns like the appearance 17 of the main character or prominent scenes and mise-en-scène which frequently 18 occur through the whole duration of a movie, lead to an overabundance of easy 19 negative instances in the contrastive learning formulation. Capitalizing on such 20 observation, we propose a hierarchical sampling policy, which despite its simplicity, 21 effectively improves the performance, particularly when learning from TV shows 22 which naturally face less semantic diversity. 23

The abundance and ease of utilizing sound, along with the fact that auditory clues1 reveal so much about what happens in the scene, make the audio-visual space a2 perfectly intuitive choice for self-supervised representation learning. However,3 the current literature suggests that training on uncurated data yields considerably4 poorer representations compared to the curated alternatives collected in supervised5 manner, and the gap only narrows when the volume of data significantly increases.6 Furthermore, the quality of learned representations is known to be heavily influ-7 enced by the size and taxonomy of the curated datasets used for self-supervised8 training. This begs the question of whether we are celebrating too early on catching9 up with supervised learning when our self-supervised efforts still rely almost exclu-10 sively on curated data. In this paper, we study the efficacy of learning from Movies11 and TV Shows as forms of uncurated data for audio-visual self-supervised learning.12 We demonstrate that a simple model based on contrastive learning, trained on a13 collection of movies and TV shows, not only dramatically outperforms more com-14 plex methods which are trained on orders of magnitudes larger uncurated datasets,15 but also performs very competitively with the state-of-the-art that learns from16 large-scale curated data. We identify that audiovisual patterns like the appearance17 of the main character or prominent scenes and mise-en-scène which frequently18 occur through the whole duration of a movie, lead to an overabundance of easy19 negative instances in the contrastive learning formulation. Capitalizing on such20 observation, we propose a hierarchical sampling policy, which despite its simplicity,21 effectively improves the performance, particularly when learning from TV shows22 which naturally face less semantic diversity.23
1 Introduction24
Recently, there has been tremendous progress in self-supervised learning from still images, where the25 standard supervised training has been outperformed in a variety of image-related tasks [7, 8, 15, 29].26 The appeal of detaching representation learning from human annotations is rooted not only in the27 non-trivial challenges of scaling-up the labeling process, but also in the ill-defined task of determining28 a proper taxonomy with generalization power and transferability. Both challenges only exacerbate as29 we move from images to videos, where the notion of time is involved and the complexity of visual30 concepts increases. Simply considering the number of training instances or even the cardinality of31 the label set is not sufficient to conclude if one large-scale supervised dataset is more suitable than32 another for transfer learning in video classification tasks [20]. That is, the abundance of attention33 which video self-supervised learning has lately received is only to be expected. While many research34 efforts in this area extend the contributions made initially in the image domain to the video domain,35
Submitted to 35th Conference on Neural Information Processing Systems (NeurIPS 2021). Do not distribute.
others, including our work, have explored harnessing additional modalities such as audio or text for36 multi-modal self-supervised learning [2, 3, 4, 22, 27, 31, 37, 36, 39].37
From the current state-of-the-art one makes two major conclusions. First, the quality of learned38 representations, evaluated by fine-tuning on downstream tasks, is heavily influenced by the size and39 taxonomy of the pretraining datasets [2, 3, 39]. Second, an uncurated pretraining dataset yields40 considerably poorer representations compared to a curated one and the gap only narrows when the41 total amount of pretraining data significantly increases [3]. Curated data refers to likes of supervised42 large-scale action recognition and audio classification datasets such as Kinetics [6], IG-Kinetics43 [12], AudioSet [11], and YouTube-8M [1]. While the human-annotated labels are not accessed for44 self-supervised pretraining, videos being trimmed and from a label set of limited cardinality with45 biased sampling distribution1 implicitly acts as a sort of supervision. On the other hand, an uncurated46 data refers to likes of IG-Random[3], simply a body of unlabeled videos collected blindly with47 none of the aforementioned careful human-involvements. That being said, we know that something48 as simple as having access to a clean object-centric training data, like Imagenet, can be indirectly49 exploited by contrastive self-supervised learning in image domain to obtain additional performance50 gain [41] on the downstream tasks which exhibit similar properties. The analogous to it of course are51 the well trimmed closed-set curated datasets which are being extensively used in the literature for52 video self-supervised pretraining, while downstream evaluations focus on benchmarks with similar53 characteristics. Our work aims at comprehensively exploring the efficacy of learning from Movies54 and TV Shows, as forms of uncurated data, for audio-visual self-supervised learning.55
Many of us can relate to an experience in movie theaters when the sound of the engine, first perceived56 by our left ear, is gradually heard more by the right ear as a car moves from the left side of the57 screen to the right side. Another example is a scene in which an object, like a helicopter, approaches58 the camera from distance and eventually flies over it. In this case, the perceived sound not only59 changes in loudness but also transitions from front to back, in concert with the visuals, giving the60 audience a more realistic feeling as if they are indeed positioned behind the camera. Besides, with61 art being inherently novel, two movies even if they share genres or revolve around similar story62 lines often deliver quite different experiences and portray distinct visuals, thanks to the extremely63 artist-driven creative process behind such productions. We hypothesize that the aforementioned high64 audio fidelity, and inherent semantic diversity characterize long-form content2 as potentially a very65 rich source for self-supervised multi-modal representation learning. It is worth emphasizing that in66 spirit of uncurated data, we not only blindly sample from a large collection of movies and TV shows67 when constructing our pretraining dataset, but also perform ablation studies on the effect of genre68 distribution, the closest we have to taxonomy in the curated datasets, confirming that the quality of69 learned representations is agnostic with respect to such statistics.70
To the best of our knowledge, we are the first to solely rely on uncurated data and study the efficacy of71 self-supervised multi-modal representation learning from movies and TV shows. Despite meaningful72 domain gap between our pretraining data and the space of downstream tasks, we obtain representations73 which are very competitive with those learned from curated datasets. This is particularly important as74 we follow a much simpler modeling approach in comparison with the state-of-the art.75
2 Related Work76
Self-supervised learning techniques define pretext tasks, mostly inspired by the natural structures77 in the data, in order to generate supervisory signals for training. Despite the plethora of proposed78 pretext tasks in the literature, these approaches can be coarsely divided into two groups, namely79 pretext learning, and pretext-invariant methods. Approaches which fall in the former bucket, usually80 apply a form of transform, randomly drawn from a parametric family, to the input data then optimize81 for predicting the parameters of the chosen transformation. Predicting the relative position of image82 patches [9], solving jigsaw puzzles [33], estimating artificial rotations [13], colorization [50], context83 encoders learned through inpainting [38], and learning by counting scale and split invariant visual84 primitives [34], are among many methods which belong to this category. Similar techniques have85 been extended from images to videos [10, 21, 24, 25, 30, 46, 48, 49], where in addition to the86 spatial context, the temporal domain, and the arrow of time have been heavily exploited. In contrast,87
1the associated taxonomy has similarities with those of downstream benchmarks [3] 2alternatively referring to movies and TV shows
pretext-invariant methods [5, 7, 8, 15, 18, 17, 29, 35, 39, 44] are built on the concept of maximizing88 mutual information across augmented versions of a single instance, and are mostly formulated as89 contrastive learning. In other words, a pretext is used to generate different views of a single input90 for which the learning algorithm aims to maximize the intra-instance similarity, across variety of91 transformations. Our work falls within this category, however we function in a multi-modal realm92 employing both audio and video.93
Earlier works which harnessed audio and video for representation learning, have leveraged audio-94 visual temporal synchronization [22, 36], correspondence [4], and cross-modal clustering [3, 37]. The95 work by Patrick et al.[39] proposes a generalized data transformation in order to unify a variety of96 audio-visual self-supervised pretext tasks through a noise contrastive formulation. This work is close97 to ours in choice of objective function and data type, yet we employ no augmentation (except modality98 projection in the terminology of [39]), and solely focus on capitalizing the advantages of learning99 from long-form content. Morgado et al.[31] show that cross-modal discrimination is important for100 learning good audio and video representations, something which was also pointed out earlier in a101 clustering framework [3]. Beyond that, [31] generalizes the notion of instance-level positive and102 negative examples by exploring cross-modal agreement where multiple instances are grouped together103 as positives by measuring their similarity in both the video and audio feature spaces. While we also104 adopt a cross-modal noise contrastive estimation loss, we stick with the vanilla version, instance-level105 positive and negatives, and do not use any memory bank feature representations. Finally, Alayrac et106 al.[2] recently proposed a multi-modal versatile network capable of simultaneously learning from107 audio, video and text. Building on the intuition that different modalities are of different semantic108 granularity, audio and video are first compared in a fine-grained space while text is compared with109 the aforementioned modalities in a lower dimensional coarse-grained space. In our experiments, we110 compare with a variant of [2] where only audio and video modalities are utilized.111
3 Approach112
Notations and Architecture. Our pretraining dataset is denoted by X = {Xn|n ∈ [1 · · ·N ]},113 where Xn = {xn,m|m ∈ [1 · · ·Mn]} contains Mn non-overlapping audiovisual snippets which are114 temporally segmented from the duration of the nth long-form content in the dataset. Each snippet115 includes both audio and video modalities, formally xn,m = (an,m, vn,m), where an,m ∈ R1×P×Q116 and vn,m ∈ R3×T×H×W . T , H , and W denote the number of frames, height and width of the video,117 while P , andQ respectively stand for the number of mel filters, and audio frames. Video and audio are118 processed through 18-layers deep R(2+1)D [45] and ResNet [16] architectures, respectively referred119 to as f : R3 → Rdf and g : R1 → Rdg . Inspired by [7], we use projection heads, hf : Rdf → Rd and120 hg : Rdg → Rd, to map corresponding representations into a common d-dimensional space before121 computing the contrastive loss. The shallow architecture of hf and hg consists of two convolution122 layers, separated by Batch Normalization [19] and ReLU [32], followed by global average pooling.123 Once self-supervised pretraining finished, we discard the projection heads and fine-tune f and g for124 respective downstream tasks.125
Loss Function. With a slight abuse of notation3, B = {xi = (ai, vi)|i ∈ [1 · · ·B]} represents a126 minibatch of sizeB, where video and audio modalities associated with the ith sample, xi, are denoted127 by vi and ai. We use ziv = hf (f(vi)) and z i a = hg(g(ai)) to represent the associated embeddings128 generated by projection heads, and optimize the noise-contrastive loss [14] shown in 1 in order129 to maximize the symmetric joint probability between audio and video. For the ith element in the130 minibatch, (ziv, z i a) serves as the positive pair, while assuming negative pairs for both modalities,131 Ni = {(ziv, zja), (zjv, zia)|j ∈ [1 · · ·B], i 6= j} constitutes the set of negative pairs.132
L = − B∑ i=1 log
( e(z i v) ᵀ(zia)
e(z i v) ᵀ(zia) + ∑
(z′v,z ′ a)∈Ni
e(z ′ v) ᵀ(z′a)
) (1)
Most of the previous works [2, 31, 39] normalize the embeddings before computing the contrastive133 loss and employ a temperature hyper-parameter, often denoted by τ as in [2, 31], to control the134
3i enumerates elements in the minibatch
smoothness for the distribution of pairwise similarities. In contrast, we have chosen to operate in an135 unnormalized embedding space. Besides the obvious benefit of eliminating the need for tuning τ , we136 empirically show that such decision does not affect the quality of the learned representations.137
Sampling Policy. Contrastive loss function shown in Equation 1 is computed over B training138 instances, each in form of an audiovisual snippet. A naive sampling policy may ignore the fact that139 snippets comprising the pretraining dataset are in fact temporal segments that were trimmed from140 longer-form contents, i.e. movies and TV shows. Such an assumption treats our training data as141 independent and identically distributed random variables from ⋃N n=1 Xn, which constitutes the default142 sampling policy that is commonly used in the general deep learning literature. However, in reality,143 commonalities and correlations do exist along the temporal axis of a movie or TV show, things like144 audio mastering artifacts, frequent appearance of the main character’s face and voice, thematic music,145 repetitive scenes and mise-en-scène4, all of which contribute to breaking the previously discussed146 i.i.d assumption. This is even more pronounced when we deal with multiple episodes of the same TV147 show appearing in the pretraining dataset5. Note that, sampling from no video data is going to be i.i.d148 but in this case the temporal correlations extend for much longer given our entities are movies and TV149 shows. Thus, it is more accurate to think of X having multiple underlying domains, oriented towards150 exclusive properties which different long-form contents are characterized by. We hypothesize that151 during training, model gradually discovers such patterns of commonalities, which are not semantically152 valuable, and latches onto those to quickly minimize Equation 1 leading to poor generalization6. The153 reason being B N , hence for n ∼ U(1, N) and m 6= m′, P(xn,m ∈ B ∧ xn,m′ ∈ B) is negligible.154 In other words, the set of negative pairs in Equation 1 mainly includes pairs for which audio and155 video come from two different movies or TV shows, thus due to the aforementioned artifacts behave156 as easy negatives.157
In order to quantitatively measure our hypothesis, we define different distributions, shown in Equation158 2, over the space of audio-visual similarity. S+ indicates the space of correct matches, i.e. where159 audio and video correspond to the same snippet. S− indicates the space where audio and video do160 not correspond yet belong to the same movie or TV show. Finally, S 6= indicates the space in which161 audio and video are sampled from two distinct long-form content, hence naturally do not correspond.162
(zn,mv ) ᵀ(zn
′,m′
a ) ∼  S+, if n = n′ ∧m = m′ S−, if n = n′ ∧m 6= m′ S 6=, if n 6= n′ ∧ ∀(m,m′)
(2)
With that, and KL denoting Kullback–Leibler divergence, KL(S− ‖ S+) measures the expected163 difference between positive and negative pairs within the same movie or TV show. Ideally, this should164 increase as the training progresses, since the model gradually learns audio-video correspondence by165 minimizing Equation 1. Meanwhile, the i.i.d assumption suggests KL(S− ‖ S+) ' KL(S 6= ‖ S+)166 and KL(S− ‖ S 6=) ' 0, yet as we empirically illustrate later, KL(S− ‖ S+) < KL(S 6= ‖ S+) and167 KL(S− ‖ S 6=) is rather large, indicating that, upon convergence and on a held-out set, model has168 a harder time pushing apart negative pairs when audio and video come from the same underlying169 long-form content. Next, we explain how a simple alternative policy which samples k snippets170 from each long-form content effectively reduces both of the discrepancy measures, referring to171 KL(S− ‖ S 6=) and KL(S 6= ‖ S+)− KL(S− ‖ S+), while yielding better generalization on a range172 of downstream tasks.173
To ameliorate the aforementioned optimization challenge, we take a hierarchical approach. In174 particular, we first uniformly sample a long-form content, n ∼ U(1, N), and then draw k distinct175 snippets from Xn, creating {xn,m|m ∈Mn}, whereMn ⊂ [1 · · ·Mn] and |Mn| = k. This ensures176 that for xi ∈ B, Ni always includes 2k − 2 pairs sampled from the same movie or TV show to177 which xi belongs. By putting constraints onMn, specifically how temporally far from each other178 the k samples are drawn, we may go one step further and to some extent control the audiovisual179 similarity between snippets. This serves as an additional nob to tune for hard negative sampling.180 The intuition is that, the larger narrative of a professionally made movie or TV show is composed of181 shorter units called scene. Each scene comprises a complete event, action, or block of storytelling and182
4collectively referred to as content-exclusive artifacts 5alternatively think of it as a very long movie created by stitching different episodes together 6refer to supplemental material for illustrations of training loss
normally takes place in one location and deals with one action. That is, if our samples are temporally183 close, it is more likely for corresponding snippets to be highly correlated and/or look/sound alike.184 k ≤ max[Mn] − min[Mn] + 1 ≤ w ≤ Mn defines the bounds on our sampling policy, where w,185 standing for a sampling window, determines the farthest two out of k samples drawn from Xn can186 be. Accordingly, w = k represents the case where all k samples are temporally adjacent, hence187 the expected audiovisual similarity is maximized due to temporal continuity in content. We show188 that having such level of hard negatives, even with a small k, prevents proper training and results in189 performance degradation. On the other hand, w =Mn indicates random sampling where no temporal190 constraint is imposed onMn, thus samples are less likely to be drawn from adjacent time-stamps.191 In this case, expected audiovisual similarity (i.e. hardness of negative pairs) is mainly derived from192 global content-exclusive artifacts like, color palette, frequent appearance of the main character’s face193 and voice, repetitive scenes, and etc. The rest of the spectrum provides middle grounds where two194 samples drawn from Xn can at most be w + 1 snippets apart, something reminiscent of temporal195 locality. Our sampling policy can be easily implemented in a few lines of Python. Please refer to196 supplemental material for further details.197
4 Experiments198
4.1 Experimental Setup199
Datasets and Reproducibility. We use full-length movies and episodes of TV shows for self-200 supervised pretraining. Titles are randomly chosen from a large collection spanning over a variety of201 genres, namely Drama, Comedy, Action, Horror, Thriller, Sci-Fi and Romance. All audio is in English202 language. Our Movie dataset, consists of 3.6K films with an average duration of 105 minutes. Our TV203 dataset includes 9.2K episodes from a total of 581 shows with an average duration of 42 minutes per204 episode. Each of our datasets comprises 0.7 years worth of uncurated audiovisual content, which is205 significantly smaller than IG-Random [3] with variants at 5 and 21 years. Scaling up our pretraining206 datasets to volumes comparable to the IG-Random [3] while possible is non-trivial and demands207 dramatically larger compute resources for training, something which we currently cannot afford.208 Given that we cannot publicly release our dataset due to copyright reasons, we acknowledge that it is209 not possible for other research groups to fully reproduce our results. However, we intend to make210 available the pretrained models and hope that research community finds them, along with the other211 contributions of this work, of value whether within the context of self-supervised learning or adoption212 for various downstream tasks. We would like to emphasize that similar limitations have precedents213 in multiple earlier works including but not limited to [3, 12, 26, 43]. To evaluate the efficacy of214 self-supervised audio-visual representation learning from movies and TV shows, we follow recent215 works [3, 39, 31, 2] and benchmark UCF101[42] and HMDB51[23] for action recognition, along216 with ESC50[40] for audio classification. Results for the ablation studies are reported on the split-1 of217 the corresponding datasets. Following the standard protocol, we report the average performance over218 all splits when we are comparing with the state-of-the-art.219
Pretraining. Unless mentioned otherwise, we use video snippets with 16 frames at 5 fps. For data220 augmentation, we resize the shorter side to 190 pixels, then randomly crop them into 158×158 pixels.221 As for sound, we compute mel spectrogram from the raw audio at 48K sample rate using 96 mel222 filters and an FFT window of 2048, while the number of samples between successive frames is set to223 512. For data augmentation, we randomly drop out up to 25% from either temporal or frequency axis224 of the 2-D mel spectrogram image. Training uses a batch size of 512 and takes on average 42 hours225 on 8 NVIDIA A100 GPUs. The dimension of audio-video joint embedding space, d, is set to 512.226
Downstream Evaluation. For training on UCF101 [42] and HMDB51 [23], we use video clips that227 are 32 frames long at 10 fps. Unless mentioned otherwise, these clips are randomly chosen from the228 duration of the video instances. A scale jittering range of [181, 226] pixels is used and we randomly229 crop the video into 158 × 158 pixels. Furthermore, random horizontal flipping and color jittering230 are employed. During inference, 10 temporal clips are uniformly sampled where each is spatially231 cropped in 3 ways (left, center, right) resulting in a total of 30 views. We then average the model232 predictions across these 30 views and report top-1 classification accuracy. For training on ESC50233 [40], we use 3-seconds clips which are randomly chosen from the duration of the audio instances234 and apply time and frequency masking to spectrogram images for data augmentation. The maximum235 possible length of the mask is 50% of the corresponding axis. We do not use any scale jittering or236 random cropping on the spectrograms. During inference, 10 temporal clips are uniformly sampled237
and we average the model predictions across these 10 views and report top-1 classification accuracy.238 For further implementation details, please refer to the supplemental material.239
4.2 Ablation Study240
In the following, we discuss multiple ablation studies to241 assess our main hypothesis that, a hierarchical sampling242 policy, as described in Section 3, enables better repre-243 sentations to be learned by increasing the portion of hard244 negative pairs which the contrastive loss function observes.245 Here, pretraining uses 90% of either Movie or TV dataset,246 while the remaining 10% constitute a held-out validation247 set7 on which we report the discrepancy measures.248
Sample size (k) Figure 1a illustrates that compared to249 the baseline sampling denoted by k = 1, our approach250 (k > 1) effectively shrinks the gap between S− and S 6=251 when measured either directly or against S+. Its pattern252 of behavior also perfectly follows our earlier intuition (ref.253 Section 3). In particular, given a fixed minibatch budget,254 a larger k favors more training instances to be sampled255 from fewer number of long-form contents. That increases256 the portion of hard negative pairs, thus pushes the con-257 trastive loss to more aggressively separate mismatched258
audio-video pairs from the same movie, which leads model to maintain less of the content-exclusive259 artifacts in the embedding space. In the most extreme case, k = 64, all the training instances are260
7Given a TV show, either all or none of its episodes are included in the held-out set.
sampled from the same movie. From Table 1, we observe that different variants of our sampling261 policy, with no imposed temporal constraint, i.e. w = Mn, outperform the baseline on all three262 downstream tasks263
Sampling window (w) Smaller w forces samples264 that belong to same movie to be drawn from a shorter265 temporal window, hence growing the probability that266 they look/sound very much alike (i.e. harder neg-267 ative pairs). That is, it should further diminish the268 discrepancy measures. Figure 1b illustrates this be-269 havior where we gradually increase w while k = 16.270 However, from Table 1, it does not seem that tuning271 for w, i.e w 6= Mn, provides a meaningful gain on272 downstream tasks. This implies that commonalities273 which persist throughout the duration of a movie are274 sufficiently powerful signals to be exploited for gen-275 erating hard negatives. We hypothesize that different276 scenes both within and across different movies and277 TV shows are of variety of length, thus a fixed w278 is sub-optimal. Ideally, we should identify scene279 boundaries and and dynamically modify w during280 sampling, something which we leave for future iter-281 ations of this work.282
Temporally adjacent samples. Along the lines283 of previous observations, Figure 1c shows that in-284 deed drawing temporally adjacent snippets from285 the same long-form content, i.e. w = k, results286 in aggressively reducing the discrepancy measures.287 This behavior is agnostic with respect to k yet ex-288 acerbates as k grows. Note that, the contrastive289 loss is an instance-discrimination objective func-290 tion. Therefore, forcing it to distinguish between291 temporally adjacent snippets, that naturally sound292 and look extremely similar, leaves no choice for293 the model but to discard valuable semantic notions,294 which predictably leads to poor representations, also295 confirmed by result reported in Table 1.296
Movies vs. TV Shows. To confirm that our sampling policy behaves consistently across both movies297 and TV shows, Figure 1d illustrates the discrepancy measures computed on TV dataset. We observe298 similar effectiveness when using k and w as tuning nobs for reducing either KL(S− ‖ S 6=) or the299 gap between KL(S− ‖ S+) and KL(S 6= ‖ S+). Table 1 demonstrates that different variants of our300 approach significantly outperform the baseline, i.e. k = 1. We attribute the larger gains achieved301 when using TV instead of Movie dataset to the fact that content diversity is naturally lower when302 pretraining on TV shows since each one includes many episodes that all are characterized with the303 same content-exclusive artifacts.304
Color jitter. We have established so far that commonalities which persist throughout the duration305 of a long-form content, things likely associated with color pallet, frequent appearance of the main306 character’s face and voice, and repetitive scenes can be exploited for learning better representations.307 That is, one may naturally assume that employing data augmentation techniques like color jitter308 should be helpful since by distorting content-exclusive visual artifacts, color jitter is expected to309 reduce KL(S− ‖ S 6=). Figure 2 illustrates the effect of color jitter, where brightness, contrast, and310 saturation jitter values are chosen uniformly from [max(0, 1 − σ), 1 + σ]. We observe that color311 jitter reduces the discrepancy measures for the baseline but not as much as it can be obtained by our312 proposed sampling policy (k > 1), and even then according to Table 2 only yields a slight gain on313 downstream tasks.314
`2-normalized feature space. The common practice [2, 31, 39, 7] is to compute contrastive loss in315 an `2-normalized feature space, where according to [47] the temperature hyper-parameter, τ , controls316
the strength of penalties on hard negative samples. We explored this with two widely-used τ values.317 From Table 2, we observe that compared to operating in an unnormalized embedding space, adopting318 such design choice results in a large performance drop on HMDB51[23] while other downstream319 benchmarks see only negligible gains.320
Curated vs. Uncurated data. To the best of321 our knowledge, the only other uncurated dataset322 used for audio-visual self-supervised learning is IG-323 Random[3]8. Table 4 confirms that learning from un-324 curated movies and TV shows is extremely effective.325 Our results significantly exceed those of XDC[3]326 obtained on IG-Random16M despite using a sim-327 pler model and 7 times smaller volume of pretrain-328 ing data. Even in comparison to IG-Random65M329 with 30 times larger data, we obtain better perfor-330 mances on 2 out of 3 benchmarks. The most promis-331 ing of our findings though is how competitive our332 results are against XDC[3] when it is trained on333 variants of IG-Kinetics which are not only curated334 but also orders of magnitude larger. With all that,335 we confidently reject the notion that audio-visual336 self-supervised learning from uncurated data con-337 siderably lags behind utilizing large-scale curated338 datasets.339
Effect of genre. The distribution of genre among340 movies used in our pretraining is the closest we341 have to taxonomy in the curated datasets. So, it is342 worth examining the quality of our learned repre-343 sentations under various genre distributions. To do344 so, given a fixed pretraining budget (N =1.6K), we345 compare four different scenarios where movies used346 in the pretraining are distributed i) non-uniformly347 over all genres except Drama, and Comedies, ii)348 non-uniformly over Drama, and Comedies, iii) uni-349 formly over all genres, and iv) non-uniformly over350
all genres. Table 3 confirms that indeed there is very little difference between the aforementioned351 setups when it comes to transfer learning to the downstream tasks.352
4.3 Comparison with state-of-the-art353
Table 5 compares our proposed approach of learning from Movies and TV shows against the best354 performing audio-visual self-supervised learning methods. In general, our numbers are comparable355 with the best existing results reported in the literature, even with much less data and considerably356 simpler model/training procedure9. It is interesting that training on Movie dataset alone obtains357
8the data is not publicly available, and similarly the implementation to train XDC[3] 9supplemental material includes comparison of training costs
comparable performance to the cases where both TV and Movie datasets are used for pretraining.358 This further confirms the richness of the training data which movies and TV shows can provide359 to self-supervised learning problems. We also see that increasing k even beyond 8 gives further360 incremental gains on action recognition benchmarks.361
5 Conclusion362
Despite its amazing recent progress, state-of-the-art self-supervised learning still heavily relies on363 supervised, i.e. curated, large-scale datasets for pretraining. In this work, we have shown that364 pretraining solely on uncurated data in forms of movies and TV shows, even at a comparatively365 small scale, can give rise to representations which are capable of competing with the state-of-the-366 art of more complex architectures trained on larger curated datasets. This comes contrary to the367 current literature which tends to suggest that learning from uncurated data largely falls behind368 the use of curated alternatives. We intentionally made design decisions to keep our approach and369 training strategy as simple as possible to demonstrate that learning decently powerful audio-visual370 representations does not necessarily require gigantic data and compute resources. Through extensive371 set of experiments, our work establishes for the first time the efficacy of self-supervised learning of372 audio-visual representations from movies and TV shows.373
6 Broader impact374
Potential benefits. Our work shows that competitive multimodal representations can be learned375 from a comparatively small volume of uncurated data in the form of movies and TV shows. Besides376 minimizing any sort of human-involvement, which we believe must have already been paid an377 extra attention to in the literature, our work demonstrates that one does not require gigantic data and378 compute resources for effective self-supervised pretraining. Such results promise a more democratized379 research arena where smaller groups are not alienated due lack of sufficient compute resources. More380 importantly, lowering the compute requirements naturally reduces any environmental effects which381 training these models can potentially have.382
Potential risks. Any machine learning method is susceptible to the potential underlying biases in383 the data. This is more important for self-supervised methods that deal with huge volumes, often not384 evaluated by diverse group of humans for any fairness concerns. The same is generally true in our385 case which requires us to make sure that titles that are included in training are diverse and inclusive.386
References387 [1] S. Abu-El-Haija, N. Kothari, J. Lee, P. Natsev, G. Toderici, B. Varadarajan, and S. Vijayanarasimhan.388 Youtube-8m: A large-scale video classification benchmark. arXiv preprint arXiv:1609.08675, 2016.389
[2] J.-B. Alayrac, A. Recasens, R. Schneider, R. Arandjelović, J. Ramapuram, J. De Fauw, L. Smaira,390 S. Dieleman, and A. Zisserman. Self-Supervised MultiModal Versatile Networks. In NeurIPS, 2020.391
[3] H. Alwassel, D. Mahajan, B. Korbar, L. Torresani, B. Ghanem, and D. Tran. Self-supervised learning by392 cross-modal audio-video clustering. In Advances in Neural Information Processing Systems, volume 33,393 pages 9758–9770, 2020.394
[4] R. Arandjelovic and A. Zisserman. Look, listen and learn. In Proceedings of the IEEE International395 Conference on Computer Vision, pages 609–617, 2017.396
[5] P. Bachman, R. D. Hjelm, and W. Buchwalter. Learning representations by maximizing mutual information397 across views. In Advances in Neural Information Processing Systems, pages 15535–15545, 2019.398
[6] J. Carreira and A. Zisserman. Quo vadis, action recognition? a new model and the kinetics dataset. In399 proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 6299–6308,400 2017.401
[7] T. Chen, S. Kornblith, M. Norouzi, and G. Hinton. A simple framework for contrastive learning of visual402 representations. arXiv preprint arXiv:2002.05709, 2020.403
[8] T. Chen, S. Kornblith, K. Swersky, M. Norouzi, and G. Hinton. Big self-supervised models are strong404 semi-supervised learners. arXiv preprint arXiv:2006.10029, 2020.405
[9] C. Doersch, A. Gupta, and A. A. Efros. Unsupervised visual representation learning by context prediction.406 In Proceedings of the IEEE international conference on computer vision, pages 1422–1430, 2015.407
[10] B. Fernando, H. Bilen, E. Gavves, and S. Gould. Self-supervised video representation learning with408 odd-one-out networks. In Proceedings of the IEEE conference on computer vision and pattern recognition,409 pages 3636–3645, 2017.410
[11] J. F. Gemmeke, D. P. Ellis, D. Freedman, A. Jansen, W. Lawrence, R. C. Moore, M. Plakal, and M. Ritter.411 Audio set: An ontology and human-labeled dataset for audio events. In 2017 IEEE International Conference412 on Acoustics, Speech and Signal Processing (ICASSP), pages 776–780. IEEE, 2017.413
[12] D. Ghadiyaram, D. Tran, and D. Mahajan. Large-scale weakly-supervised pre-training for video action414 recognition. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages415 12046–12055, 2019.416
[13] S. Gidaris, P. Singh, and N. Komodakis. Unsupervised representation learning by predicting image417 rotations. arXiv preprint arXiv:1803.07728, 2018.418
[14] M. Gutmann and A. Hyvärinen. Noise-contrastive estimation: A new estimation principle for unnormalized419 statistical models. In Proceedings of the Thirteenth International Conference on Artificial Intelligence and420 Statistics, pages 297–304, 2010.421
[15] K. He, H. Fan, Y. Wu, S. Xie, and R. Girshick. Momentum contrast for unsupervised visual representation422 learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages423 9729–9738, 2020.424
[16] K. He, X. Zhang, S. Ren, and J. Sun. Deep residual learning for image recognition. In Proceedings of the425 IEEE conference on computer vision and pattern recognition, pages 770–778, 2016.426
[17] O. J. Hénaff, A. Srinivas, J. De Fauw, A. Razavi, C. Doersch, S. Eslami, and A. v. d. Oord. Data-efficient427 image recognition with contrastive predictive coding. arXiv preprint arXiv:1905.09272, 2019.428
[18] R. D. Hjelm, A. Fedorov, S. Lavoie-Marchildon, K. Grewal, P. Bachman, A. Trischler, and Y. Bengio.429 Learning deep representations by mutual information estimation and maximization. arXiv preprint430 arXiv:1808.06670, 2018.431
[19] S. Ioffe and C. Szegedy. Batch normalization: Accelerating deep network training by reducing internal432 covariate shift. arXiv preprint arXiv:1502.03167, 2015.433
[20] H. Kataoka, T. Wakamiya, K. Hara, and Y. Satoh. Would mega-scale datasets further enhance spatiotempo-434 ral 3d cnns? arXiv preprint arXiv:2004.04968, 2020.435
[21] D. Kim, D. Cho, and I. S. Kweon. Self-supervised video representation learning with space-time cubic436 puzzles. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 33, pages 8545–8552,437 2019.438
[22] B. Korbar, D. Tran, and L. Torresani. Cooperative learning of audio and video models from self-supervised439 synchronization. In Advances in Neural Information Processing Systems, pages 7763–7774, 2018.440
[23] H. Kuehne, H. Jhuang, E. Garrote, T. Poggio, and T. Serre. Hmdb: a large video database for human441 motion recognition. In 2011 International Conference on Computer Vision, pages 2556–2563. IEEE, 2011.442
[24] Z. Lai and W. Xie. Self-supervised learning for video correspondence flow. arXiv preprint443 arXiv:1905.00875, 2019.444
[25] H.-Y. Lee, J.-B. Huang, M. Singh, and M.-H. Yang. Unsupervised representation learning by sorting445 sequences. In Proceedings of the IEEE International Conference on Computer Vision, pages 667–676,446 2017.447
[26] D. Mahajan, R. Girshick, V. Ramanathan, K. He, M. Paluri, Y. Li, A. Bharambe, and L. van der Maaten.448 Exploring the limits of weakly supervised pretraining. In Proceedings of the European Conference on449 Computer Vision (ECCV), pages 181–196, 2018.450
[27] A. Miech, J.-B. Alayrac, L. Smaira, I. Laptev, J. Sivic, and A. Zisserman. End-to-end learning of visual451 representations from uncurated instructional videos. In Proceedings of the IEEE/CVF Conference on452 Computer Vision and Pattern Recognition, pages 9879–9889, 2020.453
[28] A. Miech, D. Zhukov, J.-B. Alayrac, M. Tapaswi, I. Laptev, and J. Sivic. Howto100m: Learning a454 text-video embedding by watching hundred million narrated video clips. In Proceedings of the IEEE455 international conference on computer vision, pages 2630–2640, 2019.456
[29] I. Misra and L. v. d. Maaten. Self-supervised learning of pretext-invariant representations. In Proceedings457 of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 6707–6717, 2020.458
[30] I. Misra, C. L. Zitnick, and M. Hebert. Shuffle and learn: unsupervised learning using temporal order459 verification. In European Conference on Computer Vision, pages 527–544. Springer, 2016.460
[31] P. Morgado, N. Vasconcelos, and I. Misra. Audio-visual instance discrimination with cross-modal461 agreement. arXiv preprint arXiv:2004.12943, 2020.462
[32] V. Nair and G. E. Hinton. Rectified linear units improve restricted boltzmann machines. In ICML, 2010.463
[33] M. Noroozi and P. Favaro. Unsupervised learning of visual representations by solving jigsaw puzzles. In464 European Conference on Computer Vision, pages 69–84. Springer, 2016.465
[34] M. Noroozi, H. Pirsiavash, and P. Favaro. Representation learning by learning to count. In Proceedings of466 the IEEE International Conference on Computer Vision, pages 5898–5906, 2017.467
[35] A. v. d. Oord, Y. Li, and O. Vinyals. Representation learning with contrastive predictive coding. arXiv468 preprint arXiv:1807.03748, 2018.469
[36] A. Owens and A. A. Efros. Audio-visual scene analysis with self-supervised multisensory features. In470 Proceedings of the European Conference on Computer Vision (ECCV), pages 631–648, 2018.471
[37] A. Owens, J. Wu, J. H. McDermott, W. T. Freeman, and A. Torralba. Ambient sound provides supervision472 for visual learning. In European conference on computer vision, pages 801–816. Springer, 2016.473
[38] D. Pathak, P. Krahenbuhl, J. Donahue, T. Darrell, and A. A. Efros. Context encoders: Feature learning474 by inpainting. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages475 2536–2544, 2016.476
[39] M. Patrick, Y. M. Asano, R. Fong, J. F. Henriques, G. Zweig, and A. Vedaldi. Multi-modal self-supervision477 from generalized data transformations. arXiv preprint arXiv:2003.04298, 2020.478
[40] K. J. Piczak. Esc: Dataset for environmental sound classification. In Proceedings of the 23rd ACM479 international conference on Multimedia, pages 1015–1018, 2015.480
[41] S. Purushwalkam and A. Gupta. Demystifying contrastive self-supervised learning: Invariances, augmen-481 tations and dataset biases. In H. Larochelle, M. Ranzato, R. Hadsell, M. F. Balcan, and H. Lin, editors,482 Advances in Neural Information Processing Systems, volume 33, pages 3407–3418, 2020.483
[42] K. Soomro, A. R. Zamir, and M. Shah. Ucf101: A dataset of 101 human actions classes from videos in the484 wild. arXiv preprint arXiv:1212.0402, 2012.485
[43] C. Sun, A. Shrivastava, S. Singh, and A. Gupta. Revisiting unreasonable effectiveness of data in deep486 learning era. In Proceedings of the IEEE international conference on computer vision, pages 843–852,487 2017.488
[44] Y. Tian, D. Krishnan, and P. Isola. Contrastive multiview coding. arXiv preprint arXiv:1906.05849, 2019.489
[45] D. Tran, H. Wang, L. Torresani, J. Ray, Y. LeCun, and M. Paluri. A closer look at spatiotemporal490 convolutions for action recognition. In Proceedings of the IEEE conference on Computer Vision and491 Pattern Recognition, pages 6450–6459, 2018.492
[46] C. Vondrick, A. Shrivastava, A. Fathi, S. Guadarrama, and K. Murphy. Tracking emerges by colorizing493 videos. In Proceedings of the European conference on computer vision (ECCV), pages 391–408, 2018.494
[47] F. Wang and H. Liu. Understanding the behaviour of contrastive loss. arXiv preprint arXiv:2012.09740,495 2020.496
[48] J. Wang, J. Jiao, L. Bao, S. He, Y. Liu, and W. Liu. Self-supervised spatio-temporal representation learning497 for videos by predicting motion and appearance statistics. In Proceedings of the IEEE Conference on498 Computer Vision and Pattern Recognition, pages 4006–4015, 2019.499
[49] D. Xu, J. Xiao, Z. Zhao, J. Shao, D. Xie, and Y. Zhuang. Self-supervised spatiotemporal learning via video500 clip order prediction. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition,501 pages 10334–10343, 2019.502
[50] R. Zhang, P. Isola, and A. A. Efros. Colorful image colorization. In European conference on computer503 vision, pages 649–666. Springer, 2016.504
Checklist505
1. For all authors...506
(a) Do the main claims made in the abstract and introduction accurately reflect the paper’s507 contributions and scope? [Yes]508
(b) Did you describe the limitations of your work? [Yes] Specifically, training data509 being proprietary creates concerns around reproducibility, which has precedence in the510 literature as mentioned in the paper. We address that partially by planning to publicly511 release pretrained models.512
(c) Did you discuss any potential negative societal impacts of your work? [Yes]513 (d) Have you read the ethics review guidelines and ensured that your paper conforms to514 them? [Yes]515
2. If you are including theoretical results...516
(a) Did you state the full set of assumptions of all theoretical results? [N/A]517 (b) Did you include complete proofs of all theoretical results? [N/A]518
3. If you ran experiments...519
(a) Did you include the code, data, and instructions needed to reproduce the main exper-520 imental results (either in the supplemental material or as a URL)? [No] The data is521 proprietary. However, we have provided implementation of the proposed method in522 supplemental material and aim to publicly release the pretrained models.523
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they524 were chosen)? [Yes] They are all discussed in detail either in the main submission or in525 supplemental material.526
(c) Did you report error bars (e.g., with respect to the random seed after running experi-527 ments multiple times)? [Yes] We only observed meaningful differences after running528 experiments multiple times, for ESC50[40] downstream experiments. Corresponding529 standard errors are reported in supplemental material.530
(d) Did you include the total amount of compute and the type of resources used (e.g., type531 of GPUs, internal cluster, or cloud provider)? [Yes] Please refer to Section 4.1.532
4. If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...533 (a) If your work uses existing assets, did you cite the creators? [N/A]534 (b) Did you mention the license of the assets? [N/A]535 (c) Did you include any new assets either in the supplemental material or as a URL? [N/A]536
537
(d) Did you discuss whether and how consent was obtained from people whose data you’re538 using/curating? [N/A]539
(e) Did you discuss whether the data you are using/curating contains personally identifiable540 information or offensive content? [N/A]541
5. If you used crowdsourcing or conducted research with human subjects...542 (a) Did you include the full text of instructions given to participants and screenshots, if543 applicable? [N/A]544 (b) Did you describe any potential participant risks, with links to Institutional Review545 Board (IRB) approvals, if applicable? [N/A]546 (c) Did you include the estimated hourly wage paid to participants and the total amount547 spent on participant compensation? [N/A]548

Review summaries:
This paper performs self-supervised, audio-visual pretraining of deep CNNs on movies and TV shows. The downstream tasks investigated are action classification in videos (HMDB51 and UCF101 datasets) and environmental audio classification (ESC50 dataset). The paper shows that pre-training on 0.7-1.4 years worth of TV and movies results in downstream task performance that is competitive with, but lags slightly behind, the current state-of-the-art approaches that pre-train on datasets like HowTo100M, AudioSet, and IG-Kinetics65M. 

The paper works on self-supervised audio-visual representation learning. It shows uncurated movies and TV shows are good sources for learning audio-visual representation. The proposed model targets on short video clips like 3-second clips, and learned representation is evaluated on action classification and audio classification. 

The paper proposes a method to learn self-supervised audio-visual representations from long (order of half to couple hours) uncurated videos namely from movies and TV shows. A new sampling approach is presented to ensure batches are created while taking into consideration the effect of re-occurring audio-visual patterns (main character or background scenes). Picking multiple clips from the same long video that are spaced by some minimum distance is found to be effective as compared against random sampling. Pretrained models are evaluated on standard action recognition tasks: UCF101 and HMDB, and audio classification on ESC50.

The task is to learn audio-visual representations from uncurated data such as movies and TV videos. Unlike previous works that show learning from uncurated data is ineffective, the authors show good performance from learning from these videos. They propose hierachical sampling policy, with negatives being sampled from both within and across different sources. The model is trained on the dataset that the authors collected, which consists of thousands of hours. The performance on various downstream tasks (action recognition, sound classification) is on par compared to using much larger uncurated datasets such as IG-Random.