Multi-modal Self-Supervision from Generalized Data Transformations
Authors: Mandela Patrick
Yuki Asano
Polina Kuznetsova
Ruth Fong
Joao F. Henriques
Geoffrey Zweig
Andrea Vedaldi
Venue:  (2021)
Keywords: video representation learning
multi-modal learning
self-supervised learning
audio-visual learning
noise-contrastive learning

Abstract: In the image domain, excellent representation can be learned by inducing invariance to content-preserving transformations, such as image distortions. In this paper, we show that, for videos, the answer is more complex, and that better results can be obtained by accounting for the interplay between invariance, distinctiveness, multiple modalities and time.  We introduce Generalized Data Transformations (GDTs) as a way to capture this interplay. GDTs reduce most previous self-supervised approaches to a choice of data transformations, even when this was not the case in the original formulations. They also allow to choose whether the representation should be invariant or distinctive w.r.t. each effect and tell which combinations are valid, thus allowing us to explore the space of combinations systematically. We show in this manner that being invariant to certain transformations and distinctive to others is critical to learning effective video representations, improving the state-of-the-art by a large margin, and even surpassing supervised pretraining. We demonstrate results on a variety of downstream video and audio classification and retrieval tasks, on datasets such as HMDB-51, UCF-101, DCASE2014, ESC-50 and VGG-Sound. In particular, we achieve new state-of-the-art accuracies of 72.8% on HMDB-51 and 95.2% on UCF-101.

Under review as a conference paper at ICLR 2021
MULTI-MODAL SELF-SUPERVISION FROM GENERALIZED DATA TRANSFORMATIONS
Anonymous authors Paper under double-blind review
ABSTRACT
In the image domain, excellent representations can be learned by inducing invari- ance to content-preserving transformations, such as image distortions. In this pa- per, we show that, for videos, the answer is more complex, and that better results can be obtained by accounting for the interplay between invariance, distinctive- ness, multiple modalities, and time. We introduce Generalized Data Transforma- tions (GDTs) as a way to capture this interplay. GDTs reduce most previous self- supervised approaches to a choice of data transformations, even when this was not the case in the original formulations. They also allow to choose whether the representation should invariant or distinctive w.r.t. each effect and tell which combinations valid, thus allowing to explore the space of combinations systematically, We show in this manner that being invariant to certain transforma- tions and distinctive others is critical to learning effective video representations, improving the ofte--of-hheatt by a large margin, and even surpassing supervised pretraining. We demonstrate results a variety of downstream video and au- dio classification and retrieval tasks, on datasets such as HMDB-51, UCF-101, DCASE2014, ESC-50 and VGG-Sound. In particular, we achieve new state-of- the-art accuracies of 72.8% on HMDB-51 and 95.2% on UCF-101.
INTRODUCTION
Recent works such as PIRL Miira van der Maaten 2020). MoCo (He al. 2019 and Sim- CLR (Tian et al. 2019) have shown that it is possible to pre-train state-of-the-art image represen- tations without the use of any manually-provide
labels. Furthermore, many of these approaches use variants of noise contrastive learning (Gutmann & Hyvärinen| 2010). Their idea is to learn representation that is invariant to transformations that leave the meaning of an image unchanged (e.g. geometric distortion cropping) and distinctive changes that likely to alter its meaning (e.g. replacing an image with another chosen at random).
An analysis of such works shows that dominant factor for performance is the choice of the transfor- mations applied to the data. So far, authors have explored ad-hoc combinations of several transfor- mations (e.g. random scale changes, crops, or contrast changes). Videos further allow to leverage the time dimension and multiple modalities. For example, AAaandjelovic & Zisserman 2017); Owens) (2016) learn representations by matching visual and audio streams, that have a coherent appearance and sound. Their formulation is similar to noise contrastive ones, but does not quite follow the pattern of expressing the loss in terms of data transformations. Oth- ers (Chung & Zisserman 2016 Korbar et al.) 20188) Owens & Efros, (208) depart further from stan- dard contrastive schemes by learning representations that can tell whether visual and audio streams are in sync not; the difference here is that the representation is encouraged to be distinctive rather than invariant to a time shift.
proxy for objects
Overall, it seems that finding optimal noise contrastive formulation for videos will require com- bining several transformations while accounting for time and multiple modalities, and understanding how invariance and distinctiveness should relate the transformations. However, the ad-hoo nature of these choices in previous contributions make a systematic exploration of this space rather difficult.
In this paper, we propose solution to this problem by introducing the Generalized Data Trans- formations (GDT; fig. [ framework. GDTs reduce most previous methods, contrastive or not, to noise contrastive formulation that is expressed in terms of data transformations only, making it

---

Under review as a conference paper at ICLR 2021
A: Hierarchical transformations
B: Learning Hypotheses
C: Cross-modal NCE learning
video
video
asdic
audio
72
74
75
T6
TE
Wiu
92
T2
95
T3
T4
0
T4
1
0
T2
0
1
T7
PM. mh
95 T7
O: Repnisive
1,, Igmored
Repuisive
Attractive
Attraetive
Fig. 1: Schematic overview of our framework. A: Hierarchical sampling process of general- ized transformations T = tM o 1 for the multi-modal training study case. B: Subset of the c" contrast matrix which shows which pairs are repelling (0) and attracting (1) (see text for details). C: With generalized data transformations (GDT), the network learns meaningful em- bedding via learning desirable invariances and distinctiveness to transformations (realigned here for clarity) across modalities and time. The embedding is learned via noise contrastive estimation against clips of other source videos. Illustrational videos taken from YouTube (Google 2020).
simpler to systematically explore the space of possible combinations. This is true in particular for multi-modal data, where separating different modalities can also be seen as transformation of an input video. The formalism also shows which combinations of different transformations are valid and how to enumerate them. It also clarifies how invariance and distinctiveness to different ef- fects can be incorporated in the formulation and when doing so leads to a valid learning objective. These two aspects allows the search space of potentially optimal transformations be significantly constrained, making it amenable to grid-search or more sophisticated methods such as Bayesian optimisation.
By using GDTs, we make several findings. First, we find that using our framework, most previous pretext representation learning tasks can be formulated in a noise-contrastive manner, unifying pre- viously distinct domains. Second, we show that just learning representations that are invariant to more and more transformations is not optimal, at least when it comes video data; instead, bal- ancing invariance to certain factors with distinctiveness to others performs best. Third, we find that by investigating what to be variant to can lead to large gains in downstream performances, for both visual and audio tasks.
With this, we are able to set the new state of the art in audio-visual representation learning, with both small and large video pretraining datasets on a variety of visual and audio downstream tasks. In particular, we achieve 95.2% and 72.8% on the standardized UCF-101 and HMDB-51 action recognition benchmarks.
2 RELATED WORK
Self-supervised learning from images and videos. A variety of pretext tasks have been proposed to Iearn representations from unlabelled images. Some tasks leverage the spatial context in images (Doersch al.) 2015; Noroozi & Favaro (016)) to train CNNs, while others create pscudo clas- sification labels via artificial rotations (Gidaris et al. 2018), or clustering features (Asano et al. 2020b Caron al. 2018 2019 Gidaris al. 2020 Ji et al. 2018). Colorization (Zhang al. 2016 2017). inpainting (Pathak et al. 2016), solving jigsaw puzzles (Noroozi et al. 2017). as well the contrastive methods detailed below, have been proposed for self-supervised image represen- tation learning. Some of the tasks that use the space dimension of images have been extended to the space-paaeeeeee dimensions of videos by crafting equivalent tasks. These include jigsaw puzzles (Kim et al.) 2019), and predicting rotations QUing & Tian 2018) or future frames (Han et al. 2019).
Other tasks leverage the temporal dimension of videos to learn representations by predicting shuf-
fled frames (Misra et al.) 2016). the direction of time (Wei et al.) 2018). motion (Wang et al. 2019).
clip and sequence order (Lee et al. 2017 Xu et al. 2201)), and playback speed (Benaim et al. 2020 Cho et al. 2020 Fermanno et al. (0177). These pretext-tasks can be framed as GDTs.
Multi-modal learning. Videos, unlike images, are a rich source of variety of modalities such speech, audio, and optical flow, and their correlation can be used as supervisory signal. This
2

---

Under review as a conference paper at ICLR 2021
idea has been present early as 1993 de Sa 1994). Only recently, however, has multi-
learning been used to successfully learn effective representations by leveraging the natural corre-
spondence (Alwassel et al.] [020 Arandjelovic & Zisserman 2017 AAsan et al. 2020a;
et al.
2016 Morgado et al. 2020] Owens et al.) 2016) and synchronization (Chung & Zisserman 2016]
Korbar et al. 2018: Owens & Efros 2018) between the audio and visual streams. A number of
recent papers have leveraged speech as a weak supervisory signal to train video representations (Li & Wang 2020 Miech et al. 2020; Nagrani al.
2020; Sun et al., 2019a b) and recently Alayracc et al. (2020), which uses speech, audio and video. Other works incorporate optical flow and other modalities (Han et al., 2020; Liu et al., 2019 Piergiovanni et al.) 2020 Zhao et al.) 2019) to learn representations. In (Tian et al. 2201)), representations are learned with different views such as differ- ent color channels or modalities) to induce invariances. In contrast, our work analyses multi-modal transformations and examines their utility when used as an invariant or variant learning signal.
Noise Contrastive Loss. Noise contrastive losses (Gutmann Hyvärinen 2010 Hadsell et al, 2006) measure the similarity between sample pairs in representational space and are at the core of several recent works on unsupervised feature learning. It has been shown to yield good performance
for learning image (Chen al., 2020b; He et al., 2019 Hénaff et al. 2019
Hjelm et al. 2019 Li
iet al.
2020 Misra & van der Maaten 2020: Oord et al. [0118] Tian et al.
2019; 2020 Wu et al. Morgado et al., 2020
2018 and video (Han et al.)
2019 Li & Wang 2020 Miech et al. 2020
Sohn, 2016 [Sun et al.] 2019a) representations, and circumvents the need to explicitly specify what
information needs to be discarded via a designed task.
We leverage the noise contrastive loss as a learning framework to encourage the network to learn desired invariance and distinctiveness to data transformations. The GDT framework can be used to combine and extend many of these cues, contrastive or not, in a single noise contrastive formulation.
3 METHOD
A data representation is a function S X' + RD mapping data points æ to vectors [xx) Repre- sentations are useful because they help to solve tasks such as image classification. Based on the nature of the data and the task. we often know priori some of the invariances that the represen- tation should possess (for example, rotating an image usually does not change its class). We can capture those by means of the confrast functior[ 22) = f(z)))(((s)) where c(x1, = 1 means that f is invariant to substituting 22 for I1. while c(21, .22) = 0 means that f is distinc- tive to this change. Any partial knowledge of the contrast e can be used as a cue to learn f. but e is not arbitrary: in order for c to be valid, the expression c(z1, x22 = 1 must be an equiva- lence relation on X,, i.e. be reflexive o(r, x) = 1, symmetric 2(x1,52) = c(#2, 21 ) and transitive c(21, (()) c(22, = > (()).T3) = 1. This is justified in Appendix A. and will be important in establishing which particular learning formulations are valid and which are not.
We introduce next our Generalized Data Transformations (GDTs) framework by generalizing two typical formulations: the first is analogous to 'standard' methods such as MoCo (He et al. 2019) and SimCLR (Chen et al. 2020b) and the second tackles multi-modal data.
Standard contrastive formulation. Recall that the goal is to learn a function ff that is compatible with a known contrast c, in the sense explained above. In order to learn f. we require positive (c(21, 222) = 1) and negative (c(11, = 0) example pairs (21, by sampling æ1 from data source and then by setting 222 g(x1) random transformation of the first sample, where E G is called a data augmentation (e.g. image rotation). We also generate negative pairs by sampling 2 and 22 independently.
We generate positive pairs
It is convenient to express these concepts via transformations only. To this end, let D (f1...., .N). X be a collection of N i.i.d. training data samples. A Generalized Data Transfor- mation (GDT) T:; x + Z is mapping that acts on the set of training samples D to produce a new sample 2 TD. Note that the GDT is applied to the entire training set, so that sampling itself can be seen as a transformation. In the simplest case, 2 = X and a GDT T = (i.g) extracts the sample corresponding to a certain index and applies an augmentation g X - X' to i.e. TD = gzx)).
We use the symbol 5 to denote the Kronecker delta.
3

---

Under review as a conference paper at ICLR 2021
Usually, we want the function f to be distinctive to the choice of sample but invariant to its aug- mentation. This is captured by setting the contrast c(T, T' to c((i.g). (d',g)) = ó'=r. Given a batch T (fi....., Tx) of K GDTs, contrastive loss (Chen et al., 2020b, Gutmann & Hyvärinen 2010 Oord et al. 2018 Tian et al.)
then optimize pairwise-wiighee version of the noise-
2019
Wu et al. 2018), the GDT-NCE loss:
exp ((TTD),
((I:T)
(T,T'))(TTTT log ETTET w(T,T") exp (J(TD) S(T' Σ TT
Here, the scalar P is temperature parameter and the weights w(T, T") are set to TT in order to discount contrasting identical transformations, which would result in a weak learning signal. Minimizing eq. pulls together vectors f (TDD) and D(T if c(T,T")
1 and pushes them 0, similar to a margin loss, but with a better handling of hard negatives (Chen al. 2020b Khosla et al. 2020: Tian al. 2019 When using single modality, T 7 and positive pairs are computed from two differently augmented versions.
apart if c(T,
Multi-modal contrastive formulation. We now further extend GDTs handle multi-mode data. In this case, several papers (Arandjelovic & Zisserman 2017 Aytar et al., 2016 Korbar et al.) 2018 Owens et al., 2016; Wei et al.) (018) have suggested learn from the correlation between modalities, albeit usually not in contss-cnt manner. In order to encode this with a GDT, we introduce modality projection transformations m .M. For example, video
(v, a) has visual component and an audio component and we we have two projections M = {ma, m.) extracting respectively the visual m.
and audio ma (x) = a signals. We can plug this directly in eq. Q by considering GDTs T (i, m) and setting TD m(x;), learning representation which IS distinctive to the choice of input video, but invariant to the choice of modality
General case. Existing noise contrastive formulations learn representations that are invariant to an ad-hoc selection of transformations. We show here how to use GDTs to build systematically new valid combinations of transformations while choosing whether to encode invariance distinctive- ness to each factor. Together with the fact that all components, including data sampling and modality projection, interpreted transformations, this results in a powerful approach to explore vast space of possible formulations systematically, especially for the case of video data with its several dimensions.
In order to do so, note that to write the contrastive loss eq. we only require: the contrast c(T, T'), the weight wT,, T") and way of sampling the transformations T in the batch. Assuming that each generalized transformation T = tm o .... o t is a sequence of M transformations fm. we start by defining the contrast for individual factors as:
1,
if we hypothesize invariance,
cllm (mm)
8 =
if we hypothesize distinctiveness.
The overall contrast is then c(T, II c(tm: f.m)) In this way, each contrast c(tm: t'm) is an cquivalence relation and so is c(T,, T") (see Appendix [I. making it valid in the sense discussed above. We also assume that w=T, T*) unless otherwise stated.
Next, we require a way of sampling transformations T in the batch. Note that each batch must con- tain transformations that can be meaningfully contrasted, forming mix of invariant and distinctive pairs, so they cannot be sampled independently at random. Furthermore, based on the definition above, single 'distinctive' factor in eq. (2) such that Im * l',m implies that c(T, T") = 0. Thus, the batch must contain several transformations that have equal distinctive factors in order to generate a useful learning signal.
A simple way to satisfy these constraints is to use a hierarchical sampling scheme (fig. D First, we sample K₁ instances of transformation t;; then, for each sample t1, we sample K₂ instances
NNote that, differently from the previous section, we have now defined e on transformations T rather than on samples æ directly. In Appendix A. we show that this is acceptable provided that c(T, = also defines an cquivalence relation.
? We think of eq. [1] as softmax cross-entropy loss for classification problem where the classes the equivalence classes 7/c of transformations.
*For this, as f must accept either visual audio signal as input, consider pair of representations (fe: fa), one for each modality.
4

---

Under review as a conference paper at ICLR 2021
of transformation tz and so on, obtaining batch of K I Km transformations T. In this . x Km+1 transformations that share the same first m factors (t1 , t. While other schemes
manner, the batch contains exactly KM X X...
possible, in Appendix [A.2.1] [ we show that this is sufficient to express a large variety of self-supervised learning cues that have been proposed in the literature. In the rest of the manuscript, however, we focus on audio-visual data.
f'm
3.1 EXPLORING CONTRASTIVE AUDIO- VISUAL SELF-SUPERVISION
Within multi-modal settings, video representation learning audio-visual data is particularly well suited for exploring the GDT framework. Especially compared to still images, the space of transfor- mations is much larger in videos due to the additional time dimension and modality. It is therefore an ideal domain to explore how GDTs can be used to limit and explore the space possible trans- formations and their quality as learning signal when used variances or invariances. In order to apply our framework to audio-visual data, we start by specifying how transformations are sampled by using the hierarchical scheme introduced above (see also Figure [T.. We consider in particular GDTs of the type T = (i, T, m, combining the following transformations. The first component i selects video in the dataset. We sample K; > 2 indices/videos and assume distinctiveness, so that c(i. i') = ò: The second component contrasts different temporal shifts. We sample K, 2 different values of delay т uniformly at random, extracting Is clip 24y starting at time T. For this contrast, we will test the distinctiveness and invariance hypotheses. The third component m contrasts modalities, projecting the video Far to either its visual or audio component m(x;+)- We assume invariance c(m, m') 1 and always sample two such transformations my and m. to extract both modalities, so Km 2. The fourth and final component g applies a spatial and aural augmentation TD = g(m(x++)), )). also normalizing the data. We assume invariance c(g, ((g.9") = 1 and pick Kg = 1. The transformation g comprises a pair of augmentations (ge- 9a), where (v) ex- tracts a fixed-size tensor by resizing to fixed resolution random spatial crop of the input video v, and 9a (a) extracts a spectrogram representation of the audio signal followed by SpecAugment (Park) et al. 2019) with frequency and time masking. These choices lead to K iransformations I' in the batch T.
Km Kg 4K,
Testing invariance and distinctiveness hypotheses. The transformations given above combine cues that were partly explored in prior work, contrastive and non-contrastive. For example, Korbar al. 2018) (not goise-contrastivg
learns detect temporal shifts across modalities. With our for- mulation, we can test whether distinctiveness or invariance to shifts is preferable, simply by setting c(T,T')= = 1 or c(T,7)) = ó (this illustrated in fig. D.. We can also set w(T,T') = 00 for & 7 to ignore comparisons that involve different temporal shifts. We also test distinctiveness and invari- ance time reversal (Wei et al., 2018), which has not previously been explored cross-modally, or contrastively. This is given by transformation ri flips the time dimension of its input tensor. We chose these transformations, time reversal and time shift, because videos, unlike images, have temporal dimension and we hypothesize that these signals are very discriminative for representation learning.
R {ro, ri }. where To is the identity and
Ignoring comparisons. Another degree of freedom is the choice of weighting function w(T, T'). Empirically, we found that cross-modal supervision is a much stronger signal than hithii-modality supervision, so if T and T' slice the same modality, we set (T, T') 0 (see Appendix for ablation).
Understanding combinations. Finally, one may ask what is the effect of combining several dif- ferent transformations in learning the representation f. A first answer is the rule given in eq. (2) to combine individual contrasts c(4m, ('m) in a consistent manner. Because of this rule, to a first approximation, ʃ possesses the union of the invariances and distinctivenesses of the individual fac- tors. To obtain more accurate answer, however, one should also account for the details of the batch sampling scheme and of the choice of weighing function w. This can be done by consulting the diagrams given in fig. [ by: (1) choosing a pair of transformations T; and Tj. (2) checking the value in the table (where stands for invariance, 0 for distinctiveness and for ignoring), and (3) looking up the composition of T; and T, in the tree to find out the sub-transformations that differ between them as the source of invarianceeddistinttivnsss
5

---

Under review as a conference paper at ICLR 2021
4 EXPERIMENTS
We compare self-supervised methods on pretraining audio-visual representations. Quality is as- sessed based how well the pretrained representation transfers to other (supervised) downstream tasks. We first study the model in order to determine the best learning transformations and setup. Then, we use the latter to train for longer and compare them to the state of the art.
Self-supervised pretraining. For pretraining, we consider the standard audio-visual pretraining datasets, Kinetics-400 (Kay al., (0177) and AudioSet (Gemmeke al. (017), and additionally, the recently released, VGGi-Suund dataset (Chen et al., 2020a). Finally, we also explore how our algorithm scales even larger, less-curated datasets and train on IG65M (Ghadiyaram et al.) 2019) as done in XDC (Alwassel et al. 2020).
Our method learns pair of representations f (fo. for visual and audio information respectively and we refer to Appendix A.6 for architectural details.
Table 1: Learning hypothesis ab- lation. Results on action classifi- cation performance shown for finetuning accuracy (Acc) and frozen action retrieval (recall@5). GDT can leverage signals from both invariance and stronger yariance trans- formation signals, that sole data- sample (DS) variance misses.
HMDB-51 is
Downstream tasks. To assess the visual representation fer we consider standard action recognition benchmark datasets, UCF-101 (Soomro et al.) 2012) and HMDB- 51 (Kuehne et al.) 101 We test the performance our pretrained models on the tasks of finetuning the pretrained representation, conducting few-shot learning and video action retrieval. To assess the audío representation Sa. we train a linear classifier on frozen features for the common ESC-50 (Piczak 2015) and DCASE2014 (Stowell al.) benchmarks and finetune for VGG-Sound (Chen et al. 2020a). The full details are given in the Appendix.
DS TR TS Mod. Acc r@5
SimCLR: DS-variance only
2015)
(a)
47.1
32.5
(b)
i
39.5 31.9
(c)
i
46.9
34.5
4.1 ANALYSIS
(d)
i
i
46.6
33.4
OF GENERALIZED TRANSFORMATIONS
GDT: 1-variance
(e)
v
AV
56.9
49.3
In this section, we conduct an extensive study on each parameter of the GDT transformation studied here, T (i, T, m,g), and evaluate the performance by finetuning our network on the UCF-101 and HMDB-51 action recog- nition benchmarks.
(N
i
AV
56.1
49.7
(g)
i
AV
57.2
45.2
(h)
i
AV
56.6
44.8
GDT: 2-varinnces
(i)
i
AV
57.5
46.8
Sample distinctiveness and invariances. First, we ex- periment with extending SimCLR to video data, as shown (ableT[a)-(h) This an important base case is the standard approach followed by all recent supervupe methods (Chen al. 2020b He et al. 2019 Wu al.) 2018).
(j)
Y v i
AV
57.0
46.2
(k)
V
AV
58.0
50.2
(I) v AV
58.2 50.2
GDT: 3-variances
(m) v
60.0 47.8
For this, consider GDT of the type = (i, m,, g) described above and set K; = 768 (the largest we can fit in our setup), K,, (only visual modality) and K2 and only pick single time shift K, = 1. We also set all transformation components to invariance (c(tm t'm) = 1) except the first that does sample selection. Comparing row (a) to (b-d), we find that adding invariances to time-shift (TS) and time-reversal (TR) consistently degrades the performance compared to the baseline in (a).
GDT variances and invariances Our framework allows fine-grained and expressive control of which invariance and distinctiveness are learned. To demonstrate this flexibility, we first experiment with having single audio-visude (AV) invariance transformation, in this case data-sampling (DS), iee. T (i, r, m, m.g). We find immediately an improvement in finetuning and retrieval performance compared to the SimCLR baselines, due the added audio-visual invariance. Second, also find that adding invariances to TR and TS does not yield consistent benefits, showing that invariance to these transformations is not useful signal for learning.
In rows (i-l), we explore the effect of being variant to two transformations, which is unique to our method. We find that: (1) explicitly encoding variance improves representation performance for the TS and TR transformations (58.0 and 58.2 vs 56.9). (2) Ignoring (-) the other transformation as

---

Under review as a conference paper at ICLR 2021
Table 2: Retrieval and Few Shot Learning. Re- trieval accuracy in (%) via nearest neighbors and few shot learning accuracy (%) via training a linear SVM on fixed representations.
Table Audio classification. Down- stream task accuracies on standard audio classification benchmarks.
HMDB
UCF
Method
Acc% DC ESC
20
Retrieval Few-shet
Random
3.0 4.5 2.3 6.8
ConvRBM (Sailor et al.
2017
86.5
3DRtt (Jing & Tian 2018
17.0
AVTS
Korbar el al. 20188)
94
82.3
DMC
Hu ct al. 2019
82.6
GDT (ours)
13.4 20.8
26.3 49.4
XDC
Awassel ct al. (2020
95
84.8
ClipOrder (Xu et al.
2019
7.6 48.8 14.1 51.1
AVID
Morgado ct al 2020
96
89.2
VCP (Cho et al. 2020)
7.6 53.6 18.6 53.5
GDT (ours)
98 88.5
GDT (ours)
25.4 75.0 57.4 88.1
Human (Piczak 2015)
81.3
opposed to forcefully being invariant to it works better (58.2 vs 57.0 and 58.0 vs 57.5). Finally, row (m), shows the (DS, TR, TS)-variance case, yields the best performance when finetuned and improves upon the initial SimCLR baseline by more than 12% in accuracy and more than 15% in retrieval @5 performance. (DS, TR, TS) Compared to row (I), we find that using three variances compared to two does give boost in finetuning performance (58.2 vs 60.0), but there is a slight decrease in retrieval performance (50.2 vs 47.8). We hypothesize that this decrease in retrieval might be due to the 3-variance model becoming more tailored to the pretraining dataset and, while still generalizeable (which the finetuning evaluation tests), its frozen features have a slightly higher domain gap compared to the downstream dataset.
Intuition While we only analyse subset of possible transformations for video data, we neverthe- less find consistent signals: While both time-reversal and time-shift could function as a meaningful invariance transformation to provide the model with more difficult positives a-priori, we find that using them instead to force variances consistently works better. One explanation for this might be that there useful signal in being distinct these transformations. E.g., for time-reversal, opening door carries different semanties from from closing one, and for time-shift, the model might profit from being able to differentiate between an athlete running vs an athlete landing in a sandpit, which could be both in the same video. These findings noteworthy, they contradict results from the image self self-supervised learning domain, where learning pretext-invariance can lead to more transfer- able representations (Misra & van der Maaten 2020). This is likely due the fact that time shift and reversal are useful signals that both require learning strong video representations to pick up on. If instead invariance learned against these, the "free" information that we have from construction is discarded and performance degrades. Instead, GDT allows one to leverage these strong signals for learning robust representations.
4.2 COMPARISON TO THE STATE OF THE ART
Given of our best learning setups from Sec. 4.1 (row (I)), we train for longer and compare feature representations to the state of the art in common visual and aural downstream benchmarks.
Downstream visual benchmarks.
For video retrieval we report recall at 1, 5, 20 retrieved samples for split-1 of the HMDB-51 and UCF-101 datasets in table (the results for recall 10 and 50 are provided in the Appendix). Using our model trained on Kinetics-400, GDTsignificantl beats all other self-supervised methods by margin of over 35% for both datasets.
For few-shot classification, as shown in table 2 UCF-101 by more than 10% on average for each shot with our Kinetics-400 pretrained model.
we significantly beat the RotNer3D baseline on
For video action recognition, we finetune our GDT pretrained network for UCF-101 and HMDB-51 video classification, and compare against state-of-the-art self-supervised methods in table 4 When constrained to pretraining on the Kinetics datasets, we find that our GDT pretrained model achieves very good results, similar to Morgado et (2020) (developed concurrently to our own work). When

---

Under review as a conference paper at ICLR 2021
Table State-of-the-art on video action recognition. Self- and fully-supervisedll trained methods on UCF-101 and HMDB-51 benchmarks. We follow the standard protocol and report the average top-1 accuracy over the official splits for finetuning the whole network. Methods with i use video titles supervision, with use ASR generated text. See A.3 A.3 an extended version including recent/concurrent works.
Method
Architecture
Pretraining
Top-1 Acc%
HMDB
UCF
Full supervision (Alwassel et al.
2020
R(2+1)D-18 R(2+1)D-18
Kinetics-400
65.1
94.2
Full supervision (ours)
Kinetics-400
70.4
95.0
Using Kinetics
AoT Wei et al.
2018
T-CAM
Kinetics-400
79.4
XDC Alwassel et al.
2020
R(2+1)D-18
Kinetics-400
52.6
86.8
AV Sync*RorNer (X iao et al.
2020
Fast
Kinetics-400
54.6
87.0
AVTS Korbar et 2018)
MC3-18
Kinetics-400
56.9
85.8
CPD
L1 & Wang 2020
3D-Resnet50
Kinetics-400
57.7
88.7
AVID Morgado al. 2020
R(2+1)D-18
Kinetics-400
60.8
87.5
GDT (ours)
R(2+1)D-18
Kinetics-400
60.0
89.3
Using other datasets
MIL-NCE (Miech et al. 2020
S3D
HowTo 10M00M
61.0
91.3
AVTS
Korbar al. 2018)
MC3-18
AudioSet
61.6
89.0
XDC Alwassel et al. 2020)
R(2+1)D-18
AudioSet
63.7
93.0
AVID
Morgado al.
2020
R(2+1)D-18
AudioSet
64.7
91.5
ELo
Piergiovanni et (020)
R(2+1)D-50x3
Youtube-2M
67.4
93.8
XDC
Alwassel et al.
2020)
R(2+1)D-18
IG65M
68.9
95.5
GDT (ours)
R(2+1)D-18
VGGSound
61.9
89.4
GDT (ours)
R(2+1)D-18
AudioSet
66.1
92.5
GDT (ours)
R(2+1)D-18
IG65M
72.8
95.2
constrained pretraining the AudioSet (Gemmeke et al. 2017) dataset, also find state-of- the-art performance among all self-supervised methods, particularly on HMDB-51.
We get similar performance XDC on UCF-101. Lastly, show the scalability and flexibility of our GDT framework by pretraining on the IG65M dataset (Ghadiyaram et al. (2019). With this, our visual feature representation sets larly by a margin of >> 4% on the HMDB-51 dataset. On UCF-101, we set similar state-of-the-at performance with XDC. Along with XDC, beat the Kinetics supervised pretraining baseline using the same architecture and finetuning protocol.
state of the art among all self-supervised methods, particu-
For audio classification art performance among all self-supervised methods on both DCASE2014 (DC) and ESC-50 (ESC), and also surpass super- vised performance on VGG-Sound with 54.8% mAP and 97.5%
find that we achieve state-of-the-
Table 5: VGG-Sound.
Audio classification metrics af- ter full-finttuning.
AUC (see Tab. 5
Method mAP AUC
CONCLUSION
Supervised 51.6 96.8
GDT (ours) 54.8 97.5 2.77
We introduced the framework of Generalized Data Transforma- tions (GDTss, which allows one capture, in single noise-contrastive objective, cues used in sev- eral prior contrastive and non-contrastive learning formulations, as well easily incorporate new ones. The framework shows how new meaningful combinations of transformations can be obtained, encoding valuable invariance and distinctiveness that we want our representations to learn. Follow- ing this methodology, achieved state-of-the. art results for self-supervised pretraining standard downstream video action recognition benchmarks, even surpassing supervised pretraining. Overall, method significantly increases the expressiveness of contrastive learning for self-supervision, making it a flexible tool for many multi-modal settings, where large pool of transformations exist. and optimal combination is sought.
8

---

Under review as a conference paper at ICLR 2021
REFERENCES
Lean-Baptistt Alayrac, Adrià Recasens, Rosalia Schneider, Relja Arandjelović, Jason Ramapuram, Jeffrey De Fauw, Lucas Smaira. Sander Dieleman, and Andrew Zisserman. Self-supervised mul- timodal versatile networks. In NeurIPS, 2020.
Humam Alwassel, Bruno Korbar, Dhruv Mahajan, Lorenzo Torresani, Bernard Ghanem, and Du Tran. Self-supervised learning by cross-modal audio-video clustering. In NeurlPS, 2020.
Relja Arandjelovic and Andrew Zisserman. Look, listen and leamn In ICCV, 2017.
Yuki M Asano, Mandela Patrick, Christian Rupprecht, and Andrea Vedaldi. Labelling unlabelled videos from scratch with multi-modal self-supervision. NeurIPS, 2020a.
Yuki Asano, Christian Rupprecht, and Andrea Vedaldi. Self-labelling via simultaneous clustering and representation learning. In ICLR, 2020b.
Yusuf Aytar, Carl Vondrick. and Antonio Torralba. Soundnet: Learning sound representations from unlabeled video. NeurIPS, 2016.
Philip Bachman, R Devon Hjelm, and William Buchwalter. Learning representations by maximizing mutual information across views. In NeurIPS, 2019.
Sagie Benaim, Ariel Ephrat, Oran Lang, Inbar Mosseri, William T. Freeman, Michael Rubinstein, Michal Irani, and Tali Dekel. Speednet: Learning the speediness in videos. In CVPR, 2020.
Uta Buchler, Biagio Brattoli, and Bjorn Ommer. Improving spatiotemporal self-supervision by deep reinforcement learning. In ECCV, 2018.
Mathilde Caron, Piotr Bojanowski, Armand Joulin, and Matthijs Douze. Deep clustering for unsu- pervised learning of visual features. In ECCV, 2018
Mathilde Caron, Piotr Bojanowski, Julien Mairal, and Armand Joulin. Unsupervised pre-training of image features on non-curated data. In ICCV, 2019.
Honglie Chen, Weidi Xie, Andrea Vedaldi, and Andrew Zisserman. Vggsound: A large-scale audio- visual dataset. ICASSP, 2020a.
Ting Chen, Simon Korbblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. arXiv preprint 20Xiv:2002.05709, 2020b.
Hyeon Cho, Tachoon Kim, Hyung Jin Chang, and Wonjun Hwang. Self-supervised spatio-
temporal representation learning using variable playback speed prediction. arXiv preprint 2xXiv:2033.0692. 2020.
Joon Son Chung and Andrew Zisserman. Out of time: automated lip syne the wild. Workshop on Multi-view Lip-reading, ACCV, 2016.
Virginia R. de Sa. Learning classification with unlabeled data. In NeurIPS, 1994.
Ali Diba, Vivek Sharma, Luc Van Gool, and Rainer Stiefelhagen. Dynamonet: Dynamic action and motion network. In ICCV, 2019.
Carl Doersch, Abhinav Gupta, and Alexei A Efros. Unsupervised visual representation learning by context prediction. In ICCV, 2015.
Basura Fernando, Hakan Bilen, Efstratios Gavves, and Stephen Gould. Self-supervised video rep- resentation learning with odd-one-out networks. In Proc. CVPR, 2017.
Chuang Gan, Boqing Gong, Kun Liu, Hao Su, and Leonidas J Guibas. Geometry guided convolu- tional neural networks for self-supervised video representation learning. In CVPR, 2019.
Jort F. Gemmeke, Daniel P. W. Ellis, Dylan Freedman, Aren Jansen, Wade Lawrence, R. Channing Moore, Manoj Plakal, and Marvin Ritter. Audio set: An ontology and human-labeled dataset for audio events. In /CASSP, 2017.

---

Under review as a conference paper at ICLR 2021
Deepti Ghadiyaram, Du Tran, and Dhruv Mahajan. Large-scale weakly-supervised pre-training for video action recognition. In CVPR, 2019.
Spyros Gidaris, Praveer Singh, and Nikos Komodakis. Unsupervised representation learning by predicting image rotations. ICLR, 2018.
Spyros Gidaris, Andrei Bursuc, Nikos Komodakis, Patrick Pérez, and Matthieu Cord. Learning representations by predicting bags of visual words. In CVPR, 2020.
Google. Youtube.
TELDs://youtu.bb//krrowooo
attps://youtu.a
3muSpprk3U 2020.
Priya Goyal, Piotr Dollár, Ross Girshick, Pieter Noordhuis, Lukasz Wesolowski, Aapo Kyrola, An- drew Tulloch, Yangqing Jia, and Kaiming He. Accurate, large minibatch SGD: training imagenet in hour. arXiv preprint arXiv:1706.02677 2017.
Michael Gutmann and Aapo Hyvärinen. Noise-contrastive estimation: A new estimation principle for unnormalized statistical models. In AISTATS, 2010.
Raia Hadsell, Sumit Chopra, and Yann LeCun. Dimensionality reduction by learning invariant mapping. In CVPR, 2006.
Tengda Han, Weidi Xie, and Andrew Zisserman. Video representation learning by dense predictive coding. In ICCV Workshops. 2019.
Tengda Han, Weidi Xie, and Andrew Zisserman. Self-supervised co-training for video representa- tion learning. In NeurlPS, 2020.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. CVPR, 2016.
Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. Momentum contrast for unsupervised visual representation learning, 2019.
Olivier J Hénaff, Ali Razavi, Carl Doersch, SM Eslami, and Aaron den Oord. Data-efficient image recognition with contrastive predictive coding. arXiv preprint arXiv:1905.090: 1905.0927,, 2019.
R Devon Hjelm, Alex Fedorov, Samuel Lavoie-Marchildon, Karan Grewal, Phil Bachman, Adam Trischler, and Yoshua Bengio. Learning deep representations by mutual information estimation and maximization. In ICLR, 2019.
Di Hu, Feiping Nie. and Xuelong Li. Deep multimodal clustering for unsupervised audiovisual learning. CVPR, 2019.
Xu Ji, JoNo F. Henriques, and Andrea Vedaldi. Invariant information clustering for unsupervised image classification and segmentation, 2018.
Longlong Jing and Yingli Tian. Self-supervised spatiotemporal feature learning by video geometric transformations. arXiv preprint 1.. 1387, 2018.
Will Kay, Joao Carreira, Karen Simonyan, Brian Zhang, Chloe Hillier, Sudheendra Vijaya- narasimhan, Fabio Viola, Tim Green, Trevor Back, Paul Natsev, et al. The kinetics human action video dataset. arXiv preprint arXiv:1705.06950, 2017.
Prannay Khosla, Piotr Teterwak, Chen Wang, Aaron Sarna, Yonglong Tian, Phillip Isola, Aaron Maschinot, Ce Liu, and Dilip Krishnan. Supervised contrastive learning. arXiv preprint arXiv:2004. 1162, 2020.
Dahun Kim, Donghyeon Cho, and In So Kweon. Self-supervised video representation learning with space-time cubic puzzles. In AAAI, 2019.
Diederik Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In ICLR, 2015.
Bruno Korbar, Du Tran, and Lorenzo Torresani. Cooperative learning of audio and video models from self-supervised synchronization. In NeurlPS, 2018.
10

---

Under review as a conference paper at ICLR 2021
H. Kuehne, H. Jhuang, E. Garrote, T. Poggio, and T. Serre. HMDB: large video database for human motion recognition. In ICCV, 2011a.
Hildegard Kuehne, Hueihan Jhuang, Estíbaliz Garrote, Tomaso Poggio, and Thomas Serre. HMDB: large video database for human motion recognition. In ICCV, 201 lb.
Hsin-Ying Lee, Jia-Bin Huang, Maneesh Singh, and Ming-Hsuan Yang. Unsupervised representa- tion learning by sorting sequences. In ICCV, 2017.
Junnan Li, Pan Zhou, Caiming Xiong, Richard Socher, and Steven CH Hoi. Prototypical contrastive learning of unsupervised representations. arXiv preprint #rXi::2005004666 2020.
Tianhao Li and Limin Wang. Learning spatiotemporal features via video and text pair discrimina- tion. arXiv preprini arXiv:2001.05691, 2020.
Yang Liu, Samuel Albanie, Arsha Nagrani, and Andrew Zisserman. Use what you have: Video retrieval using representations from collaborative experts. In BMVC, 2019.
Dezhao Luo, Chang Liu, Yu Zhou, Dongbao Yang, Can Ma, Qixiang Ye, and Weiping Wang. Video cloze procedure for self-supervised spatio-temporal learning. In AAAI, 2020.
Zelun Luo, Boya Peng, De-An Huang, Alexandre Alahi, and Li Fei-Fei. Unsupervised learning of long-term motion dynamics for vidcos. In CVPR, 2017.
Antoine Miech, Jean- -aptiste Alayrac, Lucas Smaira, Ivan Laptev, Josef Sivic, and Andrew Zisser- man. End-to-end learning of visual representations from uncurated instructional videos. arXivees, abs/1912.06430,, 2019.
Antoine Miech, Jean-Baphen Baptiste Alayrac, Lucas Smaira, Ivan Laptev, Josef Sivic, and Andrew Zis- serman. End-to-end learning of visual representations from uncurated instructional videos. In CVPR, 2020.
Tomas Mikolov, Kai Chen, Greg Corrado, and Jeffrey Dean. Efficient estimation of word represen- tations in vector space. arXiv preprint arXiv:1301.378. 2013.
Ishan Misra and Laurens van der Maaten. Self-supervised learning of pretext-invariant representa- tions. In CVPR, 2020.
Ishan Misra, C Lawrence Zitnick, and Martial Hebert. Shuffle and learn: unsupervised learning using temporal order verification. In ECCV, 2016.
Pedro Morgado, Nuno Vasconcelos. and Ishan Misra. Audio visual instance discrimination with cross-modal agreement. arXiv preprint arXiv:2004. 12944 2020.
Arsha Nagrani, Chen Sun, David Ross, Rahul Sukthankar, Cordelia Schmid, and Andrew Zisserman. Speech2action: Cross-modal supervision for action recognition. In CVPR, 2020.
Mehdi Noroozi and Paolo Favaro. Unsupervised learning of visual representatiins by solving jigsaw puzzles. In ECCV, 2016.
Mehdi Noroozi, Hamed Pirsiavash, and Paolo Favaro. Representation learning by learning to count. In ICCV, 2017.
Aaron van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predic- tive coding. arXiv preprint arXiv:1807.037488 2018.
Andrew Owens and Alexei A Efros. Audio-visual scene analysis with self-supervised multisensory features. In ECCV, 2018.
Andrew Owens, Jiajun Wu, Josh McDermott, William T Freeman, and Antonio Torralba. Ambient sound provides supervision for visual learning. In ECCV, 2016.
Daniel S. Park, William Chan, Yu Zhang, Chung-Chem Chiu, Barret Zoph, Ekin D. Cubuk, and Quoe Le. Specaugment: A simple data augmentation method for automatic speech recognition. In INTERSPEECH, 2019.
11

---

Under review as a conference paper at ICLR 2021
Deepak Pathak, Philipp Krahenbuhl, Jeff Donahue, Trevor Darrell, and Alexei A Efros. Context encoders: Feature learning by inpainting. In CVPR, 2016.
Jeffrey Pennington, Richard Socher, and Christopher D Manning. Glove: Global vectors for word representation. In Proceedings of the 2014 4 conference empirical methods in natural language processing (EMNLP), pp. 1532-1543, 2014.
Karol J. Piczak. Esc: Dataset for environmental sound classification. In ACM Multimedia, 2015.
AJ Piergiovanni, Anelia Angelova, and Michael S. Ryoo. Evolving losses for unsupervised video representation learning. In CVPR, 2020.
Hardik B. Sailor, Dharmesh M Agrawal, and Hemant A Patil. Unsupervised filterbank learning using convolutional restricted boltzmann machine for environmental sound classification. In IN- TERSPEECH, 2017.
Nawid Sayed, Biagio Brattoli, and Bjöm Ommer. Cross and learn: Cross-modal self-supervision. German Conference on Pattern Recognition, 2018.
Kihyuk Sohn. Improved deep metric learning with multi-class n-pair loss objective. In NeurlPS, 2016.
Khurram Soomro, Amir Roshan Zamir, and Mubarak Shah. UCFI01: A dataset of 101 human action classes from videos in the wild. In CRCV-TR-12-01, 2012.
D. Stowell, D. Giannoulis, E. Benetos, M. Lagrange, and M. D. Plumbley. Detection and classifica- tion of acoustic scenes and events. IEEE Transactions on Multimedia, 2015.
Chen Sun, Fabien Baradel, Kevin Murphy, and Cordelia Schmid, Contrastive bidirectional trans- former for temporal representation learning. arXiv preprint arXiv: 05743, 2019a.
Chen Sun, Austin Myers, Carl Vondrick, Kevin Murphy, and Cordelia Schmid. Videobert: A joint model for video and language representation learning. In ICCV, 2019b.
Yonglong Tian, Dilip Krishnan, and Phillip Isola. Contrastive multiview coding. arXiv preprint arXiv:1906.05849, 2019.
Yonglong Tian, Chen Sun, Ben Poole, Dilip Krishnan, Cordelia Schmid, and Phillip Isola. What makes for good views for contrastive learning. arXiv preprint arXiv:2005. /0244 2020.
Du Tran, Heng Wang, Lorenzo Torresani, Jamie Ray, Yann LeCun, and Manohar Paluri. closer look spatiotemporal convolutions for action recognition. In CVPR, 2018.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention is all you need. In NeurlPS, 2017.
Carl Vondrick, Hamed Pirsiavash, and Antonio Torralba. Generating videos with scene dynamics. In NeurIPS, 2016
Jiangliu Wang, Jianbo Jiao, Linchao Bao, Shengfeng He, Yunhui Liu, and Wei Liu. Self-supervised spatio-temporal representation learning for videos by predicting motion and appearance statistics. In CVPR, 2019.
Donglai Wei, Joseph J Lim, Andrew Zisserman, and William Freeman. Learning and using the arrow of time. In CVPR, 2018
Jason Wei and Kai Zou. EDA: Easy data augmentation techniques for boosting performance on text classification tasks. Proceedings of the 2019 Conference on Empiricul Methods Natural Language Processing and the 9th International Joint Conference Natural Language Process- ing (EMNLP-IJCNLP), pp. 6382-6388 6388, Hong Kong, China, Nov 2019. Association for Com- putational Linguistics. doi: 10.18653/v1/D199 1670. URL https:/ / www acl.eeew org/
anthology/D19-1677
Zhirong Wu, Yuanjun Xiong, Stella X. Yu, and Dahua Lin. Unsupervised feature learning via non- parametric instance discrimination. In CVPR, 2018.
12

---

Under review as a conference paper at ICLR 2021
Fanyi Xiao, Yong Jae Lee, Kristen Grauman, Jitendra Malik, and Christoph Feichtenhofer. Audio- visual slowfast networks for video recognition. arXiv preprint arXiv:2001.08740, 2020.
Dejing Xu, Jun Xiao, Zhou Zhao, Jian Shao, Di Xie, and Yueting Zhuang. Self-supervised spa- tiotemporal learning via video clip order prediction. In CVPR, 2019.
Richard Zhang, Phillip Isola, and Alexei Efros. Colorful image colorization. In Proc. ECCV, 2016.
Richard Zhang, Phillip Isola, and Alexci A Efros. Split-brain autoencoders: Unsupervised learning by cross-channel prediction. CVPR, 2017.
Hang Zhao, Chuang Gan, Wei-Chiu Ma, and Antonio Torralba. The sound of motions. In ICCV, 2019.
13

---

Under review as a conference paper at ICLR 2021
A
APPENDIX
A.1 THEORY
Full knowledge of the contrast function e conly specifies the level sets of the representation
Lemma 1. The contrast c(1I, x22) = ((1))=(((=))) defines f = fo up to an injection c X/ + y, X/f the projection on the quotient.
where f f is the quotient space and f
Proof. This is a well known fact in elementary algebra. Recall that the quotient X/ʃ is just the collection subsets X C X' where f(z) is constant. is easy to see that this is a partition of X. Hence, we can define the map f: X + f(x) where x is any element of X (this is consistent since f(x) has, by definition, only one value over X). Furthermore, if x + X far E X: f(x') f()) is the projection of x to its equivalence class X, we have f(x) f(((x))
O
Lemma 2. 32) 1 is an equivalence relation if. and only there exists a function such that c(x1, 22) = ((()))
Proof. If c(21, = 1 defines an equivalence relation on X, then such function is given by the projection the quotient f X' 8/c V. On the other hand, setting c(21, f(e))=(((2)) = 1 for any given function f is obviously reflexive, symmetric and transitive because the equality f(x1) = f(x2) is.
The following lemma suggests that defining contrast c(T, on transformations instead of data samples is usually acceptable.
Lemma 3. If Fc(,TT = defines an equivalence relation on GDTs, and ifTD TD' =T= (i.e. different transformations output different samples), then setting c(T DD c(T, T') defines part of an admissible sample contrast function.
Proof. If If =T= x' T are obtained from some transformations and TV, then these must be unique by assumption. Thus, setting c(x,, at), = e(T, is well posed. Reflectivity, symmetry and transitivity are then inherited from the latter.
Lemma 4. Let cllmm (/m)) be reflexive, symmetric and transitive. Their product c(T, T') IT c(tms ('m) = has then the same properties.
Proof. The reflexive and symmetric properties are obviously inherited. For the transitive property, note that c(T, T') = 1 if, and only if, Vm : c(tm, t'm)) = 1. Then consider:
cTT,T) = = (T") =1
Vm : e(tm, t'm)) = c(t'm. (hm) =
Vm c(tm, (
cTT,T") =
A.2 GENERALITY OF GDT
Here, we show that our GDT formulation can encapsulate and unify other self-supervised works in the literature. We break down it into two sections:
Mapping contrastive to GDT contrastive Recently, number of papers have presented con- trastive formulations for image representation learning such as, NPID (Wu et al. 2018). PIRL (Misra & van der Maaten 2020). MoCo (He et al. 2019) and SimCLR (Chen et al. 2020b). These meth- ods are all essentially built on what we have introduced as the "data-sampling transformation" T = (i, g). that samples image with index i and applies augmentation g. For NPID, MoCo and SimCLR, the main objective is to solely be distinctive to the image index. hence K = K,K, = B (i.e. the batchsize B) for NPID, due the use of memorybank and K K2K, 2B for SimCLR and MoCo. For PIRL, one additional transformation to be invariant to is added. For ex- ample, the case of rotation, the PIRL encodes sample-die distinciiinnnt to the non-rotated inputs
14

---

Under review as a conference paper at ICLR 2021
K KKKKK, = B in the memorybank, while the rotated examples are used for constructing both invariance to the original imputs, as well as sample distinctiveness.
Non-contrastive to GDT contrastive reduction. In non-contrastive self- ceff-supervieed formulations, one trains ((x) = y/ to regress y from z, where y is some "pretext" task label. These labels can be obtained from the data, e.g. arrow of time (Wei al. 2018), rotation (Gidaris al.. [018 Ding & Tian 2018), shuffled frames (Misra et al., 2016), Jigsaw configurations (Kim et al.
2019 Noroozi
al. 2017 , or playback speed (Benaim al. 2020 Cho al. 2020).
We can reduce these pretext tasks to GDTs in two ways. The first 'trivial' reduction amounts to interpreting the supervision y as an additional (seudo-modality Consider for example RotNet; in this case, the label should record the amount of rotation applied to the input image. We can achieve this effect by starting from data 2 (x, 0) where x is an image and 0 rotation angle. We then sample transformation f-- (rotation) and define its action as (z) (z), (O)) where (0) r simply the rotation angle applied and (x) the rotated image. We consider modality slicing transformations ma (z) = and mz) =
To form batch, we sample GDTs of the type T (i, lr m), where i is sampled at random, for each i, 4 is exhaustively sampled a set of four rotations (0, 90, 180, 270 degrees) and, for each rotation ly- m is also exhaustively sampled, for a total of K,K, = SK, transformations in the batch. We define cTT,T" c((i,t,, m), (i', frr, m')) 8,=p' (note that we do not learn to distinguish different images; GDTs allow us to express this case naturally as well). We define (T,TT) = di=r'mmma so that images treated independently in the loss and always compare a pseudo modality (rotated image) with the other (label). Finally, the network fr) = e, E {0, 1 operating on the label medaaiyy trivially encodes the latter as a I-hot vector. Then we see that the noise-contrastive loss reduces
ΣΣ log exp(J(G(1ss)).e) exp((((((()
which is nearly exactly the same as softmax loss for predicting the rotation class applied to an image.
There are other reductions as well, which capture the spirit if not the letter of a training signal. For instance, in RotNet, we may ask if two images are rotated by the same amount. This is an interesting example we do not wish to be distinctive to which image sample is taken, only to which rotation is applied. This can also be captured as a GDT because the sampling process itself is a transformation. In this case, the set of negatives will be the images rotated by a different amount, while the positive example will be an image rotated by the same amount.
Thus, pretext task-originating transformations that have even been explored yet can be put into our framework and, as we show in this paper, be naturally combined with other transformations leading to even stronger representations.
A.2.1 POTENTIAL APPLICATION TO VIXT-VIDEO LEARNING
While foeus on audio-visual representation learning due to the multitude of potentially in- teresting learning signals, it is also possible to apply our framework to other multi-modal set- tings, such video-text. Instead of ResNet-9 as audio encoder, text-encoder such as word- embeddings (Mikolov et al. 2013 Pennington et al. 2014) with an MLP or a transformer (Vaswani al. 2017) can be used for encoding the textual inputs and can train with cross-modk NCE loss as done currently for audio-visual representation learning in our GDT framework. While the vi- sual transformations can be kept as described in the paper, we can use transformations for text, such as sentence shuffling (Wei & Zou 2019). or random word swaps (Wei & Zou| 2019). Moreover, un- like prior works in the literature (Alayrac al., 2020; Li & Wang 2020; et al., (2019), which mostly focused on model and loss improvements for video-text learning, our framework would al- low us to investigate whether it is more desirable to encode either invariance or disctinctiveness to these text transformations for effective video-text representation learning.
A.3 MODALITY ABLATION
In Table A.I we provide the results of running our baseline model (samplediitiinttiennss only) within-modall instead of across modalities and find sharp drop in performance.
15

---

Under review as a conference paper at ICLR 2021
Table A.I: Multi-modal learning, mm
Modalities Epochs
HMDB
UCF
50
100
50
100
Within-modal
29.1
32.9
68.3
72.2
Cross-modal
55.1
56.9
85.1
87.9
A.4 DATASET DETAILS
The Kinetics-400 dataset (Kay et al. (017)) is human action video dataset, consisting of 240k training videos, with each video representing of 400 action classes. After filtering out videos without audio, we are left with 230k training videos, which we use for pretraining our model.
VGGSound (Chen al. 2020a) is recently released audio-visual dataset consisting of 200k short video clips of audio sounds, extracted from videos uploaded to YouTube. We use the training split after filtering out audio 7170k) for pretraining our model.
Audioset (Gemmeke et al., 2017) is a large-scale audio-visual dataset of 2. videos spanning 632 audio event classes. We use the training split (1.8M) for pretraining model.
IG65M (Ghadiyaram et al. 2019) is large-scale weakly supervised dataset collected from a social media website, consisting of 65M videos of human action events. We the all the videos in the dataset for pretraining.
HMDB-51 (Kuehne et al. HMDB-51 has three train/test splits of size 5k/2k respectively.
2011a) consists of 7K video clips spanning 51 different human activities.
UCF-101 (Soomro et al. train/test splits of size 1l/k respectively.
2012) contains 13K videos from 101 human action classes, and has three
environmental sound classification dataset which has 2K sound clips of 50 different audio classes. ESC-50 has 5 train/test splits of size 1.6k/400 respectively.
ESC-50 (Piczak 2015)
DCASE2014 (Stowell al. 2015 an acoustic scenes and event classification dataset which has 100 training and 100 testing sound clips spanning 10 different audio classes.
A.5 PREPROCESSING DETAILS
The video inputs are 30 consecutive frames from a randomly chosen starting point in the video. These frames are resized such that the shorter side is between 128 and 160, and center crop of size 112 is extracted, with no color-jiteerng applied. A random horizontal flip is then applied with probability 0.5, and then the inputs' channels are z-normalized using mean and standard deviation statistics calculated across each dataset.
One second of audio is processed after random volume jittering between 90% and 110% is applied to
x 257 99 image, by taking the log-met mel bank features with
257 filters and 199 waveform, similar to (Arandjelovic & Zisserman 2017). The spectrogram is then Z-normalized, as in (Korbar et al.) 2018). Spec-Augment is then used to apply random frequency masking to the spectrogram with maximal blocking width and sampled I times. Similarly, time-masking is applied with maximum width 6 and sampled times.
A.6 PRETRAINING DETAILS
We use R(2+1)D-18 Tran et al. 2018 the visual encoder fo and ResNet (He et al. 2016) with 9 layers the audio encoder fa unless otherwise noted; both encoders produce tixed-dimensional output (512-D) after global spatio-temporal average pooling. Both vectors are then passed through two fully-connedn man et al. 2019) which are normalized by their L.2 2-norm (Wu et al. 2018). The embedding is used for computing the contrastive loss, while for downstream tasks, linear layer after the global spatio- temporal average pooling is randomly intialized. For NCE contrastive learning, the temperature set 1/0.07. For optimizing these networks,
layers with intermediate size of 512 to produce 256-D embeddings in (Bach-)
SGD. The SGD weight decay is 10-5 and
16

---

Under review as a conference paper at ICLR 2021
the SGD momentum is 0.9. We mini-batch size of 12 each of our 64 GPUs giving effective batch size of 768 for distributed training. The initial learning rate set to 0.01 which we linearly scale with the number of GPUs, after following gradual warm-up schedule for the first 10 epochs (Goyal et a 2017). For both Kinetics and VGG-Sound, we train for 200 epochs (3 days), while for Audioset and IG65M, train for 50 epochs (5 days) and 2 epochs (7 days) respectively.
A.7 ABLATION EXPERIMENT DETAILS
For the ablations, we only train for 100 epochs on the Kinetics-400 dataset.
For both downstream tasks, we only evaluate on the first fold each but found the performance be- tween folds to be close (within 1-2%).
A.8 FULL VIDEO ACTION RETRIEVAL TABLE
In Table A.2 we show the full table on video action retrieval and compare to several of our models, pretrained on different datasets.
Table A.2: Full retrieval table.
HMDB
UCF
Recall @
5
10
20
50
5
10
20
50
Supervised (Kinetics)
49.1
74.4
83.9
90.6
96.4
86.9
94.6
96.5
98.1
99.0
ST-Puzzle (imm et al.
2019
19.7
28.5
33.5
40.0
49.4
OPN (Lee el al. 2017
19.9
28.7
34.0
40.6
51.6
ST Order Buchler cl al.
2018
25.7
36.2
42.2
49.2
59.5
ClipOrder Xu al. 2019
7.6
22.9
34.4
48.8
68.9
14.1
30.3
40.4
51.1
66.5
Speed [Benaum
2020
13.0
28.1
37.5
49.5
65.0
VCP Luo et 2020
7.6
24.4
36.3
53.6
76.4
18.6
33.6
42.5
53.5
68.1
VSP Cho et al.
2020
10.3
26.6
38.8
54.6
76.8
24.6
41.9
51.3
62.7
76.9
GDT (Kinetics)
25.4
51.4
63.9
75.0
87.8
57.4
73.4
80.8
88.1
92.9
GDT (VGG-Sound)
28.4
55.1
67.2
79.3
91.1
63.4
79.6
85.0
90.1
95.2
GDT (Audioset)
30.6
58.0
69.8
79.9
91.0
65.9
82.6
88.2
92.2 96.6
GDT (IG65M)
36.1
61.1
70.8
79.7
92.1
75.7
87.2
90.7
93.5
96.6
17

---

Under review as a conference paper at ICLR 2021
A.9 FULL VIDEO ACTION RECOGNITION TABLE
Table A.3: State-of-the-art on action recognition. Self-supervised and supervised methods on UCF101 and HMDB51 benchmarks. We follow the standard protocol and report the average top-1 accuracy over the official splits and show results for finetuning the whole network. Note that we find the supervised baseline to be around 6% and 2% better than reported in (Alwassel et al. 2020) as we use a different finetuning strategy. Methods with indicate the additional use of video titles as supervision. Methods with use ASR generated text. Methods in gray are concurrent works.
Method
Architecture
Pretrain Dataset
Top-1 Ace% HMDB UCF
Full supervision
R(2+1)D-18
ImageNet
46.7
82.8
Full supervision (Alwassel et
2020
R(2+1JD-18
Kinetics-400
65.1
94.2
Full supervision (ours)
R(2+1)D-18
Kinetics-400
70.4
95.0
Full supervision (Tran al.
2018
R(2+1)D-34
Kinetics-400
74.5
96.8
Using UCF
Shuffle and Learn (Misra et al.
2016
CaffeNet
UCF
18.1
50.2
VGAN (Vondrick
2016
VGAN
Flickr
52.1
LT-Motion (Luo et 2017
VGG-16
UCF
53.0
Geometry (Gan et al
2019
CaffeNet
UCF
23.3
55.1
OPN (Lec
2017
VGG
UCF
23.8
56.3
ST Order (Buchler of
2018
CaffeNet
UCF
25.0
58.6
CMC
al. 20019)
CaffeNet
UCF
26.7
59.1
VCP
[Luo et al.
2020)
R(2+1)D-18
UCF
32.2
66.3
Cross and Learn
(Sayed et al. 2018
VGG-16
UCF
33.0
70.5
Using Kinetics
ClipOrder (uu et
2019
R(2+1)D-18
Kinetics-400
30.9
72.4
MotionPred [Wang
2019
C3D
Kineties-400
33.4
61.2
RotNet3D [ing
Tian
2018)
3D-ResNet18
Kinetics-600
33.7
62.9
ST-Puzzle Kim et aL
2019
3D-ResNet18
Kinetics-400
33.7
65.8
DPC Han et al.
2019
3D-ResNet34
Kinetics-400
35.7
75.7
VPS Cho
2020
R3D
Kinetics-400
36.8
74.8
Speed [Benaim
2020
I3D
Kinetics-400
43.7
66.7
AoT (Wi ct al.
2018)
T-CAM
Kinetics-400
79.4
CBT Sun et al.
2019a
S3D
Kinetics-600
44.6
79.5
Multisensory (Owens & Efros
2018
3D-ResNet18
Kinetics-400
82.1
XDC (Alwassel al. 2020)
R(2+1)D-18
Kinetics-400
52.6
86.8
AV Sync+RotNet [X1
2020
AVSlowFast
Kinetics-400
54.6
87.0
AVTS KKrbarr et all 2018
MC3-18
Kinetics-400
56.9
85.8
CPD (T & Wang 2020
3D-Resnet50
Kinetics-400
57.7
88.7
AVID Morgado et al. 2020
R(2+1)D-18
Kinetics-400
60.8
87.5
CoCLR Han et al
S3D
Kinetics-400
62.9
90.6
GDT (ours)
R(2+1)D-11
Kinetics-400
60.0
89.3
Using other datosets
L (Arandjelovic & Zisserman
2017
VGG-16
AudioSet
40.2
72.3
SpeechZActron" (Nagram et al
2020)
S3D-G
MovieDataset
58.1
DynamoNet (Diba et 2019
ResNext101
YoutubeSM
58.6
87.3
MIL-NCE (Miech ct al.
2020
S3D
HowTo 100M
61.0
91.3
AVTS Korbar et al. 2018
MC3-18
AudioSet
61.6
89.0
XDC Alwasscl ct al
2020
R(2+1)D-18
AudioSet
63.7
93.0
AVID
Morgado el al.
2020
R(2+1)D-18
AudioSet
64.7
91.5
MMV
Alayrac
2120
R(2+1)D-11
Audioset
70.1
91.5
Piergiovanni et al.
2020
R(2+1)D-50x3
Youtube-2M
67.4
93.8
XDC
Alwassel et al.
2020)
R(2+1)D-18
IG65M
68.9
95.5
MMV
Alayrac et a
2020
TSM-50x2
AudioSet+HT100M
75.0
95.2
GDT (ours)
R(2+1)D-18
VGGSound (170K)
61.9
89.4
GDT (ours)
R(2+1)D-18
AudioSet (1.7M)
66.1
92.5
GDT (ours)
R(2+1)D-11
IG65M
72.8
95.2
GDT (ours) (only finetune fc)
R(2+1)D--18
IG65M
55.1
85.4
18

---

Under review as a conference paper at ICLR 2021
A.10 EVALUATION DETAILS
All evaluation code is provided in the Supplementary Material.
Video During training, we take 10 random clips of length 32 frames from each video. For video clip augmentations, we follow a standard protocol as in (Korbar et al., 2018). During evaluation, we uniformly sample 10 clips from each video, average softmax scores, and predict the class having the highest mean softmax score. We then measure the mean video top-1 accuracy across all videos and all official folds. During training, we use SGD with initial learning rate 0.0025, which we gradually warm up to 2- 10-00 in the first 2 epochs. The weight decay is set to 5 10-3 and momentum to 0.9. We use mini-batch size of 32 and train for 12 epochs with the learning rate multiplied by 5- 10-2 at 6 and 10 epochs. We compare our GDT pretrained model with both self-supervised methods, and supervised pretraining, and report average top-1 accuracies on UCF101 and HMDB-51 action recognition task across three folds in table
Few-shot classification We follow the protocol in (Jing & Tian 2018) and evaluate our our GDT pretrained network using few-shot classification on HMDB-51. We randomly sample n videos per class from the train set, average the encoder's global average pooling features from ten clips per training sample and measure classification accu- racy performance on the validation set using a k-nearest neighbor classifier, with & set to 1.
the UCF-101 101 dataset, and additionally
Retrieval We follow the standard protocol outlined in (Xu et al. 2019). We use the split 1 of UCF101, and additionally HMDB-51. We uniformly sample 10 clips per video, and average the max-pooled features after the last residual block for each clip per video. We use these averaged features from the validation set to query the videos in the training set. The cosine distance of representations between the query clip and all clips in the training set are computed. When the class of a test clip appears in the classes of k nearest training clips, it is considered to be correctly predicted. We report accuracies for k 1, 5, 10, 20, 50 and compare with other self-supervised methods on UCF101 and HMDB-51 in A.2
Audio We extract 10 equally spaced 2-second sub-clips from each full audio sample of ESC- 50 (Piczak 2015) and 60 I-second sub-clips from each full sample of DCASE2014 (Stowell et al.) 2205). We save the activations that result from the audio encoder to quickly train the linear classi- fiers. We use activations after the last convolutional layer of the ResNet-9 and apply max pooling with kernelsize (1,3) and stride (1,2) without padding to the output. For both datasets, we then optimize L2 regularized linear layer with batch size 512 using the Adam optimizer (Kingma & Bal 2015) with learning rate 1 10 weight-decay set to 5 10 and the default parameters. The clas- sification score for each audio sample is computed by averaging the sub-clip scores in the sample, and then predicting the class with the highest score. The mean top-1 accuracy is then taken across all audio clips and averaged across all official folds. For VGG-Sound (Chen al. 2020a), we fol- low their evaluation metrics but follow a much shorter training schedule as our model is pretrained. We optimize the network with batch size 128 using the Adam optimizer (Kingma & Ba 2015) with learning rate 1 10-4 for the pretrained backbone and 1 10 for the newly randomly initialized linear layer, weight-decay set to 1 10-4 and the default parameters. We drop the learning rate at 10 and 20 epochs and train for 30 epochs, which takes less than 10h on single Nvidia GTX 1080 Titan GPU.
19


Reviews:
[{"review_id": "wW7DIuEpVw9", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "## Summary\n\nThe paper introduces a general framework dubbed Generalized Data Transformations (GDT) for self supervised learning. The framework is used to perform video-audio self supervised learning and analyze what kind of transformations the representations should be invariant to or on the contrary variant to thanks to a contrastive loss. The author demonstrate the effectiveness of the proposed approach by showing that the resulting learned video representations achieve very good performance on the HMDB51 and UCF101 downstream task. \n\n## Strengths\n\n- Overall the paper is well written\n- There are some interesting findings about the paper. I am notably thinking about the results in Table 1 indicating that it is beneficial to be variant to time reversal which demonstrate that some augmentation should actually be used as negatives rather than positives in contrastive learning.\n- The final results are really good\n\n## Weaknesses\n\n**About the GDT formulation**: The idea of trying to have a general framework that can encompass all self supervised contrastive methods is a valuable effort. However, one feeling that I have about the GDT framework is that it brings more complexity (many notations are introduced e.g. $c$ the weights $w$, the different transformations $T$,...) than it actually brings new insights and benefits.\n\n-  I have notably the feeling that one could have written a paper that would have put more emphasis on the interesting findings of the specific multi-modal case that is explore here (video-audio) rather than trying to fit the findings into cumbersome notations. Things might have been different if more than just the setup of video-audio had been explored to better illustrate the versatility of the proposed framework.\n\n- Also I am questioning the generality of the framework. In particular for the multimodal case, I am unsure that the actual mathematical formulation works as plugging $f=(f_v, f_a)$ in equation (1) does not work and I guess does not correspond to the actual thing that is done in the experiments. What is actually done is that $f$ changes depending on the transformation (it becomes $f_a$ if the transformation corresponds to extracting the audio and $f_v$ otherwise), but this does not seem to be completely covered by the formulation. A similar issue would arise if we wanted to have different networks for different transformation of the same modality. \n\nIn short: what are the advantages of having this framework? Did this framework helped the authors to construct new intuitions? Since it seems to be one of the main contribution of the paper this is important that the authors address that point. \n\n**About originality**:  If we put the introduction of the GDT aside (given the previous raised point), the paper does not bring impressive conceptual innovations for training multimodal representations as the method is similar to Korbar et al. 2018, Arandjlevovic 2017 and more recently from AVID and XDC that also learn representation by using the self supervision contained in the cross modality of video and audio. In particular the loss is not novel, the architecture used to merge the modalities are not novel and the overall conclusion is in line with previous work (that the best thing seems to be to use the other modality as an extra view for learning good representations). \n\n**About TR and TS invariances**\n\n- If I understand correctly there is a single negative coming from the same video that has been time reversed (TR) (or time shifted TS), however there would be many more negatives coming from other videos (in the denominator of equation (1)). I wonder if it would be beneficial to try to upweight these single negatives coming from those specific transformations? Is this something that the authors have considered? \n\n- A related question is whether or not TR and TS can be combined to obtain 3 invariances? Would that be beneficial? From an intuitive point of view it seems that the two signals could be complementary. \n\n\n**Resolution of the video 112x112** \n\nIn the appendix it is mentioned that the resolution used are 112x112. What would happen if you were to use higher input resolution (e.g. 224x224). In particular in XDC this is the resolution used and this might lead to improvements in your case as well that could further improve the performance of the method.\n\n**Linear evaluation on UCF/HMDB**: It would be nice to also evaluate the representations on the frozen setting on UCF and HMDB (as more recent methods like ELo and MILNCE are doing). This would make the comparison in Table 2 a bit more stronger than using the retrieval or the few-shot setup that was used by methods that were not leveraging multiple modalities for learning. \n\n**Would the final trained models be available?** In particular the IG65M dataset is not open sourced so its important that the authors release the weights of the trained models. \n\n## Conclusion and assessment\n\nOverall the paper is well written and well executed. The results are strong for self supervised learning from audio and video. Nonetheless I have some global concerns about the work, notably the limited usefulness of the introduced general framework (GDT), and the overall lack of novel concepts or new insights provided by the work (despite the TR and TS findings that seem new to me). That is why as of now I feel the paper is borderline. I am still leaning towards acceptance since I feel the paper is an important milestone for self supervised learning from video and audio but I will wait for the answers of the authors to take a final informed decision.\n\n## Post Rebuttal comment\n\nThe authors have clarified the contributions of their work and improved the manuscript accordingly. Given this and the other positive points about the paper I am willing to increase my score to accept.\n\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "tjM_2nZBh7g", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper is well written and the motivation is simple and clear. The conclusions are supported by sufficient experiments on various datasets and tasks, e.g.,  video and audio classification and retrieval tasks on datasets such as HMDB-51, UCF-101, DCASE2014, ESC-50 and VGG-Sound. The self-supervised pretraining is conducted on VGGSound, AudioSet, and IG65, showing the benefits of GDT in multiple source datasets. The introduced Generalized Data Transformations could potentially benefit multi-modal self-supervised learning in incorporating more transformations. \n  \nI have the following comments:\n\n1. In Section 3.1, the authors discussed contrastive audio-visual self-supervision. Can the model generalize to other supervision signals? If so, what's the limitation of the selection process generalization to more self-supervised signals?\n\n2. This paper focuses on video and audio learning. How can this framework be generalized to video and text? The authors may share some insights in the conclusion. \n\n3. Will the order of the hierarchical transformation affect the feature learning process? For example, the order of t1 and t2 is swapped in Fig. 1 A.", "extra_scores": {}, "extra_text": {}}, {"review_id": "ZRTrjv5YMU3", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper presents Generalized Data Transformations (GDT), a framework that unifies different self-supervised learning methods. Using this framework for unsupervised video representations, the paper gets state-of-the-art results on downstream tasks.\n\n### Strengths:\n- Unification of different self-supervised methods into a single framework, which is clean and general. This includes two main very related ideas. First, the different methods are reframed to work in a contrastive setting. Second, in order to organize the combination of the methods, the authors propose a way to do it in a systematic way.\n- Following the previous framework, the paper proposes combinations of self-supervised methods for video representations, and it shows better results than competing methods. Both the baselines and ablations are sensible.\n- Overall well written and easy to follow.\n- Code is provided.\n\n### Weaknesses\n- It is not clear whether the main contribution is the GDT framework or its application to video. After reading the experiments section, GDT feels more like a way of structuring the experiments, and less like a unified way of understanding self-supervised learning. This feeling is reinforced by the seemingly arbitrary task the authors select (audio-visual data in videos), given the method. While video representations is a very important topic, the method does not lead to video representations as its most direct application. \n- The GDT framework consists of two ideas, and they are not properly separated in the paper (or not totally unified in a single one). On the one hand, there is the formulation of different self-supervised methods as contrastive losses. On the other hand, there is the organization of data augmentation combinations. Please note that the second is not strictly necessary for the first one to work (therefore the previous point about \"structuring the experiments\").\n- The results lack a lot of intuition and analyses. Why some data augmentations work better than others? Why opposite augmentations (being variant _and_ invariant to time shift) are useful separately? Also, the application is specific to video, so explanations and analyses of the results on video tasks should be discussed. Why do these combinations of augmentations work for video? What information are the representations encoding?\n- Other unifying frameworks have been proposed for self-supervised contrastive formulations. Specifically, [1] (which is cited in the paper but not discussed) proposes to view all of these methods as multiple views of a scene (which are the positives). What conceptual contribution does this paper add on top (or instead of) the one presented in [1]?\n- While the framework is theoretically clean and general, in practice there are a lot of corner cases, and exceptions. With only two augmentations (excluding the \"sampling\" one), a lot of the combinations are already not possible, and the authors have to propose specific ways of combining them that are very tailored to the problem. This implies that the method is general but it has a lot of hyperparameters that need to be tuned or decided manually. Similarly, the augmentations used in the paper are not systematically selected following any rules or framework, but chosen directly by the authors.\n\n### Additional comments and questions:\n- Is the reference to SimCLR incorrect? Right now it shows Tian et al 2019, in all the cases it is cited. Tian et al 2019 is the Contrastive Multiview Coding paper, not SimCLR.\n- Is the format the correct one for ICLR 2021?\n\n### Final recommendation\nOverall, I believe the strengths outweigh the weaknesses and I recommend this paper to be accepted to ICLR, but I suggest the authors address the previously mentioned points.\n\n### References\n[1] Yonglong Tian, Dilip Krishnan, and Phillip Isola. _Contrastive multiview coding_. arXiv preprint arXiv:1906.05849, 2019.\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "Qi2s16X4T6F", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The authors propose to integrate a few data transformations into a generalized formulation. Similarly to the motivation of previous contrastive learning, the generalized transformation are required to learn robust representation while balancing between invariance and distinctiveness. Experiments show some validations of the proposed framework on audio-visual scenarios.\n\n+ The authors provide a good summarization of existing contrastive augmentations and data sampling into a generalized formulation.\n+ Video transformations in contrastive learning has not been carefully investigated before.\n+ The raised problem of balancing (or enumerating) between distinctive vs invariant transformations is underexplored and worth studying.\n\n- While the introduced formulation is a good wrap-up of possible contrastive augmentations, it has no practical impact until the users find the best combination through a brute-force enumeration of candidate transformations.\n- I believe only the formulation is general, while their method or framework would not be generalizable to other datasets / modalities / self-tasks. With different scenarios, different combinations have to be experimented one by one.\n- The experiment is done on a very specific scenario: audio-visual task, from which I believe that the main contribution of this work is more of the improvement of a specific audio-visual self-sup method, rather than a generalized formulation of the transformations.\n- Minor: Quite many symbols hurt the readability. ", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The authors provided a comprehensive rebuttal to the reviewers' feedback that addressed most of the concerns. AnonReviewer3 raised some major concerns that were partially resolved in a revision. The paper has received a split recommendation from the reviewers but within the review and discussion periods, there was no strong support towards accepting the paper. Although the paper has received some positive feedback, some of the reviewers' concerns were not fully addressed. I'd recommend the authors to address all the comments and add clarifying notes to the paper to avoid such misunderstandings if they decide to resubmit the paper to another venue.  
Decision: Reject