Active Contrastive Learning of Audio-Visual Video Representations
Authors: Shuang Ma
Zhaoyang Zeng
Daniel McDuff
Yale Song
Venue: ICLR 2021 Poster (2021)
Keywords: self-supervised learning
contrastive representation learning
active learning
audio-visual representation
video recognition

Abstract: Contrastive learning has been shown to produce generalizable representations of audio and visual data by maximizing the lower bound on the mutual information (MI) between different views of an instance. However, obtaining a tight lower bound requires a sample size exponential in MI and thus a large set of negative samples. We can incorporate more samples by building a large queue-based dictionary, but there are theoretical limits to performance improvements even with a large number of negative samples. We hypothesize that random negative sampling leads to a highly redundant dictionary that results in suboptimal representations for downstream tasks. In this paper, we propose an active contrastive learning approach that builds an actively sampled dictionary with diverse and informative items, which improves the quality of negative samples and improves performances on tasks where there is high mutual information in the data, e.g., video classification. Our model achieves state-of-the-art performance on challenging audio and visual downstream benchmarks including UCF101, HMDB51 and ESC50. 

Published as a conference paper at ICLR 2021
ACTIVE CONTRASTIVE LEARNING OF AUDIO-VISUAL VIDEO REPRESENTATIONS
Shuang Ma
Zhaoyang Zeng*
Daniel McDuff
Yale Song Microsoft Research Redmond, WA, USA
Microsoft
Sun Yat-sen University Guangzhou, China
Microsoft Research Redmond, WA, USA
Redmond, WA, USA
ABSTRACT
Contrastive learning has been shown to produce generalizable representations of audio and visual data by maximizing the lower bound the mutual information (Mf) between different views of an instance. However, obtaining a tight lower bound requires sample size exponential in MI and thus large set of negative samples. We can incorporate more samples by building a large queue-based dic- tionary, but there are theoretical limits performance improvements even with a large number of negative samples. We hypothesize that random negative sampling leads to a highly redundant dictionary that results suboptimal representations for downstream tasks. In this paper, we propose an active contrastive learning approach that builds an actively sampled dictionary with diverse and informative items, which improves the quality of negative samples and improves performances on tasks where there is high mutual information in the data, c.g., video classifica- tion. Our model achieves state-of-the-ar performance challenging audio and visual downstream benchmarks including UCFI01, HMDB51 and ESC50
INTRODUCTION
Contrastive learning of audio and visual representations has delivered impressive results on various downstream scenarios (Oord et al.) 2018; Hénaff et al. 2019, Schneider et al.] 2019 Chen et al.) 2020). This sule-super training process can be understood as building dynamic dictionary per mini-batch, where "keys" are typically randomly sampled from the data. The encoders are trained to perform dictionary look-up: an encoded "query" should be similar to the value of its matching key and dissimilar to others. This training objective maximizes a lower bound of mutual information (MI) between representations and the data (Hjelm al. 2018 Arora al. 2019). However, such lower bounds are tight only for sample sizes exponential in the MI (McAllester & Stratos 2020). suggesting the importance of building a large and consistent dictionary across mini-batches.
Recently, He et al. (2200) designed Momentum Contrast (MoCo) that builds a queue-based dic- tionary with momentum updates. It achieves large and consistent dictionary by decoupling the dictionary size from the GPU/TPU memory capacity. However, Arora et al. (2019) showed that simply increasing the dictionary size beyond a threshold does not improve (and sometimes can even harm) the performance on downstream tasks. Furthermore, we find that MoCo can suffer when there high redundancy in the data, because only relevant - and thus limited - parts of the dictionary are updated in each iteration, ultimately leading to dictionary of redundant items (we show this empir- ically in Fig. 33. We argue that random negative sampling is much responsible for this: randomly constructed dictionary will contain more "biased keys" (similar keys that belong the same class) and "ineffective keys" (keys that can be easily discriminated by the current model) than a carefully constructed one. Furthermore, this issue can get aggravated when the dictionary size is large.
In this paper, we focus on learning audio-visurl representations of video data by leveraging the natural correspondence between the two modalities, which serves useful self-supervisory signal (Owens & Efros 2018 Owens et al. 2016 Alwassel et al. 2019). Our starting point is contrastive learning (Gutmann & Hyvärinen 1010:Ord et 2018 with momentum updates (He al. 2020
"Equal contribution
Code is available

---

Published as a conference paper at ICLR 2021
However, discussed above, there are both practical challenges and theoretical limits to the dictionary size. This issue is common to all natural data but especially severe in video; successive frames contain highly redundant information, and from the informationnthhorreee perspective, audio- visual channels of video data contain higher MI than images because the higher dimensionality i.e., temporal and multimodal reduces the uncertainty between successive video clips. Therefore, a dictionary of randomly sampled video clips would contain highly redundant information, causing the contrastive learning to be ineffective. Therefore, we propose an actively sampled dictionary to sample informative and diverse set of negative instances. Our approach is inspired by active learning (Settles) 2009) that aims to identify and label only the maximally informative samples, so that one can train a high-performing classifier with minimal labeling effort. We adapt this idea to construct non-redundant dictionary with informative negative samples.
Our approach, Cross-Modal Active Contrastive Coding (CM-ACC), learns discriminative audio- visual representations and achieves substantially better results on video data with a high amount of redundancy (and thus high MI). We show that our actively sampled dictionary contains negative samples from a wider variety of semantic categories than a randomly sampled dictionary. As a re- sult, our approach can benefit from large dictionaries even when randomly sampled dictionaries of the same size start to have a deleterious effect on model performance. When pretrained on AudioSet (Gemmeke et al.] (017), our approach achieves new state-of-the-art classification performance on UCF101 (Soomro et al. 2012). HMDB51 (Kuehne et al. 2011), and ESC50 (Piczak 2015b).
BACKGROUND
Contrastive learning optimizes an objective that encourages similar samples to have similar repre- sentations than with dissimilar ones (called negative samples) (Oord et al. 2018):
ef(a00)) ((a
min E Exppx 9y.0h
-log
(1)
( 30,) (T:0)))Ti( 30,3
The samples - and are drawn from the same distribution ilar and dissimilar to x, respectively. The objective encourages f(-) and fi(-) to learn representations of x such that (x, xx)) have a higher similarity than all the other pairs of (I,
X, and are assumed to be sim-
We can interpret it as dynamic dictionary look-up process: Given a "query" x, it finds the correct "key" x among the other irrelevant keys x in a dictionary. Denoting the query by q = fz), the correct key by k+ = h(x*t ), and the dictionary of K negative samples by fk; h(x,) ) E [1, K],
we can express equation in softmax form, mino, 0 En-px [loy x 2,77 where 0, and Oz
are parameters of the query and key encoders, respectively, and F is a temperature term that controls the shape of the probability distribution computed by the softmax function.
Momentum Contrast (MoCo) decouples the dictionary size from the mini-bated size by imple- menting queue-based dictionary, i.e., current mini-batch samples are enqueued while the oldest are dequeued (He et al., 2020). It then applies momentum updates to parameters of a key encoder Og with respect parameters of a query encoder, 02 + m0, + (1 m)),. where m E [0, 1) is momentum coefficient. Only the parameters 0, are updated by back-propagation, while the parame- ters 0g are defined as a moving average of 0g with exponential smoothing. These two modifications allow MoCo to build a large and slowly-changing (and thus consistent) dictionary.
Theoretical Limitations of Contrastive Learning. Recent work provides theoretical analysis of the shortcomings of contrastive learning. McAllester & Stratos (2020) show that lower bounds to the MI are only tight for sample size exponential in the MI, suggesting that a large amount of data are required to achieve tighter lower bound on MI. He et al. (2020) empirically showed that increasing negative samples has shown to improve the learned presentations. However, Arora et al. (2019) showed that such phenomenon does not always hold: Excessive negative samples can sometimes hurt performance. Also, when the number of negative samples is large, the chance of sampling redundant instances increases, limiting the effectiveness of contrastive learning. One of our main contributions is to address this issue with active sampling of negative instances, which reduces redundancy and improves diversity, leading improved performance on various downstream tasks.
2

---

Published as a conference paper at ICLR 2021
contrastive loss
contrastive loss
Prqqueue
backprop
buckprop
D
D2 dequeue
Da
uctive sampling
D₂
dequeue
Kmeans
Kmeans
Rg.
00
momm entumm sppdute
momentcum
gradient space
gradient space
upplute
Ry
fl
fa
No
Jay
guery
encoded key pools
(a) Cross-Modal Active Contrastive Coding (CM-ACC)
(b) Active Sampling
Figure (a) We extend contrastive learning to the cross-modal scenario and adapt momentum contrast (MoCo) (He et al. (0200) to the dictionary update. Different from all existing work, we propose an active learning idea to the negative sampling. (b) To sample negatives, we use the
gradient space of our key encoders to estimate the uncertainty of each candidate in audio/visual pools, and take diverse set of negatives in that space using the k-MEANS .812 algorithm.
APPROACH
3.1
CROSS-MODAL CONTRASTIVE REPRESENTATION LEARNING
Our learning objective encourages the representations of audio and visual clips to be similar if they come from the same temporal block of video. Let A = {an, ........ .N-1) and V {vn, ......... ,NN be collections of audio and visual clips, where each pair (a,, 12)) is from the same block of a video. We define query encoders fa and fo and key encoders he and hy for audio and visual clips. respec- tively, with learnable parameters {0-- 0²) for the query encoders and { 0% for the key encoders. These encoders compute representations of audio and visual clips as queries and keys,
UN-1)
q" (uguere), ka = (ukev), qah (aquuery) ka = h (a
(2)
We train our encoders to perform cross-modal dictionary look-up, e.g., given a query video clip yuuery, we find the corresponding audio clip key from dictionary Da. Adapting MoCo (He al. 2020 to our cross-modal setup, we implement a queue-based dictionary Da that stores keys of audio clips {k. Te where K is the dictionary size. We compute the contrastive loss and backpropagate the gradients only to the visual query encoder fu and update the parameters By. For the audio encoder ha, we apply the momentum update (He al. 2020).
0
+ m6 (1 m)00
(3)
The parameter 0 is not updated in this contrastive coding step; we update it during the audio-to- visual step (similar as above with the opposite modalities). Here we explain the visual-to-audio step only; we perform bi-directional contrastive coding and train the whole model end-to-end.
3.2 ACTIVE SAMPLING OF NEGATIVE INSTANCES: UNCERTAINTY AND DIVERSITY
The quality of negative samples crucial in contrastive learning. Existing work typically adopts random negative sampling. However, we want diverse set of negative samples so that comparisons between positive and negative pairs are the most informative they can be. Motivated by active learn- ing (Settles 2009), we propose gradient-based active sampling approach of negative samples. In active learning, the learner chooses samples that seem maximally informa- tive and queries an oracle for labels obtain an optimal solution with a minimal labeling budget. Adapting this to our setting, we can empower the learner to choose the maximally informative neg- ative samples to construct dictionary; the main question how to measure the informativeness of samples without labels.
improve the quality
One way to measure informativeness is through the lens of uncertainty: If model is highly uncertain about its prediction of sample, we can ensure the maximum update the model by including the sample in a mini-batch (conversely, if the uncertainly is low for all samples in a mini-batch the model update will be small). Ash al. (2020) showed that gradients of loss function with respect to the model's s most confident predictions can approximate the uncertainty of samples, demonstrating its effectiveness in active learning. They provide theoretical justification by showing that gradient
3

---

Published as a conference paper at ICLR 2021
norms of the last layer of neural network with respect to pseudo-labels provides lower bound on gradient norms induced by any other labels. In this work, we use gradients of the last layer to measure the uncertainty and encourage our model to include samples that have the highest gradient magnitudes to constitute dictionary.
While the uncertainty of each individual samples is important, the diversity of samples is also critical measure of informativeness. Intuitively, it is possible that a model is highly uncertain about samples from particular semantic categories, but constructing a mini-batch of samples from just those categories can severely bias gradients and ultimately lead to a bad local minima. There are several principled approaches to ensure diversity. e.g., submodular optimization (Fujishige) 2005) and Determinantal Point Processes (DPP) (Macchil [1755 Kulesza & Taskar] (011)) Unfortunately, those methods are typically inefficient because of the combinatorial search space (Nemhauser al.) 1978 Gilks et al. 1995). In this work, instead of using the expensive solutions, we opt to the fast solution Ash et al. (2020) and use the initialization scheme of the k-MEANS+ seeding algorithm (Arthur & Vassilvitskii 2007) to sample a diverse set of negative samples.
3.3 CROSS-MODAL ACTIVE CONTRASTIVE CODING
Algorithm [ describes our proposed cross-modal active contrastive coding (we provide a simplified version here; we include a more detailed version and another version without active sampling in Appendix). At a high-level, we initialize the dictionaries D, and D. with K randomly drawn sam- ples from V and A, respectively (lines 3-4). For each epoch, we construct "negative candidate pools" U, and Ua with N random samples from V and A, respectively (lines 6-7). For each iteration within an epoch, we actively select the most informative negative samples S, and S, from the pools U, and Ua. respectively, and enqueue them into the dictionaries D, and Da. respectively (lines 9-21). We then perform cross-modal contrastive coding, update the parameters of query encoders 8 and 0% via backpropagation, and apply momentum updates the parameters of key encoders 0% and 0% (lines 22-27).
Algorithm 1 Cross-MMdal Active Contrastive Coding
1: Require: Audio-visual clips A, V; encoders for for by, has dictionary size K; pool size N; batch size M
2: Initialize parameters, 07: 0% 07 Of in Uniform((. 1)
3: Draw random dictionary, De <- (21,". ,0K] in V, Do < (a),... "aa.] ~
4: Encode dictionary samples, k." <- ho (wx),, Yos E Dr, ke t ha ha(a)) Yas E D.
5: for epoch = 1 to #epochs: do
6: Draw random pool. U, (u1,... uN) ~ V. Un + far.... aN) i A
7: Encode pool samples, K. <- ho (v(l)), Von E Uv. kig + ha la((a)) ) Van Ua
for = to #mini-batches: do
9:
Draw mini-batch, B, + [111 vart t ~ V, Ba + fai." aar ~ A Active sampling of negative video keys for D..
10:
11:
Encode mini-batch samples, of <- fa(a.), Va, Ba
12:
Compute pseudo-labels, Vi <- arg max p(1)% Vo: B.), Ve, E U, Compute gradients qu.. using the pseudo-labels Gü. Vn E [] Obtain S., <- k-MEANS tut ((g.., Un UUA.D. D.),, #seeds M)
13:
14:
15:
Update D., +-
16:
D Active sampling of negative audio keys for D.
17:
Encode maiibbtch samples, q.¹ + fv(),, Vu, E B Compute pseudo-label, label, 5n +- arg max p(0" on, B.). Va, U. Do Compute gradients Ia.. using the pseudo-labels Da. Vn E [1
18:
19:
20:
Obtain S. + k- MEMANSSTT (1ga.. o. E U.. #seeds = M)
21:
Update Da ENQUEUE(DEQUEUE((...
22:
Cross-modal contrastive predictive coding
23:
Encode mini-batch samples, k. <- hy (w)), Vu; E Bo. R +- ha (ai), Va, E Ba Compute p(yč' (v,, ox, D.) and p(yilas, vs D.), Vi E [1, M]
24:
25:
& Update model parameters
26:
Update parameters of query encoders 02 and 0g with backpropagation Momentum update parameters key encoders 0g and 0%
27:
28:
end for
29:
end for
30:
return Optimal solution 8,, 07, 8 8%
4

---

Published as a conference paper at ICLR 2021
Active sampling. To measure uncertainty, define pseudo-label space induced by the queries from the other modality, and take the gradient of the last layer of query encoder with respect to the most confident prediction, which we call the pseudo-label ÿ. For instance, in the case of sampling negative video keys from the pool U, (lines 10-15) 5), we compute the pseudo-posterior a video key Un E Da.
P(Un Un, Ba) = exp(ki 99)
Vj E [1.M]
(4)
E exp(k; o.)
where B, is the current mini-batch of audio queries and defines the pseudo-label space. Note that we consider only the samples in Up\De D, to rule out samples already D,.. Intuitively, this computes the posterior by the dot- dot-product similarity between un and all of E Ba, producing an M-dimensional probability distribution. We then take the most confident class category as the pseudo-label yn (line 12) and compute the gradient according to the cross-entropy loss
à Diiast
Go
LCE (p(U%) Un: Bu), Bk) |0=0;
(5)
where Olast is the parameters of the last layer of 0 (in this case, 0g of the audio query encoder ha). Intuitively, the gradient 9o,, measures the amount of change - and thus, the uncertainty - Un will bring to the audio query encoder ha-
One can interpret this as a form of online hard negative mining: The gradient is measured with respect to the most probable pseudo-label ПN induced by the corresponding audio query of. When we compute the contrastive loss, the same audio query will be maximally confused by v, with its positive key w per dot dot-product similarity, and vn in this case serve hard negative sample.
Next, we obtain the most diverse and highly uncertain subset S, S U, DD, using the initialization scheme of k-MEANS** (Arthur& & Vassilvitskii 2007p over the gradient embeddings Iu (1 nn 14). The k-MEANS initialization scheme finds the seed cluster centroids by iteratively sampling points with probability proportion to their squared distances from the nearest centroid that has already been chosen (we provide the exact algorithm in the Appendix). Intuitively, this returns a diverse set of instances sampled greedy manner, each of which has high degree of uncertainty measured as its squared distances from other instances that have already been chosen. Finally, we enqueue S, into D, and dequeue the oldest batch from D, (line 15). We repeat this process to sample negative audio keys (lines 16-21); this concludes the active sampling process for D, and D..
Cross-modal contrastive coding. Given the updated D. and Da, we perform cross-modal con- trastive coding. For visual-to-audie coding, we compute the posteriors of all video samples v; B with respect to the negative samples in the audio dictionary Da.
plu U2 a4 Da) = exple kG /T)
Vi E M]
(6)
Ej= explo" kg/T)
where the posterior is defined over mossl space with positive and K negative pairs (line 24). Next, we backpropagate gradients only to the query encoders fo and fo (line 26),
8 0 - VeLc Vie Ne= B9 - 0 VoLce(p((" .), B 10=0
(7)
while applying momentum update to the parameters of the key encoders hG and ha (line 27),
0% + mo + (1 - -m));, 0% + m + (1 (1-m)6 m)00
(8)
The momentum update allows the dictionaries to change their states slowly, thus making them con- sistent across iterations. However, our cross-moaal formulation can cause inconsistency in dictio- nary states because the gradient used to update query encoders are not directly used to update the corresponding key encoders. To improve stability, we let the gradients flow in a cross-modal fash- ion, updating part of f. and ha using the same gradient signal from the contrastive loss. We do this by adding one FC layer on top of all encoders and applying momentum update to their parameters. For example, apply momentum update to the parameters of the FC layer top of ha using the parameters of the FC layer from f.. We omit this in Alg [ for clarity but show its importance in our ablation experiments (XMoCo (w/o fel) in Table
5

---

Published as a conference paper at ICLR 2021
#
Approach
Pretrain Ohj.
UCF101
HMDB51
ESC50
Gains
Scratch
63.3
29.7
54.3
Supervised
Supervised
86.9
53.1
78.3
SMoCo
Uni. rand.
70.7
35.2
69.0
XMoCo (w/ fel)
Cross rand.
72.9 (12.2)
37.5 (12.3)
70.9 (11.9)
AOO
XMoCo
Cross rand.
74.1 (1)
38.7 (11.2)
73.0 (12.1)
A
CM-ACC (w/o fel)
Cross active
75.8 (11.4)
39.1 (11.5)
77.3 (11.9)
A A
CM-ACC
Cross active
(1.2(
40.6(1199
79.2 (16.2)
A
Table Top-1 accuracy of unimodal vs. cross-modal pretraining on downstream tasks.
4 RELATED WORK
Self-supervised learning has been studied in vision, language, and audio domains. In the image domain, one popular idea is learning representations by maximizing the MI between different views of the same image (Belghazi et al.) 2018 Hjelm et al., 2018 Tian et al. 2019 He et al.) 2020). In
the video domain, several approaches have exploited the spatio-temporal structure of video data to
design efficient pretext tasks, e.g. by adopting ordering (Sermanet al., 2017; Wang et al. 2019b),
temporal consistency (Dwibedi et al.) (219)), and spatio-temporal statistics (Xu et al. [0199] Wang (et al., 2019a| Han et al. 2019). In the language domain, the transformer- with the masked language model (MLM) objective has been the most successful (Devlin et al.) 2019
approaches trained
Liu et al.] 2019 Yang et al. 2019). Riding on the success of BERT (Devlin et al., 20199) several
concurrent approaches generalize to learn visual linguistic representations (Lu al. 2019 Li (et al., 2020 Su et al., 2019 Tan & Bansal, 2019 Li et al. 2019). CBT (Sun et al. (2019)) and ViReoBERT (Sun et al. 2019h) made efforts on adapting BERT-style pretraining for video.
Besides vision and language signals, several approaches learn audio-visual representations in a self- supervised manner (Owens et al.) [0166] Arandjelovvi & Zisserman 2017 Owens & Efros
2018
Owens et al. 2016). Recently, audio-visual learning has been applied to enable interesting appli-
cations beyond recognition tasks, such sound source localization/separatioo (Zhao et al.
2018
Arandjelovic & Zisserman 2018 Gao et al. 2018 Gao & Grauman [019a Ephrat et al.
2018
Gan et al.] 2020) al., [209 Yang et al. [0220) and visual-to-soum generation (Hao al.,
2018 Zhou et 2018). The work of Owens & Efros (2018), Korbbar et al. (2018), and Alwassel
jt al. (019) are similar spirit to our own, but our technical approach differs substantially in the
use of active sampling and contrastive learning.
Hard negative mining used in a variety of tasks, such as detection (Li et al. 2020), tracking (Nam & Hanl 2016), and retrieval (Faghri et al. 2017 Pang et al.) 2019), to improve the quality of prediction models by incorporating negative examples that are more difficult than randomly chosen ones. Several recent work have focused on finding informative negative samples for contrastive learning. Wu et al. (2020) show that the choice negative samples is critical in contrastive learning and propose variational extension to InfoNCE with modified strategies for negative sampling. [scen et al.) (2018) propose hard examples mining for effective finetuning of pretrained networks. CCo et al. (2020) utilize negative sampling reduce the computational cost. In the context of audio-visual self- supervised learning, Korbar et al.] (211) sample negatives under the assumption that the smaller the time gap is between audio and visual clips of the same video, the harder it is differentiate them (and thus they are considered hard negatives). Our proposed approach does not make such an assumption and estimates the hardness of negatives by directly analyzing the magnitude of the gradients with respect to the contrastive learning objective.
EXPERIMENTS
Experimental Setting, We use 3D-RRsNNet18 (Hara et al. 2018) 8) our visual encoders (fr and h.)) in most of the experiments. We also use R(2+1)D-18 (Tann et al. 2018) to enable a fair comparison with previous work (see Table 4). For audio encoders (fa and h.)) we adapt ResNet-18 (He et al.) 2016) to audio signals by replacing 2D convolution kernels with 1D kernels. We employ Batch Normalization (BN) (loffe & Szegedy 2015) with the shuffling BN (He et al., 2020) in all our encoders. All models are trained end-to-end with the ADAM optimizer (Kingma & Ba 2014) with an initial learning rate 7 10-3 after warm-up period of 500 iterations. We the mini-batch

---

Published as a conference paper at ICLR 2021
25
45
100
Active Sampling (Grad Embed)
to
90
Active Sampling (Feat Embed)
Number of Categories
20
Number of Categories
Number of Categories
80
Re Random Sempling
to tr go
70
15
60
50
10
in to G
30
20
10
Batch
Batch Size = 64
Batch Size = 128
50 100 200 300
50 100 150 200 250 300
100 150 200 250 300
Number of Iterations
Number of Iterations
Number Iterations
Figure 2: Effects of random sampling and active sampling on the number of categories.
size M = 128, dictionary size K = 30 128, pool size N 300 x 128, momentum m 0.999, and temperature t = 0.7. We used 40 NVIDIA Tesla P100 GPUs for our experiments.
We pretrain our model on Kinetics- 700 (Carreira et al. 2019) and AudioSet (Gemmeke et al. 2017) when comparing with state-of-the-art approaches. For Kinetics-700, we use 240K randomly se- lected videos that contain the audio channel. On AudioSet, we use both a subset of 240K randomly selected videos and the 1.8M full set. For our ablation study, we use Kinetics-Sound (Arandjelovic & Zisserman 2017) that contains 22K videos from 34 classes that are potentially manifested both visually and audibly, and thus provides relatively clean testbed for ablation purposes. As for down- stream tasks, we evaluate our models on action recognition using UCF101 (Soomro et al. 2012)) and HMDB51 (Kuehne et al. 2011), and on sound classification using ESC50 (Piczak 2013b).
Unimodal ys. cross-modal pretraining, To validate the benefits of cross-modal pretraining, we compare it to its unimodal counterparts. We pretrain our model on Kinetics-Sound with randomly sampled dictionary (similar to MoCo (He et al.) 2020)); we call this XMoCo. For the unimodal case, we pretrain two models on visual clips and audio clips. respectively; we call these SMoCo. We also compare ours with a model trained from scratch (Scratch), and a model pretrained on Kinetics- Sound in a fully- supervisee manner (Supervii vid).. Lastly, we include XMoCo (w/o o Ecl) that is identical to XMoCo except that we do not include the additional FC layers on top of the encoders. All these models are finetuned end-to-end on each downstream task using the same protocol.
Table 1 shows the top-1 accuracy of each downstream task. We observe that all the self-supervised models outperform Scratch on all downstream tasks, suggesting the effectiveness of pretraining with contrastive learning. We also see that our cross-modal objective outperforms the unimodal objective (A)) The comparisons between XMoCO vs. XMoCO (w/o fcl) and CCC vs. CM ACC (w/o fcl) show the effectiveness of the additional FC layer top of the encoders (A(5-O), A (O) When adding the FC layer, the performance further improves on all three benchmarks. This shows the importance of letting the gradients flow in cross cross-modal fashion. Finally, the performance gap with the full-supervised case shows there is still room for improvement in the self-supervised approaches.
Next, we compare the number of unique categories the sampled instances originally belong to, using the ground-truth labels provided in the dataset, Our logic is that the more categories the samples come from, the more diverse and less redundant the samples are. We train these on UCF-101 over 300 iterations with different mini-batch sizes, M E {32, 64, 128). Fig. 2 shows that active sampling selects more categories than random sampling across all three sampling (with gradient embedding) covers 60- 70% of categories on UCF101, which is substantially more diverse than random sampling (30-40%). (A plot showing the probability of sampling unique negatives (instances from different categories) is shown in Appendix Figure While both sampling schemes perform similarly in early iterations, active sampling starts choosing more diverse instances as the training progresses; this is because the gradient embedding becomes more discriminative with respect to the uncertainty.
sizes. At M = 128, active
Random vs. active sampling. To validate the benefit of active sampling over random sampling, we compare models pretrained with different sampling approaches downstream tasks. As shown in Table our CM-ACC outperforms the XMoCo, which uses random sampling, by large margins, ie. 3.1 %,, 1.9%, and 6.2% UCF101, HMDB51, and ESC50, respectively (A()-
7

---

Published as a conference paper at ICLR 2021
Pretrain Objective
Embedding Space
UCFI01
HMDB51
ESC50
Cross-modal active
Feature Embedding
74.5
38.2
75.1
Cross-modal active
Gradient Embedding
(12.7
40.6 (12.4)
79.2 (14.1)
Table 2: Top-1 accuracy downstream tasks: feature- vs. gradient-based embedding.
UCF101
HMDB51
M=32
M=64
M=128
M=32
M=64
M=128
Random OHEM Active
61.9
63.1
66.9
33.1
33.8
35.8
50.2 (-11.7)
60.8 (-2.3)
65.7 (-1.2)
26.8 (-6.3)
30.1 (-3.7)
33.2 (-2.6)
78.0 (+16.1)
78.9 (+15.8)
79.2 (+12.3)
41.2 (48.1)
42.3 (+8.5)
42.6 (+6.8)
Table 3: Online hard example mining (OHEM) (Shrivastava al. 2016 vs. our active sampling
Feature vs. gradient embedding. We compare two ways to do active sampling: using gradient embeddings (Eqn. 5) and feature embeddings (the outputs from ha and h2)) when selecting the seed centroids with K-MEANS+ Fig. shows that gradient embeddings produce a more diverse set of negative samples than feature emheddings; this is consistent across all three batch sizes. Table 2 shows that this diversity helps achieve better downstream performances across all three benchmarks. Fig. 5 (in Appendix) provides further insights, showing that the samples with high gradient magnitudes tend to be more informative negative samples.
From a theoretical aspect, the gradient norm induced by each candidate with computed pseudo labels estimates the candidates influence the current model. The gradient embeddings convey informa- tion both about the model's uncertainty and potential update direction upon receiving a candidate. However, such messages are missing from the feature embeddings. This shows the importance of considering both uncertainty and diversity when selecting random samples: the k-MEANS* ensures the diversity in the sample set, but without the uncertainty measure we lose important discriminative information from the candidates.
Online hard example mining vs. active sampling. We compare our approach to online hard example mining (OHEM) (Shrivastava et al. (016)) which constructs negative samples by explicitly choosing the ones that incur high loss values. Specifically, we compute the pseudo-labels for all keys (negative sample candidates) with given mini-batch of queries. We then compute the classification loss based on these pseudo labels and select the top M keys with the highest loss values. We pretrain the models on Kinetics-700 (Kay et al. (017) and report the top-1 accuracy on UCF101 (Soomro et al. 2012) and HMDB51 (Kuehne et al. [011]. We use the same architecture and hyper- parameters; the only difference is the sampling approach.
Thole OHEM is generally less effective than both random sampling and our active sampling. Intuitively, OHEM promotes the dictionary to contain the most challenging keys for given mini- batch of queries. Unfortunately, this causes OHEM to produce a redundant and biased dictionary, e.g., negative samples coming from a particular semantic category. Our results show that, when M (mini-batch size) is small, the performance of OHEM is even worse than random sampling, although the gap between OHEM and random sampling decreases as M increases. We believe this is because OHEM has a higher chance of selecting similar negative instances. When M is large, this issue can be mitigated to some extent, but the performance still falls behind ours by a large margin. This suggests the importance of having a diverse set of negative samples, which is unique in our approach.
Comparisons with SOTA. Table shows our approach outperforms various self-supervised ap- proaches on action recognition. For fair comparisons, we group the SOTA approaches by different pretraining dataset sizes, i.e. small-scale (UCF/HMDB), medium-scale (Kinetics), and large-scale (AudioSet). Our gains are calculated according to this grouping. As we can see, our approach outperforms SOTA approaches across all groups. Compared with GDT (Patrick et al. 2020), the current top performing model on cross-modal self-supervised learning, our model outperforms by 1.6 % on UCF101 and 1.1 % on HMDB51. Table 5 shows audio classification transfer results. For Kinetics and AudioSet (240K), by 0.1 % and 1.8% on Kinetics and AudioSet 240K, respectively. Our approach also outperforms AVID (89.2%) 2%) pretrained on AudioSet (1.8M) by 1.6%.
model outperforms the current state-of of-the--rt...tt AVID (79.1%)
8

---

Published as a conference paper at ICLR 2021
Method
Architecture
Pretrained (size)
UCF101
HMDB51
Scratch
3D-ResNet18
46.5
17.1
Supervised (Patrick et al.
2020
R(2+1)D-18
Kineticx400 (N/A)
95.0
ShufilAL (Misra et al. 2016)
CaffeNet
UCF/HMDB
50.2
18.1
DRL Buchler
2018
CaffeNet
UCF/HMDB
58.6
25.0
OPN
VGG
UCF/HMDB
59.8
23.8
DPC Han ct al. 2019)
3D-ResNet18
UCF101
60.6
MotionPred Wang et al.
2019ap
C3D
Kinetics400 (NIA)
61.2
33.4
RotNet3D Jing 11an
2018
3D-ResNet18
Kinetics400 (N/A)
62.9
33.7
ST-Puzzle Kim
2019
3D-ResNet18
Kinetics400 (N/A)
65.8
33.7
ClipOrder (Xu al.
2019)
R(2+1)D-18
Kinetics400 (N/A)
72.4
30,9
CBT Sun ét al.
201 12
S3D & BERT
Kinetics600 (500K)
79.5
44.6
DPC
Ham et al.
2019)
3D-ResNe(34
Kinetics400 (306K)
75.7
35.7
SeLaVi
Asano
2020
R(++1)D-18
Kinetics400 (240K)
83.1
47.1
AVTS
Korbar el al.
2018
MC3
Kinetics400 (240K)
XDC
Alwassel et al.
2019
R(2+1)D-18
Kinetics400 (240K)
84.2
47.1
AVID
Morgado et al
20220
R(2+1)D-18
Kinetics400 (240K)
87.5
60.8
GDT
Patrcck et
2020)
R(++IDD-18
Kinetics400 (N/A)
89.3
60.0
AVTS
Korbar et
2018
MC3
AudioSet (240K)
86.4
AVTS
Korbar et al.
2018
MC3
AudioSet (1.8M)
89.0
61.6
XDC
Awassel et
2019
R(++I)D-18
AudioSet (1.8M)
91.2
61.0
AVID
Morgado et 2020
R(2+1)D-18
AudioSet (1.8M)
91.5
64.7
GDT
Patrick
2020
R(2+1)D-18
AudioSet (1.8M)
92.5
66.1
3D-ResNet18
UCFI01
69.1 (++.5)
33.3 (+8.3)
3D-ResNet18
Kine.-Sound (14K)
77.2 (+16.66)
40.6 (+15.6)
3D- 3D-ResNet11
Kinetics700 (240K)
90.2 (+0.9)
61.8 (+1.0)
Ours
8
AudioSet (240K)
90.7 (+1.4)
62.3 (+1.5)
3D-ResNet18
AudioSet (1.8M)
(+1.6)
66.8 (+0.7)
R(2+1)D-18
AudioSet (1.8M)
93.5 (+1.0)
67.2 (+1.1)
Table 4: Comparison of SOTA approaches action recognition. We specify pretraining dataset and the number of samples used if they are reported in the original papers (N/A: not available).
Method
Architecture
Pretrained on (size)
ESC50
Random Forest
Piczak
2015b)
ESC50
44.3
Piezak ConvNct (Piezak
2015a
ConvNet-4
ESC50
64.5
ConvRBM (Sailor et al
2017)
ConyNet-4
ESC50
86.5
SoundNet (Aytar
2016)
ConvNet-8
SoundNet (2M+)
74.2
L (Arandjelovic Zisserman
2017)
ConvNet-8
SoundNet (500K)
79.3
AVTS (Korbar et al. 20188
VGG-8
Kinetics (240K)
76.7
XDC
Alwassel
2019
ResNet-18
Kineties (240K)
78.0
AVID
Morgado al.
2020
ConvNet-9
Kineties (240K)
79.1
AVTS
Korbar et al.
2018
VGG-8
AudioSet (1 (1.8))
80.6
XDC
Alwassel
al. 2019
ResNet-18
AudioSet (1.8M)
84.8
AVID
Morgado al.
2020
ConvNet-9
AudioSet (1.8M)
89.2
GDT
atrick et al.
2020
ResNet-9
AudioSet (1.8M)
88.5
Kinetics700 (240K)
80.2 (+1.1)
Ours
ResNet-18
AudioSet (240K)
80.9 (+1.8)
AudioSet (1.8M)
90.8 (+1.6)
Table 5: Comparision of SOTA approaches on audio event classification.
CONCLUSION
We have shown that random sampling could be detrimental to contrastive learning due to the redun- dancy in negative samples, especially when the sample size is large, and have proposed an active sampling approach that yields diverse and informative negative samples. We demonstrated this on learning audio-visual representations from unlabeled videos. When pretrained on AudioSet, our ap- proach outperforms previous state-of-hhe-ee of-the-art self- supersueeeeieed approaches various audio and visual downstream benchmarks. We also show that our active sampling approach significantly improves the performance of contrastive learning random and online hard negative sampling approaches.

---

Published as a conference paper at ICLR 2021
REFERENCES
Humam Alwassel, Dhruv Mahajan, Lorenzo Torresani, Bernard Ghanem, and Du Tran. Self- supervised learning by cross-modal audio-video clustering. arXiv preprint arXiv:1911. 2019.
Relja Arandjelovic and Andrew Zisserman. Look, listen and learn. In ICCV, 2017.
Relja Arandjelovic and Andrew Zisserman. Objects that sound. In ECCV, 2018.
Sanjeev Arora, Hrishikesh Khandeparkar, Mikhail Khodak, Orestis Plevrakis, and Nikunj Saunshi. A theoretical analysis of contrastive unsupervised representation learning. In ICML, 2019.
David Arthur and Sergei Vassilvitskii. K-means++: The advantages of careful seeding. In SODA, 2007.
Yuki M Asano, Mandela Patrick, Christian Rupprecht, and Andrea Vedaldi. Labelling unlabelled videos from scratch with multi-modal self-supervision. arXiv preprint arXiv:2006. 2020.
Jordan T Ash, Chicheng Zhang, Akshay Krishnamurthy, John Langford, and Alekh Agarwal. Deep batch active learning by diverse, uncertain gradient lower bounds. In ICLR, 2020.
Yusuf Aytar, Carl Vondrick, and Antonio Torralba. Soundnet: Learning sound representations from unlabeled video. In Advances in neural information processing systems, 2016.
Mohamed Ishmael Belghazi, Aristide Baratin, Sai Rajeshwar, Sherjil Ozair, Yoshua Bengio, Aaron Courville, and Devon Hjelm. Mutual information neural estimation. In ICML, 2018
Uta Buchler, Biagio Brattoli, and Bjorn Ommer. Improving spatiotemporal self-supervision by deep reinforcement learning. In ECCV, 2018.
Yue Cao, Zhenda Xie, Bin Liu, Yutong Lin, Zheng Zhang, and Han Hu. Parametric instance clas- sification for unsupervised visual feature learning. Advances in Neural Information Processing Systems, 33, 2020.
Joao Carreira and Andrew Zisserman. Quo vadis, action recognition? a new model and the kinctics dataset. In CVPR, 2017.
Joao Carreira, Eric Noland, Chloe Hillier, and Andrew Zisserman. A short note on the kinetics-700 human action dataset. arXiv preprint arXiv:1907.06987, 2019.
Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. JCML, 2020.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: Pre-training of deep bidirectional transformers for language understanding. In ACL, 2019.
Debidatta Dwibedi, Yusuf Aytar, Jonathan Tompson, Pierre Sermanet, and Andrew Zisserman. Tem- poral cycle-consistency learning. In CVPR, 2019.
Ariel Ephrat, Inbar Mosseri, Oran Lang, Tali Dekel, Kevin Wilson, Avinatan Hassidim, William T Freeman, and Michael Rubinstein. Looking to listen at the cocktail party: A speaker-independent audio-visual model for speech separation. ACM Transactions on Graphics, 2018.
Fartash Faghri, David J Fleet, Jamie Ryan Kiros, and Sanja Fidler. VSE++: Improving visual- semantic embeddings with hard negatives. In BMVC, 2017.
Satoru Fujishige. Submodular functions and optimization. 2005.
Chuang Gan, Deng Huang. Hang Zhao, Joshua Tenenbaum, and Antonio Torralba. Music gesture for visual sound separation. In CVPR, 2020.
Ruohan Gao and Kristen Grauman. 2.5D visual sound. In CVPR, 2019a.
Ruohan Gao and Kristen Grauman. Co-separating sounds of visual objects. In ICCV, 2019b.
10

---

Published as a conference paper at ICLR 2021
Ruohan Gao, Rogerio Feris, and Kristen Grauman. Learning to separate object sounds by watching unlabeled video. In ECCV, 2018.
Jort F Gemmeke, Daniel PW Ellis, Dylan Freedman, Aren Jansen, Wade Lawrence, R Channing Moore, Manoj Plakal, and Marvin Ritter. Audio set: An ontology and human-labeled dataset for audio events. In ICASSP, 2017.
Walter R Gilks, Sylvia Richardson, and David Spiegelhalter. Markov chain Monte Carlo in practice. Chapman and Hall/CRC, 1995.
Michael Gutmann and Aapo Hyvärinen. Noise-contrastive estimation: new estimation principle for unnormalized statistical models. In AISTATS, 2010.
Tengda Han, Weidi Xie, and Andrew Zisserman. Video representation learning by dense predictive coding. In ICCV, 2019.
Wangli Hao, Zhaoxiang Zhang, and He Guan. Cmegan: A uniform framework for cross-modal visual-audio mutual generation. In AAAI, 2018.
Kensho Hara, Hirokatsu Kataoka, and Yutaka Satoh. Can spatiotemporal 3d cnns retrace the history of 2d cnns and imagenet? In CVPR, 2018.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. CVPR, 2016.
Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. Momentum contrast for unsupervised visual representation learning. In CVPR, 2020.
Olivier J Hénaff, Ali Razavi, Carl Doersch, SM Eslami, and Aaron van den Oord. Data-efficient image recognition with contrastive predictive coding. arXiv preprint arXiv:/905.09277 2019.
R Devon Hjelm, Alex Fedorov, Samuel Gavoic-Marchildee Karan Grewal, Phil Bachman, Adam Trischler, and Yoshua Bengio. Learning deep representations by mutual information estimation and maximization. In ICLR, 2018.
Sergey Ioffe and Christian Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. arXiv preprint arXiv:1502.03167, 2015.
Ahmet Iscen, Giorgos Tolias, Yannis Avrithis, and Ondřej Chum. Mining manifolds: Metric learning without labels. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 7642-7651, 2018
Longlong Jing and Yingli Tian. Self-supervised spatiotemporal feature learning by video geometric transformations. arXiv preprint arXiv:/811.111 2018.
Will Kay, Joao Carreira, Karen Simonyan, Brian Zhang, Chloe Hillier, Sudheendra Vijaya- narasimhan, Fabio Viola, Tim Green, Trevor Back, Paul Natsev, et al. The kinetics human action video dataset. arXiv preprint 1eXiv:/70.00690 2017.
Dahun Kim, Donghyeon Cho, and In So Kweon. Self-supervised video representation learning with space-time cubic puzzles. In AAAI, 2019.
Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint vrXiv:/4/2.66 6,880, 2014.
Bruno Korbar, Du Tran, and Lorenzo Torresani. Cooperative learning of audio and video models from self-supervised synchronization. In Advances Neural Information Processing Systems, 2018.
Hildegard Kuehne, Hueihan Jhuang, Estibaliz Garrote, Tomaso Poggio, and Thomas Serre. Hmdb: large video database for human motion recognition. In ICCV, 2011.
Alex Kulesza and Ben Taskar. k-dpps: Fixed-size determinantal point processes. In ICML, 2011.

---

Published as a conference paper at ICLR 2021
Brenden M Lake, Ruslan Salakhutdinov, and Joshua B Tenenbaum. Human-level concept learning through probabilistic program induction. Science, 330(6266):1332--13333 2015.
Hsin-Ying Lee, Jia-Bin Huang, Maneesh Singh, and Ming-Hsuan Yang. Unsupervised representa- tion learning by sorting sequences. In ICCV, 2017.
Gen Li, Nan Duan, Yuejian Fang, Ming Gong, Daxin Jiang, and Ming Zhou. Unicoder-vl; A universal encoder for vision and language by cross-modal pre-training. In AAAI, 2020.
Liunian Harold Li, Mark Yatskar, Da Yin, Cho-Jui Hsieh, and Kai-Wei Chang. VisualBERT: A simple and performant baseline for vision and language. arXiv preprint (rxxv::908.0355/// 2019.
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoee, and Veselin Stoyanov. Roberta: A robustly optimized bert pretraining approach. arXiv preprint 1907 2019.
Jiasen Lu, Dhruv Batra, Devi Parikh, and Stefan Lee. ViLBERT: Pretraining Task-Agnostic Visi- olinguistic Representations for Vision-and-Language Tasks, In Advances in neural information processing systems, 2019.
Odile Macchi. The coincidence approach to stochastic point processes. Advances in Applied Prob- ability, 7(1), 1975.
David McAllester and Karl Stratos. Formal limitations on the measurement mutual information. In AISTATS, 2020.
Ishan Misra, C. Lawrence Zitnick, and Martial Hebert. Shuffle and learn: Unsupervised learning using temporal order verification. In ECCV, 2016.
Pedro Morgado, Nuno Vasconcelos, and Ishan Misra. Audio-visual instance discrimination with cross-modal agreement. arXiv preprint arXiv:2004.12945 2020.
Hyconseob Nam and Bohyung Han. Learning multi-domain convolutional neural networks for vi- sual tracking. In CVPR, 2016.
George L. Nemhauser, Laurence A Wolsey, and Marshall L. Fisher. An analysis of approximations for maximizing submodular set functions-i. Mathematical programming, 14(1), 1978.
Aäron den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predic- tive coding. arXiv preprint arXiv:1807.0374.. 2018.
Andrew Owens and Alexei A Efros. Audio-visual scene analysis with self-supervised multisensory features. In ECCV, 2018
Andrew Owens, Jiajun Wu, Josh H McDermott, William T Freeman, and Antonio Torralba. Ambient sound provides supervision for visual learning. ECCV, 2016.
Sherjil Ozair, Corey Lynch, Yoshua Bengio, Aaron Van den Oord, Sergey Levine, and Pierre Ser- manet. Wasserstein dependency measure for representation learning. In Advances in Neural Information Processing Systems, 2019.
Jiangmiao Pang, Kai Chen, Jianping Shi, Huajun Feng, Wanli Ouyang, and Dahua Lin. Libra R- CNN: Towards balanced learning for object detection. In CVPR, 2019.
Mandela Patrick, Yuki M Asano, Ruth Fong, João F Henriques, Geoffrey Zweig, and Andrea Vedaldi. Multi-modal self-supervision from generalized data transformations. arXiv preprint AXXi:2003304298, 2020.
Karol J Piczak. Environmental sound classification with convolutional neural networks. In Interna- tional Workshop on Machine Learning for Signal Processing (MLSP), 2015a.
Karol Piczak. ESC:
Dataset for envirommental sound classification. In Proceedings of the 23rd ACM international conference Multimedia, 2015b.
12

---

Published as a conference paper at ICLR 2021
Hardik B Sailor, Dharmesh M Agrawal, and Hemant A Patil. Unsupervised filterbank learning using convolutional restricted boltzmann machine for environmental sound classification. In IN- TERSPEECH, 2017.
Steffen Schneider, Alexei Baevski, Ronan Collobert, and Michael Auli. wav2vee: Unsupervised pre-training for speech recognition. arXiv preprint arXivv1904.05500 2019.
Pierre Sermanet, Corey Lynch, Jasmine Hsu, and Sergey Levine. Time-contrastive networks: Self-
supervised learning from multi-view observation. In CVPRW, 2017.
Burr Settles. Active learning literature survey. Technical report, University of Wisconsin-Madisst Department of Computer Sciences, 2009.
Abhinav Shrivastava, Abhinav Gupta, and Ross Girshick. Training region-based object detectors with online hard example mining. In CVPR, 2016.
Khurram Soomro, Amir Roshan Zamir, and Mubarak Shah. Ucf101: A dataset of 101 human actions classes from videos in the wild. arXiv preprint arXiv: 12 22.. 12.04002 2012.
Weijie Su, Xizhou Zhu, Yue Cao, Bin Li, Lewei Lu, Furu Wei, and Jifeng Dai. VI-bert: Pre-training of generic lingul-in representations. arXiv preprint 1508::9900088300 2019.
Chen Sun, Fabien Baradel, Kevin Murphy, and Cordelia Schmid. Contrastive bidirectional trans- former for temporal representation learning. arXiv preprint arXiv:1906. 05743, 2019a.
Chen Sun, Austin Myers, Carl Vondrick, Kevin Murphy, and Cordelia Schmid. Videobert: A joint model for video and language representation learning. In ICCV, 2019b.
Hao Tan and Mohit Bansal. LXMERT: Learning cross-modality encoder representations from trans- formers. arXiv preprini arXiv: wrXiv::908.074 077900000744 2019.
Yonglong Tian, Dilip Krishnan, and Phillip Isola. Contrastive multiview coding. arXiv preprint
UrXiv:/9066 1906.0584. 2019.
Du Tran, Heng Wang, Lorenzo Torresani, Jamie Ray, Yann LeCun, and Manohar Paluri. closer look at spatiotemporal convolutions for action recognition. In CVPR, 2018.
Jiangliu Wang, Jianbo Jiao, Linchao Bao, Shengfeng He, Yunhui Liu, and Wei Liu. Self-supervised spatio-temporal representation learning for videos by predicting motion and appearance statistics. In CVPR, 2019a.
Xiaolong Wang, Allan Jabri, and Alexei Efros. Learning correspondence from the cycle- consistency of time. In CVPR, 2019b.
Mike Wu, Chengxu Zhuang. Milan Mosse, Daniel Yamins, and Noah Goodman. On mutual in- formation in contrastive learning for visual representations. arXiv preprint arXiv:2005.13149, 2020.
Dejing Xu, Jun Xiao, Zhou Zhao, Jian Shao, Di Xie, and Yueting Zhuang. Self-supervised spa- tiotemporal learning via video clip order prediction. In CVPR, 2019.
Karren Yang, Bryan Russell, and Justin Salamon. Telling left from right: Learning spatial corre- spondence of sight and sound. In CVPR, 2020.
Zhilin Yang, Zihang Dai, Yiming Yang. Jaime Carbonell, Russ R Salakhutdinov, and Quoe V Le. XInet: Generalized autoregressive pretraining for language understanding. Advances in neural information processing systems, 2019.
Hang Zhao, Chuang Gan, Andrew Rouditchenko, Carl Vondrick, Josh McDermott, and Antonio Torralba. The sound of pixels. In ECCV, 2018.
Hang Zhao, Chuang Gan, Wei-Chiu Ma, and Antonio Torralba. The sound of motions. In ICCV, 2019.
Yipin Zhou, Zhaowen Wang, Chen Fang, Trung Bui, and Tamara Berg. Visual to sound: Generat- ing natural sound for videos in the wild. In CVPR, 2018.
13

---

Published as a conference paper at ICLR 2021
A DETAILS ON DATA PROCESSING
We preprocess video frames by sampling at 10 FPS and applying random cropping, horizontal flip- ping, gray-scaling, and temporal jittering. We resize video frames to 3-channel images of 224 x 224; we set the clip length to 16 frames during pretraining, and 32 frames during finetuning on down- stream tasks. For audio channel, we extract meeesppets from the raw waveform using the LibROSA library and get 80 x ' matrix with 80 frequency bands; T is proportionate to the length of an audio clip. We then segment the mel-speetr according to the corresponding video clips to ensure temporal synchrony. We treat the mel-spectrograms as an 80-channel ID signal.
As for downstream tasks, we evaluate our models on action recognition using UCF101 (Soomro et al. 2012) and HMDB51 (Kuehne et al. [011], }. and on sound classification using ESC50 (Piczak] 2015b). UCF101 contains 13K video clips from 101 action categories, HMDB51 contains 7K video clips from 51 categories, and ESC50 has 2K audio clips from 50 categories. UCFI01 and HMDB51 have official train/test splits, while ESC50 has 5 splits. We conduct our ablation study using split-1 of each dataset. We report our average performance over all splits when compare with prior work.
B ADDITIONAL EXPERIMENTS
Effeet of mutual information. We investigate the impact of the amount of MI on contrastive learn- ing using the Spatial-MultiOmmnn
dataset (Ozair et al. 2019). It contains paired images (ar, v) of Omniglot characters (Lake et al.] 2015) with each image arranged in an m x n grid (each grid cell is 32 x 32 pixels). Let 4 be the alphabet size for the 1th character in each image, then the MI (zx, y) = Ee logl;- This way, we can easily control the MI by adding or removing characters. We follow the experimental protocol of Ozair et al. (Ozair et al.] 2019), keeping the training dataset size fixed at 50K and using the same alphabet sets: Tifinagh (55 characters), Hiragana (52), Gujarati (48), Katakana (47), Bengali (46), Grantha (43), Sanskrit (42), Armenian (41), and Mkhedruli (41).
(a) 100
(b) 75
XMoCo
SMoCo
70
80
Ours
Mean Accuracy
Mean Accuracy
65
60
60
40
-
55
g
50
20
-0-8
45
40
2
10"
10
Number of Characters
Dictionary Size
Figure 3:
The effect of a) mutual information (Spatial-MultiOmniiloo) and b) dictionary size on the accuracy of classification (UCF101).
Fig. ((a) shows the results as the number of characters (and thus the MI) increases. We that all approaches achieve nearly 99% accuracy with less than 3 characters; this is the case when the exponent of the MI is smaller than the dataset size (50K), i.e., ( = 55 with one char- acter, ( 2, 860 with 2 characters. However, starting from 3 characters, the performance of the regular MoCo (SMoCo) drops significantly; this is because the exponent of the MI (=137,280) (5x52x48)) much larger than the dataset size. Although our model also drops performance when the MI is increased, it outperforms the other approaches by a large margin. We also observe that XMoCo outperforms SMoCo in mild conditions (1-5 characters) but performs nearly the same as SMoCo with severe conditions (6-9 characters). This suggests that, while cross- modal prediction helps to learn good representations, it also suffers with the same issue when the MI is large, thus adopting active sampling is beneficial.
Effect of dictionary size. Fig. 3 (b) shows how the dictionary size affects downstream task per- formance. Here pretrain model Kinetics-700 and finetune it on UCF-101. Overall, all
14

---

Published as a conference paper at ICLR 2021
three approaches benefit from large dictionaries up threshold (at about 109), which is consistent with previous empirical findings (He et al, 2020). However, both XMoCo and SMoCo starts deteri- orating performance after about 10* (which is consistent with previous theoretical claims of Arora et. al (Arora et al., 2019)). whereas ours do not suffer even after 10a. This suggests that there are performance limits by simply increasing the size of a randomly-sampled dictionary, and also shows the benefit of our active sampling approach.
Effect of pretraining dataset sizes. We investigate the effects of the size of pretraining datasets, using Kinetics-Sound (22k), Kinetics (240K), and AudioSet (1.8M). We vary pretraining conditions while using the same protocol to finetune the models end-to-end on downstream tasks.
Table 6 shows that our model benefits from pretraining on video data, and that the performance improves as we use large pretraining video dataset (Kinetics and AudioSet) than the relatively smaller dataset (Kinetics-Sound) Notably, our approach even outperforms the fully-supervised pre- training approaches by pretraining on a larger video dataset (1.0%, 3.6%, and 8.5% improvement on UCF101, HMDB51, and ESC50, respectively.)
Approach
Dataset
UCFI01
HMDB51
ESC50
ImageNet (1.2M)
82.8'
46.71
Kinetics-Sound (22K)
86,9"
53.1"
78.3*
Supervised
Kinetics400 (240K)
93.17
63.61
82.3"
Kinetics-Souma (22K)
77.2
40.6
77.3
CM-ACC
Kineties700 (240K)
90.2 (-2.9)
61.8 (-1.8)
79.2 (-3.1)
AudioSet (1.8M)
94.1 (+1.0)
67.2 (43.6)
90.8 (48.5)
Table 6: Top-1 accuracy of CM-ACC pretrained on different datasets vs. sully-supervieed counter- parts (Supervised). I; the results are excerpted from Patrick et al. (2020). *; our results.
Diversity of random vs. active sampling. To compare the diversity of the chosen negatives by ran- dom vs. active sampling, we plot the probability of them on sampling of unique negatives (instances from different categories). The more categories the samples come from, we get more diverse and less redundant samples. We train these UCF-101 over 300 iterations with different mini-bater sizes, M E {32, 64, 128). As shown in Figure 4 the active sampling selects more categories than random sampling across all three mini-batch sizes. At M 128, active sampling (with gradient em- bedding) covers 60-70% of categories on UCF101, which is substantially more diverse than random sampling (30-40%)
80
Active Sampling (Feat, Embed.] Active Sampling (Grad. Embed))
Random Sampling
70
60
56
Probabiliy 186)
50
51
40
38
30
30
31
Figure 4: Probability of sampling unique nega- tives (instances from different categories) in the random vs. active sampling conditions. We com- pute the probabilities by averaging the number of unique categories across iterations and dividing them by their batch size.
20
10
128
Batch Size
C VISUALIZATION OF NEGATIVE INSTANCES
Figure shows negative instances selected by active sampling and random sampling when audio clips as the query. We visualize the center frames of the selected video clips. We can see that approach selects more challenging examples than the random sampling approach. For instance,
15

---

Published as a conference paper at ICLR 2021
given query opening bottle, approach selected video clips from the same similar semantic categories, e.g. drinking shot and opening bottle. Given snowboarding, approach selected more video clips related to categories containing the snow scene, e.g. ice f fishing snow kitinng, and tobogganing.
Audio Query: Cheerleading
Grad. Norm 1e-5
Active
200
Jumpstyle Dancing
Salsa
Shooting Goal
Shooting Goal
Skydiving
Yoga
Dancing
Random
50
Samples
Singing
Sit-up
Throwing Axe
Throwing Axe
Tobogganing
Tobogganing
Active
Rando
Audio Query: Opening
Active
250
Drinking
Drumming Fingers
Eating Chips
Opening Bottle
Opening Bottle
Texting
Shots
100
Random
Samples
Blowing
Blowing Leaves
Cheer- leading
Playing Tennis
Sit-up
Sit-up
Active
Rando
Audio Query: Playing Guitar
Grad, Norm 1e-5
Active
Playing
Playing Violin
Riding Bike
Riding a Bike
Tapping Guitar
Unboxing
Recorder
Random
4
Samples
Getting Tattoo
Reading a Book
Reading Book
Reading a Book
Riding a Camel
Riding Camel
Audio Query: Playing Tennis
Active
High
Hurling
American Football
Shooting Basketball
Stretching Leg
Zumba
Fiving
100
Random
Samples
Helmet Diving
Picking Apples
Picking Apples
Picking Apples
Taking Photo
Taking Photo
Active
Rando
Figure 5: Center frames of video clips and their gradient norms selected by active sampling and random sampling.
Furthermore, we also find that our approach selects more diverse negative samples. For exam- ple, given query snowboarding, active sampling selected video clips from 4 different cate- gories related to the snow scene: (ice fishing, playing ice hockey, snow kiting, and tobogganing). In comparison, the random sampling approach yields fewer semantic cate-
16

---

Published as a conference paper at ICLR 2021
gories in general. This suggests that active sampling approach produces more "challenging" and 'diverse" negative instances than the random sampling approach.
To clearly investigate the relationship between negative samples and their gradient magnitudes, show the gradient norm of each visualized sample in Figure 3 We can see that hard negatives tend to have larger gradient norms than easy negatives. Given query P Playing guitar, video clips containing the concept of "playing instruments" " yield the higher gradient norms, i.e. playirs violin (333.87) and tapping guitar (301.35), while concepts that are easy discriminate, e.g. riding a camel yield a significantly smaller gradient norm (5.92). This provides evidence showing the gradient magnitude is effective in measuring the uncertainty of the current model, i.e., highly-uncertaie samples (hard negatives) tend to yield gradients with larger magnitudes, while highly-confident samples (easy negatives) tend to have smaller gradient magnitudes.
D WHEN WOULD CROSS-MODAL CONTRASTIVE LEARNING FAIL?
In general, cross-modal vidco representation learning based on an assumption that the natural cor- respondence between audio and visual channels could serve as useful source of supervision. While intuitive, this assumption may not hold for certain videos in-the-wir which may cause the model to learn suboptimal representations. To investigate when our approach succeeds and fails, we con- duct post-hoc analysis by using thehttps//www.ovvrieee wttp://vereeaa..coooooorrrr/eeee ground-truth semantic category labels provided in Kinetics-700 (Carreira et al.) 2019) (which is not used during pretraining). Specifically, we use our pretrained model to solve the audio-visual contrastive pretext task (7) in the main paper) and keep track of the prediction results (cor- rect/incorrect). We then average the pretext task accuracy over 100 randomly chosen samples for each action category.
7bc0001 1e5da8
Audio Accuracy
Video Accuracy
Raying Rane
Paying Basa Guiar
Haying Gutar
Paying Billards
Beying Violn
Reying Drums
Squasn/Raquettal
Danding
Presenting Weather Forecas:
Crying
Sied Deg Racing
Slapping
Recording
Singing
Cutting Watermelon
Playing Vidlih
Karaoke
Playing Piano
Nachox
Laughing
Fising
Hugging
Waving Hand
Pumping Gas
looking at Phone
Biding like
Tossing
Skydhinng
Snitfing
Shaking Head
Assuraty
Auxuracy
Figure 6: Distribution of Kinetics-700 (Carreira al. 2019) categories sorted by the prediction accuracy.
Figure 6 shows the top-10 and bottom-5 classes by using both audio (left) and video (right) as the query. We observe that, the top ranked classes for both audio and video are the activities that have highly correlated visual-audio signals. For instance, playing bass guitar, play piano, and play violin are all activities related to music. The correlation of audio-visual signals for these activities are obvious; such highly correlated signals are easier to be learned in a cross-modal manner. On the contrary, the bottom ranked classes are those that have subtle audio-visual correla- tion, e.g. tossing coin, shaking hand, looking at phone, and hugging. We also investigate the distribution of hard-easy classes with that reported in Kinetics-700 (Carreira et al.) 2019) learned by the 13D-RGB model (Carreira Zisserman 2017). Interestingly. we find that some hard classes (e.g. karaoke and recording music) are listed in top ranked classes. We suspect that, when only learned within visual modality, some classes with cluttered or complected spatial information will bring difficulties for classification. While, cross-modal approach can
17

---

Published as a conference paper at ICLR 2021
leverage information from both auditory and visual information, model does not limited by such problems.
Algorithm Cross-Modal Active Contrastive Coding (Detailed version of Algorithm 1)
1: Require: Audio-visual clips A, V: encoders Jus Jas by, has dictionary size K; pool size N: batch size M
2: Initialize parameters, 07: 0%. q: Of - Uniform((O, 1)
3: Draw random dictionary. D, <- {un,..,, 2x3 ~n Randoon(V). Dr <- far..... ~ Random) A)
4: Encode dictionary samples, k." -ol(e)), Vos E Du, Kl (a())). ). Va, Da
5: for epoch
#epochs: do
6: Draw random pool, U, + (0i,-... UNy } ~ Random(V), Uo + fa1,... ,N) ~ Ramdom(A)
7: Encode pool samples, K + (u(V)), Ven E Uo. Ki + ha(an)) van E Un
8: for t = to #mini-batches: do
Draw mini-batch, B, < (v1,.... ,
Ba t fai.".. am) Active sampling of negative video keys for D. Encode mini-batch samples, of + fa(a.), Va, E Ba for Yu, E U,, do
10:
11:
12:
exp(h% off) EES ( of Vj E M1
13:
Compute posterros p(i"|en Ba)
14:
Compute pseudo-label, 0% + arg max (Unl end for
15:
16:
Compute gradient, go., Jilat
a_ Los (p(0%) ); Di) lo=0g, Vn N] Obtain S. + k-MEANS (fg.., Un E U., Du #seeds = M)
17:
18:
Update D., + ENQUEUE(DEQUEUE() (..) D Active sampling of negutive audio keys for Da
19:
20:
Encode mini-batch samples, qi [u(u)), Vu, E B for Va,, E U. D.: do
21:
22:
Compute pscudo-posterior, p(i (a, B₂) t EM exp((? Vj E [1, M]
exp(A)
23:
Compute pscudo-label, D2 + mrg p(0 end for Compute gradient, Ia. + D. LoE (p(82) ), Ini la=ay, Vn E [1
24:
25:
Jitast
26:
Obtain S. <- K-MEANS IN:T (1 on E U \ Da) #seeds = M)
27:
Update D. EXQUEUE(DEQUEDE((.... Cross-modal contrastive predictive coding
28:
29:
Encode mini-bater batch samples. k. <- It, (vx), Vu; E Br, ki +- ha (as),, Va: E B. exp(už Af/x)
30:
Compute p(vil+) = ΣS= expla '5/77) p(u.)t)
= Σx Σ5
expla ky/r)
(xplof-A7/1)
V [1,
31:
Update model parameters Update 0 + 0% VoLCE(p(y' ( 0g + 09 - VooCc(ppl 1:
32:
33:
Momentum update 6% t mo + (1 m)67. . 0% + mot + (1 m)69
34:
end for
35:
end for
36: return Optimal solution 0- 0g 0 02
Algorithm k-MEANS Seed Cluster Initialization
1: Require: Data X of N samples; number of centroids K
2: Choose one centroid uniformly random, C(0) Random)(
for
K 1:
4: >. Compute a cumulative probability distribution with a probability in proportion to their squared dis-
tances from the nearest centroid that has already been chosen
5: for n 0 to N 1: do
6: Compute the squared distance, D[n] (minn_istt (Xl])
7:
end for
8:
Compute the cumulative prohability distribution, P +- cumsum(D)
sum((D)
9:
b The next centroid is chosen using P(X) as a weighted probability distribution
10: Choose one centroid at random, C(k] <- P(X)
11:
end for
return C containing centroids
18

---

Published as a conference paper at ICLR 2021
Algorithm 4 Cross-Modal Contrastive Coding withoul Active Sampling
1: Require: Audio-visuee clips A, V; dictionary D.; encoders fo. fa. lt., ha:
dictionary size K: mini-batch size M; learning rate y,; momentum m
2: Initialize parameters, 0g 0%. g.. of ~ Uniform(O,
3: Load a dictionary at random, Da f [V1, uK ~ Rondom (V)
4: Load dictionary random, D. +- {01,"" , .ak) ~ Randomi A)
5: Encode dictionary samples,
ho Ves E Da, Kf ha (0i), Vou Dr
for epoch = #epoccs: do
7: for = 1 to #mini-batches: do
8: Load mini-batch of visual clips. B. (ui,".. uMA) ~ V
9:
Load mini-batch of audio clips, B_ +- fai. aa A Update dictionaries
10:
11:
Encode mini-batch samples, K. - he (wx), Vu: B.
12:
Encode mini-batch samples, A < ha (a;), Va; E Ba
13:
Update D3 +- FNNUUUE (D.), B+)
14:
Update D. (NQUUUUR(DRQUURURR(... Ba)
Cross-modal contrastive predictive coding
16:
Encode mini-batch samples, 4; < f.((x), Vu; E B.
17:
Encode mini-bate bani- samples, of <- fa(a.) Va, E Ba
18:
Compute the posterior, p(v. [ni, di Dv) Σ explo krr
exply exp(y"-4f/0) 4f/1)
Vi E M1]
19:
Compute the posterior, p(v, las, vi, D.)
explef M
Vi E M1
Σx explo
20:
Update model parameters
21:
Update 0" +- 0% 7 VoLCE(- logr(y") .). D: lo==;
22:
Update 64 + 6G VoLCal log logp(1") "1 Vor (ig))ll- la
23:
Momentum update 0% < mo (1 m)00
24:
Momentum update f% + mdg (1 m)8g
25:
end for
26:
end for
27:
return Optimal solution 0g 02, 0g. 0%
19


Reviews:
[{"review_id": "lXOqMqQM0ew", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes a cross modal (audio and video) contrastive learning scheme to pretrain on one of the modalities. They also propose a sampling scheme for the negative examples. \n\nThe experiments and the corresponding ablations suggest that the proposed methods is helping overall. The table 1 suggests that adding the proposed components (cross-modal pretraining, active sampling for negative examples) seem to help overall. \n\nAlso, the samples in the appendix seem to suggest that the proposed `'active negative sampling``'  scheme seems to provide more relevant negative samples. Figure 2 in the experiments also show that the proposed negative sampling mechanism seem to increase the diversity in the negative samples.  Figure 2, also showcases that the proposed negative sampling scheme works better than picking out examples by looking at feature embeddings. This suggests that the proposed negative sampling scheme based on looking at gradients is advantageous over a more standard feature-level approach. \n\nI am not entirely sure if table 2 is exhaustive on the state-of-the-art (SOTA), but from what I gather from the narrative of the paper, they seem to improve the SOTA numbers on three different pretraining dataset size regimes for action recognition datasets. They also provide a similar experiment for audio event classification in table 3, and claim SOTA results. (where they pretrain on different datasets, and test on another) \n\nI think the clarity of the manuscript could be improved, especially in Section 3.3. I think that perhaps you could replace the low level algorithm with a higher-level pseudocode or a diagram and take the algorithm to the supplementary materials? \n\nAlso, my understanding of the proposed dictionary building / sampling approach for negative sampling is that you look at the magnitude of the gradients with respect to the contrastive learning objective, and pick the examples which give a large gradient magnitude. I think that this could be made clearer for the reader, perhaps by including a toy example, or including a figure in the appendix which showcases the gradient magnitude vs. the chosen negative examples. ", "extra_scores": {}, "extra_text": {}}, {"review_id": "ZCHW3Y6k0d", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The goal of the paper is audio-visual self-supervised learning using an active sampling technique to mine hard negatives during training. \n\nStrengths \n- The paper is well written and clearly explained. \n- The strategy for selecting negatives based on diversity seems well reasoned and experimentally outperforms random sampling and OHEM. I like the study showing empirically more categories covered in active sampling vs random sampling. \n\nWeaknesses: \n- It would be nice to see some discussion of the other self-supervised contrastive works that also focus on the optimal selection of “negatives”: eg. Korbar et al 2018 (where they use a curriculum based on time distance of negatives from the positives), Iscen et al. 2018: https://arxiv.org/abs/1803.11095, Cao et al.2020: https://arxiv.org/abs/2006.14618, Wu et al 2020: https://arxiv.org/abs/2005.13149 (variational extension to InfoNCE with modified strategies for negative sampling). In a similar vein, the comparison to OHEM in the supplementary is quite nice and I believe should be in the main paper. \n- Why is the ablation showing the benefits of Active Sampling only on Kinetics-Sound? Would the same trends shown in Table 1 hold on a random sampled proportion of Kinetics of the same size, or even on the whole dataset?\n- The audio-visual contrastive method has been proposed in numerous papers before, so the novelty of this paper lies solely in the active sampling technique that increases diversity using K-means clustering \n- (Minor) The footnote on page 1 is a bit confusing, it’s hard to see the probability drop from just Fig. 2 since the division needs to be done, probably better to plot the probabilities directly somewhere or remove this. \n- (Minor) It would be interesting to also test the audio representations for classification on VGG-Sound (http://www.robots.ox.ac.uk/~vgg/data/vggsound/) ", "extra_scores": {}, "extra_text": {}}, {"review_id": "T0eKz4-h7sl", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "#### Summary\nIn this paper, the authors propose a cross-modal (audio-video) self-supervised representation learning method with a contrastive learning framework. To overcome the high redundancy in the negative samples, they propose an active negative sampling method. They use a gradient with respect to the pseudo label to measure the uncertainty of a negative sample. They use K-means clustering to maximize the negative sample diversity when constructing a new negative set for queueing. They show their method's efficacy on the public benchmarks: Kinetics, AudioSet for retraining, and UCF-101, HMDB-51, ESC-50 for downstream tasks. \n\n#### Strengths\n* The problem is interesting and relevant to the community. Audio-visual self-supervised learning could be interesting for many researchers in the computer vision community.\n* Design choice looks reasonable and effective. Cross-modal contrastive learning gives a significant boost to performance. Active sampling gives a significant boost. \n* Writing is straightforward and easy to follow.\n\n#### Weaknesses\nHowever, I have a few concerns about this work.\n* To show the effectiveness of the proposed gradient-based active sampling method compared to the embedding-based active sampling, the authors plot the number of action categories vs. the number of training iterations in Figure 2. However, in my opinion, more action categories might not mean high diversity in a negative set. For example, an action could be compositional. \"Drinking coffee\" action consists of \"picking up object\", \"approach an object\", \"bring an object to the actor\". Since the proposed method uses 16 frames, very short, feature encoding, the embeddings will be more towards these subactions, not the entire actions. Therefore, more action categories do not directly mean more diverse negative examples. So I would suggest the authors directly compare the downstream task performance of gradient-based sampling and embedding-based sampling.\n* The comparison of the proposed method with the fully supervised method is unfair. The fully supervised method has VGG-M-2048 as a backbone and pretrained on ImageNet, while the proposed methods have 3D-ResNet-18 or R(2+1)D-18 as a backbone and pertained on Kinetics or AudioSet. It is also unclear which row of \"ours\" the authors compare to the fully supervised method. I cannot find a 7.2p improvement on the UCF-101 and the 14.1p improvement on the HMDB-51 compared to the fully supervised method. It is okay that the proposed self-supervised method is weaker than the fully supervised method. However, the comparison should be fair, in my opinion.\n \nI will increase my rating if the authors adequately address my concerns.\n\n#### Minor comments:\n* 6. Conclusion, fourth row: \"When pertained …\" -> \"When pre-trained …\"?\n\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "JsiKI_obH_6", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "In addition to my review - directly next, I also give this paper to a junior PhD student who I believe did a very good job in the review. There's no disagreement between both reviews, and I have checked both for correctness, so I provide both of them below: [so it's a 2-in-1 review option.]\n\n================\n\nThe paper is straight-forward, easy to read and clearly written. It builds on the Momentum Contrast (MoCo) paper (2020), which aims for a moving average dictionary per modality along with cross-modal contrastive learning for self-supervised tasks, evaluated on audio-visual datasets. The paper takes this a step further by arguing for a more active way to select negative samples that form the dictionary (or negative memory). While dictionaries are efficient ways to perform contrastive learning, this paper showcases that the random selection of negative samples can have a substantive impact on the performance.\nTo resolve this, the paper proposes to select negatives based on two criteria: informativeness measured by the gradients of the sample and diversity (or representativeness of the potential classes) through a greedy (or online) K-means approach. When both are utilised, the result showcases state-of-the-art on downstream recognition tasks on 3 standard datasets. Results demonstrate the effectiveness of the approach.\n\nThe paper is scientifically correct and the proposal is explained both theoretically, with equations, with an algorithm and supported by results. \n\nMy only concern with the paper is that by the end of the introduction, the proposal and concept are already well-stated. The rest of the paper keeps repeating the same argument. Apart from the introduction, only 3.3 offers more details. Related work, sections 3.1 and 3.2, figure and algorithm all repeat the same thing in various forms. While emphasis is good, I found the repetitiveness to affect my interest in reading the paper. I recommend that the authors revisit their manuscript to remove the repetitive argument (e.g. the notion that you need both informative and diverse negative samples is made in 4 different locations). \nInstead, the authors can provide more insight into the negative dictionary for the variety of models and datasets, showcasing the impact on certain classes. They can also discuss failure cases and whether they relate to classes that are still under-represented in active negative sampling. Comparing an online to offline diversity options (to know the upper limit) would also be interesting.\n\nAdditionally, the need for fully connected layers (described in page 6) is only showcased for XMoCo, and not for the final proposal. An ablation of its effect on the final proposal would be important to include.\n\n==============\n\nSummary\n\nThis paper proposes an approach to actively sample negative examples for audio-visual representation learning and combines this with an audio-visual approach for contrastive learning. The paper suggests that an active sampling approach that samples datapoints that are are both diverse and uncertain would result in a much more informative dictionary of negative samples, and uses gradient information and the k-MEANS++ algorithm to create such a dictionary of negatives. They then perform a cross-modal dictionary lookup where the queries and keys for each modality are each represented by a learnt encoder where the parameters of the query encoders are learnt directly via backpropogation and the key encoders' parameters are defined as a moving average of the query encoders' parameters. A contrastive loss is then used to learn a representation that is effective for fine-tuning to down-stream tasks, achieving state-of-the-art results for both action recognition and audio event classification.\n\nStrengths\n\nThe paper presents novel, well motivated results in the field of cross-modal representation learning. It includes detailed comparison to the state-of-the-art across a wide selelection of papers and datasets. The proposed method shows state-of-the art results across small, medium and large pre-trained datasets for both action recognition and audio even classification.\n\nWeaknesses\n\nWhilst there are detailed comparisons between related works the paper does not discuss many of these related works in the related work section. It would be nice to have some more information on the related works, how they relate, how they differ and the gap that this work fits into.\nAdditionaly, some information on the models used for audio and visual encoders would help with understanding. They are described in the appendix but there is no reference to them in the main text. The paper adopts a style wherein the entire process is shown as a single algorithm, and the text describes the algorithm step-by-step. This is, in general, quite clear, however continually changing pages to check the algorithm can decrease readability. Perhaps putting the full algorithm in the appendices and providing a simplified version in the main body would be clearer, or alternatively, splitting the algorithm in to multiple, smaller components.\nThe comparisons to related work are quite detailed, however, in some cases the proposed method offers only a marginal improvement (particularly for the medium and large datasets). Running some experiments multiple times and including error bars would strengthen the state-of-the-art claims. The comparisons could also be improved by consistently comparing across the same architectures. For example, the proposed method is only trained with R(2+1)D-18 for the large dataset, despite the architecture being widely used within the medium dataset comparisons. Perhaps two or three architectures across each dataset size would further strengthen comparisons.\n\nFurther points:\n*) typo: \"meduim scale\"\n*) Random overleaf link in the appendix\n*) references to relevant sections in the appendix would be nice\n", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper focuses on the task of learning audiovisual representations through contrastive learning on unlabelled videos. This work is another addition to the ever-growing literature on self-supervised learning (SSL) with emphasis on video and multi-modal data. The main contribution of this work is the manner in which it tackles a well-known drawback of contrastive learning, namely the strategy used to sample negatives in the contrastive pipeline. The authors propose an active sampling strategy that adaptively chooses negative samples that are informative and diverse. This active selection technique is similar in spirit to many selector functions proposed in the active learning literature. It seems to be the first time it is used for contrastive SSL. 

Based on all the reviews and the subsequent discussions, it seems that the reviewers' comments were addressed. The authors are commended on integrating the reviewers' suggestions and making the necessary edits to the paper in a timely manner. 
Decision: Accept (Poster)