ACTIVE CONTRASTIVE LEARNING OF AUDIO-VISUAL VIDEO REPRESENTATIONS
Year: 2021

Abstract:
Contrastive learning has been shown to produce generalizable representations of audio and visual data by maximizing the lower bound on the mutual information (MI) between different views of an instance. However, obtaining a tight lower bound requires a sample size exponential in MI and thus a large set of negative samples. We can incorporate more samples by building a large queue-based dictionary, but there are theoretical limits to performance improvements even with a large number of negative samples. We hypothesize that random negative sampling leads to a highly redundant dictionary that results in suboptimal representations for downstream tasks. In this paper, we propose an active contrastive learning approach that builds an actively sampled dictionary with diverse and informative items, which improves the quality of negative samples and improves performances on tasks where there is high mutual information in the data, e.g., video classification. Our model achieves state-of-the-art performance on challenging audio and visual downstream benchmarks including UCF101, HMDB51 and ESC50.1

heading: 1 INTRODUCTION
text: Contrastive learning of audio and visual representations has delivered impressive results on various downstream scenarios (Oord et al., 2018; Hénaff et al., 2019; Schneider et al., 2019; Chen et al., 2020). This self-supervised training process can be understood as building a dynamic dictionary per mini-batch, where “keys” are typically randomly sampled from the data. The encoders are trained to perform dictionary look-up: an encoded “query” should be similar to the value of its matching key and dissimilar to others. This training objective maximizes a lower bound of mutual information (MI) between representations and the data (Hjelm et al., 2018; Arora et al., 2019). However, such lower bounds are tight only for sample sizes exponential in the MI (McAllester & Stratos, 2020), suggesting the importance of building a large and consistent dictionary across mini-batches.
Recently, He et al. (2020) designed Momentum Contrast (MoCo) that builds a queue-based dictionary with momentum updates. It achieves a large and consistent dictionary by decoupling the dictionary size from the GPU/TPU memory capacity. However, Arora et al. (2019) showed that simply increasing the dictionary size beyond a threshold does not improve (and sometimes can even harm) the performance on downstream tasks. Furthermore, we find that MoCo can suffer when there is high redundancy in the data, because only relevant – and thus limited – parts of the dictionary are updated in each iteration, ultimately leading to a dictionary of redundant items (we show this empirically in Fig. 3). We argue that random negative sampling is much responsible for this: a randomly constructed dictionary will contain more “biased keys” (similar keys that belong to the same class) and “ineffective keys” (keys that can be easily discriminated by the current model) than a carefully constructed one. Furthermore, this issue can get aggravated when the dictionary size is large.
In this paper, we focus on learning audio-visual representations of video data by leveraging the natural correspondence between the two modalities, which serves as a useful self-supervisory signal (Owens & Efros, 2018; Owens et al., 2016; Alwassel et al., 2019). Our starting point is contrastive learning (Gutmann & Hyvärinen, 2010; Oord et al., 2018) with momentum updates (He et al., 2020).
∗Equal contribution 1Code is available at: https://github.com/yunyikristy/CM-ACC
However, as we discussed above, there are both practical challenges and theoretical limits to the dictionary size. This issue is common to all natural data but is especially severe in video; successive frames contain highly redundant information, and from the information-theoretic perspective, audiovisual channels of video data contain higher MI than images because the higher dimensionality – i.e., temporal and multimodal – reduces the uncertainty between successive video clips. Therefore, a dictionary of randomly sampled video clips would contain highly redundant information, causing the contrastive learning to be ineffective. Therefore, we propose an actively sampled dictionary to sample informative and diverse set of negative instances. Our approach is inspired by active learning (Settles, 2009) that aims to identify and label only the maximally informative samples, so that one can train a high-performing classifier with minimal labeling effort. We adapt this idea to construct a non-redundant dictionary with informative negative samples.
Our approach, Cross-Modal Active Contrastive Coding (CM-ACC), learns discriminative audiovisual representations and achieves substantially better results on video data with a high amount of redundancy (and thus high MI). We show that our actively sampled dictionary contains negative samples from a wider variety of semantic categories than a randomly sampled dictionary. As a result, our approach can benefit from large dictionaries even when randomly sampled dictionaries of the same size start to have a deleterious effect on model performance. When pretrained on AudioSet (Gemmeke et al., 2017), our approach achieves new state-of-the-art classification performance on UCF101 (Soomro et al., 2012), HMDB51 (Kuehne et al., 2011), and ESC50 (Piczak, 2015b).

heading: 2 BACKGROUND
text: Contrastive learning optimizes an objective that encourages similar samples to have similar representations than with dissimilar ones (called negative samples) (Oord et al., 2018):
min θf ,θh ExvpX
[ −log ( ef(x;θf ) ᵀh(x+;θh)
ef(x;θf ) ᵀh(x+;θh) + ef(x;θf ) ᵀh(x−;θh)
)] (1)
The samples x+ and x− are drawn from the same distribution as x ∈ X , and are assumed to be similar and dissimilar to x, respectively. The objective encourages f(·) and h(·) to learn representations of x such that (x, x+) have a higher similarity than all the other pairs of (x, x−).
We can interpret it as a dynamic dictionary look-up process: Given a “query” x, it finds the correct “key” x+ among the other irrelevant keys x− in a dictionary. Denoting the query by q = f(x), the correct key by k+ = h(x+), and the dictionary of K negative samples by {ki = h(xi)}, i ∈ [1,K], we can express equation 1 in a softmax form, minθq,θk ExvpX [ −log e q·k+/τ∑K i=0 e q·ki/τ ] , where θq and θk are parameters of the query and key encoders, respectively, and τ is a temperature term that controls the shape of the probability distribution computed by the softmax function.
Momentum Contrast (MoCo) decouples the dictionary size from the mini-batch size by implementing a queue-based dictionary, i.e., current mini-batch samples are enqueued while the oldest are dequeued (He et al., 2020). It then applies momentum updates to parameters of a key encoder θk with respect to parameters of a query encoder, θk ← mθk + (1 −m)θq , where m ∈ [0, 1) is a momentum coefficient. Only the parameters θq are updated by back-propagation, while the parameters θk are defined as a moving average of θq with exponential smoothing. These two modifications allow MoCo to build a large and slowly-changing (and thus consistent) dictionary.
Theoretical Limitations of Contrastive Learning. Recent work provides theoretical analysis of the shortcomings of contrastive learning. McAllester & Stratos (2020) show that lower bounds to the MI are only tight for sample size exponential in the MI, suggesting that a large amount of data are required to achieve a tighter lower bound on MI. He et al. (2020) empirically showed that increasing negative samples has shown to improve the learned presentations. However, Arora et al. (2019) showed that such a phenomenon does not always hold: Excessive negative samples can sometimes hurt performance. Also, when the number of negative samples is large, the chance of sampling redundant instances increases, limiting the effectiveness of contrastive learning. One of our main contributions is to address this issue with active sampling of negative instances, which reduces redundancy and improves diversity, leading to improved performance on various downstream tasks.

heading: 3 APPROACH

heading: 3.1 CROSS-MODAL CONTRASTIVE REPRESENTATION LEARNING
text: Our learning objective encourages the representations of audio and visual clips to be similar if they come from the same temporal block of a video. LetA = {a0, · · · , aN−1} and V = {v0, · · · , vN−1} be collections of audio and visual clips, where each pair (ai, vi) is from the same block of a video. We define query encoders fa and fv and key encoders ha and hv for audio and visual clips, respectively, with learnable parameters {θaq , θvq} for the query encoders and {θak , θvk} for the key encoders. These encoders compute representations of audio and visual clips as queries and keys,
qv = fv(v query), kv = hv(v key), qa = fa(a query), ka = ha(a key) (2)
We train our encoders to perform cross-modal dictionary look-up, e.g., given a query video clip vquery, we find the corresponding audio clip akey from a dictionary Da. Adapting MoCo (He et al., 2020) to our cross-modal setup, we implement a queue-based dictionaryDa that stores keys of audio clips {kai }Ki=1, where K is the dictionary size. We compute the contrastive loss and backpropagate the gradients only to the visual query encoder fv and update the parameters θvq . For the audio encoder ha, we apply the momentum update (He et al., 2020),
θak ← mθak + (1−m)θaq (3) The parameter θaq is not updated in this contrastive coding step; we update it during the audio-tovisual step (similar as above with the opposite modalities). Here we explain the visual-to-audio step only; we perform bi-directional contrastive coding and train the whole model end-to-end.

heading: 3.2 ACTIVE SAMPLING OF NEGATIVE INSTANCES: UNCERTAINTY AND DIVERSITY
text: The quality of negative samples is crucial in contrastive learning. Existing work typically adopts random negative sampling. However, we want a diverse set of negative samples so that comparisons between positive and negative pairs are the most informative they can be. Motivated by active learning (Settles, 2009), we propose a gradient-based active sampling approach to improve the quality of negative samples. In active learning, the learner chooses samples that seem maximally informative and queries an oracle for labels to obtain an optimal solution with a minimal labeling budget. Adapting this to our setting, we can empower the learner to choose the maximally informative negative samples to construct a dictionary; the main question is how to measure the informativeness of samples without labels.
One way to measure informativeness is through the lens of uncertainty: If a model is highly uncertain about its prediction of a sample, we can ensure the maximum update to the model by including the sample in a mini-batch (conversely, if the uncertainly is low for all samples in a mini-batch, the model update will be small). Ash et al. (2020) showed that gradients of a loss function with respect to the model’s most confident predictions can approximate the uncertainty of samples, demonstrating its effectiveness in active learning. They provide a theoretical justification by showing that gradient
norms of the last layer of a neural network with respect to pseudo-labels provides a lower bound on gradient norms induced by any other labels. In this work, we use gradients of the last layer to measure the uncertainty and encourage our model to include samples that have the highest gradient magnitudes to constitute a dictionary.
While the uncertainty of each individual samples is important, the diversity of samples is also a critical measure of informativeness. Intuitively, it is possible that a model is highly uncertain about samples from particular semantic categories, but constructing a mini-batch of samples from just those categories can severely bias gradients and ultimately lead to a bad local minima. There are several principled approaches to ensure diversity, e.g., submodular optimization (Fujishige, 2005) and Determinantal Point Processes (DPP) (Macchi, 1975; Kulesza & Taskar, 2011). Unfortunately, those methods are typically inefficient because of the combinatorial search space (Nemhauser et al., 1978; Gilks et al., 1995). In this work, instead of using the expensive solutions, we opt to the fast solution of Ash et al. (2020) and use the initialization scheme of the k-MEANS++ seeding algorithm (Arthur & Vassilvitskii, 2007) to sample a diverse set of negative samples.

heading: 3.3 CROSS-MODAL ACTIVE CONTRASTIVE CODING
text: Algorithm 1 describes our proposed cross-modal active contrastive coding (we provide a simplified version here; we include a more detailed version and another version without active sampling in Appendix). At a high-level, we initialize the dictionaries Dv and Da with K randomly drawn samples from V and A, respectively (lines 3-4). For each epoch, we construct “negative candidate pools” Uv and Ua with N random samples from V and A, respectively (lines 6-7). For each iteration within an epoch, we actively select the most informative negative samples Sv and Sa from the pools Uv and Ua, respectively, and enqueue them into the dictionaries Dv and Da, respectively (lines 9-21). We then perform cross-modal contrastive coding, update the parameters of query encoders θvq and θ a q via backpropagation, and apply momentum updates to the parameters of key encoders θvk and θ a k (lines 22-27).
Algorithm 1 Cross-Modal Active Contrastive Coding 1: Require: Audio-visual clips A, V ; encoders fv , fa, hv , ha; dictionary size K; pool size N ; batch size M 2: Initialize parameters, θvq , θvk, θ a q , θ a k v Uniform(0, 1)
3: Draw random dictionary, Dv ← {v1, · · · , vK} v V , Da ← {a1, · · · , aK} v A 4: Encode dictionary samples, kvi ← hv(vi), ∀vi ∈ Dv , kai ← ha(ai), ∀ai ∈ Da 5: for epoch = 1 to #epochs: do 6: Draw random pool, Uv ← {v1, · · · , vN} v V , Ua ← {a1, · · · , aN} v A 7: Encode pool samples, kvn ← hv(vn), ∀vn ∈ Uv , kan ← ha(an), ∀an ∈ Ua 8: for t = 1 to #mini-batches: do 9: Draw mini-batch, Bv ← {v1, · · · , vM} v V , Ba ← {a1, · · · , aM} v A
10: . Active sampling of negative video keys for Dv 11: Encode mini-batch samples, qai ← fa(ai), ∀ai ∈ Ba 12: Compute pseudo-labels, ỹvn ← arg max p(ŷvn|vn, Ba), ∀vn ∈ Uv\Dv 13: Compute gradients gvn using the pseudo-labels ỹ v n, ∀n ∈ [1, N ] 14: Obtain Sv ← k-MEANS++INIT ({gvn : vn ∈ Uv\Dv},#seeds = M) 15: Update Dv ← ENQUEUE(DEQUEUE(Dv), Sv) 16: . Active sampling of negative audio keys for Da 17: Encode mini-batch samples, qvi ← fv(vi), ∀vi ∈ Bv 18: Compute pseudo-label, ỹan ← arg max p(ŷan|an, Bv), ∀an ∈ Ua\Da 19: Compute gradients gan using the pseudo-labels ỹ a n, ∀n ∈ [1, N ] 20: Obtain Sa ← k-MEANS++INIT ({gan : an ∈ Ua\Da},#seeds = M) 21: Update Da ← ENQUEUE(DEQUEUE(Da), Sa) 22: . Cross-modal contrastive predictive coding 23: Encode mini-batch samples, kvi ← hv(vi), ∀vi ∈ Bv , kai ← ha(ai), ∀ai ∈ Ba 24: Compute p(yvi |vi, ai, Da) and p(yai |ai, vi, Dv), ∀i ∈ [1,M ] 25: . Update model parameters 26: Update parameters of query encoders θvq and θaq with backpropagation 27: Momentum update parameters of key encoders θvk and θ a k 28: end for 29: end for 30: return Optimal solution θvq , θvk, θaq , θak
Active sampling. To measure uncertainty, we define a pseudo-label space induced by the queries from the other modality, and take the gradient of the last layer of a query encoder with respect to the most confident prediction, which we call the pseudo-label ỹ. For instance, in the case of sampling negative video keys from the pool Uv (lines 10-15), we compute the pseudo-posterior of a video key vn ∈ Uv\Da,
p(ŷvn|vn, Ba) = exp(kvn · qaj )∑M i=1 exp(k v n · qai ) ,∀j ∈ [1,M ] (4)
whereBa is the current mini-batch of audio queries and defines the pseudo-label space. Note that we consider only the samples in Uv\Dv to rule out samples already inDv . Intuitively, this computes the posterior by the dot-product similarity between vn and all qai ∈ Ba, producing an M -dimensional probability distribution. We then take the most confident class category as the pseudo-label ỹvn (line 12) and compute the gradient according to the cross-entropy loss
gvn = ∂
∂θlast LCE (p(ŷvn|vn, Ba), ỹvn) |θ=θaq (5)
where θlast is the parameters of the last layer of θ (in this case, θaq of the audio query encoder ha). Intuitively, the gradient gvn measures the amount of change – and thus, the uncertainty – vn will bring to the audio query encoder ha.
One can interpret this as a form of online hard negative mining: The gradient is measured with respect to the most probable pseudo-label ỹvn induced by the corresponding audio query q a j . When we compute the contrastive loss, the same audio query will be maximally confused by vn with its positive key v+ per dot-product similarity, and vn in this case can serve as a hard negative sample.
Next, we obtain the most diverse and highly uncertain subset Sv ⊆ Uv\Dv using the initialization scheme of k-MEANS++ (Arthur & Vassilvitskii, 2007) over the gradient embeddings gv (line 14). The k-MEANS++ initialization scheme finds the seed cluster centroids by iteratively sampling points with a probability in proportion to their squared distances from the nearest centroid that has already been chosen (we provide the exact algorithm in the Appendix). Intuitively, this returns a diverse set of instances sampled in a greedy manner, each of which has a high degree of uncertainty measured as its squared distances from other instances that have already been chosen. Finally, we enqueue Sv into Dv and dequeue the oldest batch from Dv (line 15). We repeat this process to sample negative audio keys (lines 16-21); this concludes the active sampling process for Dv and Da.
Cross-modal contrastive coding. Given the updated Dv and Da, we perform cross-modal contrastive coding. For visual-to-audio coding, we compute the posteriors of all video samples vi ∈ Bv with respect to the negative samples in the audio dictionary Da,
p(yvi |vi, ai, Da) = exp(qvi · kai /τ)∑K j=0 exp(q v i · kaj /τ) ,∀i ∈ [1,M ] (6)
where the posterior is defined over a cross-modal space with one positive and K negative pairs (line 24). Next, we backpropagate gradients only to the query encoders fv and fa (line 26),
θvq ← θvq − γ∇θLCE(p(yv| · ), yvgt)|θ=θvq , θ a q ← θaq − γ∇θLCE(p(ya| · ), yagt)|θ=θaq (7)
while applying momentum update to the parameters of the key encoders hv and ha (line 27),
θvk ← mθvk + (1−m)θvq , θak ← mθak + (1−m)θaq (8)
The momentum update allows the dictionaries to change their states slowly, thus making them consistent across iterations. However, our cross-modal formulation can cause inconsistency in dictionary states because the gradient used to update query encoders are not directly used to update the corresponding key encoders. To improve stability, we let the gradients flow in a cross-modal fashion, updating part of fv and ha using the same gradient signal from the contrastive loss. We do this by adding one FC layer on top of all encoders and applying momentum update to their parameters. For example, we apply momentum update to the parameters of the FC layer on top of ha using the parameters of the FC layer from fv . We omit this in Alg. 1 for clarity but show its importance in our ablation experiments (XMoCo (w/o fcl) in Table 1).

heading: 4 RELATED WORK
text: Self-supervised learning has been studied in vision, language, and audio domains. In the image domain, one popular idea is learning representations by maximizing the MI between different views of the same image (Belghazi et al., 2018; Hjelm et al., 2018; Tian et al., 2019; He et al., 2020). In the video domain, several approaches have exploited the spatio-temporal structure of video data to design efficient pretext tasks, e.g. by adopting ordering (Sermanet et al., 2017; Wang et al., 2019b), temporal consistency (Dwibedi et al., 2019), and spatio-temporal statistics (Xu et al., 2019; Wang et al., 2019a; Han et al., 2019). In the language domain, the transformer-based approaches trained with the masked language model (MLM) objective has been the most successful (Devlin et al., 2019; Liu et al., 2019; Yang et al., 2019). Riding on the success of BERT (Devlin et al., 2019), several concurrent approaches generalize it to learn visual-linguistic representations (Lu et al., 2019; Li et al., 2020; Su et al., 2019; Tan & Bansal, 2019; Li et al., 2019). CBT (Sun et al., 2019a) and VideoBERT (Sun et al., 2019b) made efforts on adapting BERT-style pretraining for video.
Besides vision and language signals, several approaches learn audio-visual representations in a selfsupervised manner (Owens et al., 2016; Arandjelovic & Zisserman, 2017; Owens & Efros, 2018; Owens et al., 2016). Recently, audio-visual learning has been applied to enable interesting applications beyond recognition tasks, such as sound source localization/separation (Zhao et al., 2018; Arandjelovic & Zisserman, 2018; Gao et al., 2018; Gao & Grauman, 2019a;b; Ephrat et al., 2018; Gan et al., 2020; Zhao et al., 2019; Yang et al., 2020) and visual-to-sound generation (Hao et al., 2018; Zhou et al., 2018). The work of Owens & Efros (2018), Korbar et al. (2018), and Alwassel et al. (2019) are similar in spirit to our own, but our technical approach differs substantially in the use of active sampling and contrastive learning.
Hard negative mining is used in a variety of tasks, such as detection (Li et al., 2020), tracking (Nam & Han, 2016), and retrieval (Faghri et al., 2017; Pang et al., 2019), to improve the quality of prediction models by incorporating negative examples that are more difficult than randomly chosen ones. Several recent work have focused on finding informative negative samples for contrastive learning. Wu et al. (2020) show that the choice of negative samples is critical in contrastive learning and propose variational extension to InfoNCE with modified strategies for negative sampling. Iscen et al. (2018) propose hard examples mining for effective finetuning of pretrained networks. Cao et al. (2020) utilize negative sampling to reduce the computational cost. In the context of audio-visual selfsupervised learning, Korbar et al. (2018) sample negatives under the assumption that the smaller the time gap is between audio and visual clips of the same video, the harder it is to differentiate them (and thus they are considered hard negatives). Our proposed approach does not make such an assumption and estimates the hardness of negatives by directly analyzing the magnitude of the gradients with respect to the contrastive learning objective.

heading: 5 EXPERIMENTS
text: Experimental Setting. We use 3D-ResNet18 (Hara et al., 2018) as our visual encoders (fv and hv) in most of the experiments. We also use R(2+1)D-18 (Tran et al., 2018) to enable a fair comparison with previous work (see Table 4). For audio encoders (fa and ha), we adapt ResNet-18 (He et al., 2016) to audio signals by replacing 2D convolution kernels with 1D kernels. We employ Batch Normalization (BN) (Ioffe & Szegedy, 2015) with the shuffling BN (He et al., 2020) in all our encoders. All models are trained end-to-end with the ADAM optimizer (Kingma & Ba, 2014) with an initial learning rate γ = 10−3 after a warm-up period of 500 iterations. We use the mini-batch
size M = 128, dictionary size K = 30 × 128, pool size N = 300 × 128, momentum m = 0.999, and temperature τ = 0.7. We used 40 NVIDIA Tesla P100 GPUs for our experiments.
We pretrain our model on Kinetics-700 (Carreira et al., 2019) and AudioSet (Gemmeke et al., 2017) when comparing with state-of-the-art approaches. For Kinetics-700, we use 240K randomly selected videos that contain the audio channel. On AudioSet, we use both a subset of 240K randomly selected videos and the 1.8M full set. For our ablation study, we use Kinetics-Sound (Arandjelovic & Zisserman, 2017) that contains 22K videos from 34 classes that are potentially manifested both visually and audibly, and thus provides a relatively clean testbed for ablation purposes. As for downstream tasks, we evaluate our models on action recognition using UCF101 (Soomro et al., 2012) and HMDB51 (Kuehne et al., 2011), and on sound classification using ESC50 (Piczak, 2015b).
Unimodal vs. cross-modal pretraining. To validate the benefits of cross-modal pretraining, we compare it to its unimodal counterparts. We pretrain our model on Kinetics-Sound with a randomly sampled dictionary (similar to MoCo (He et al., 2020)); we call this XMoCo. For the unimodal case, we pretrain two models on visual clips and audio clips, respectively; we call these SMoCo. We also compare ours with a model trained from scratch (Scratch), and a model pretrained on KineticsSound in a fully-supervised manner (Supervised). Lastly, we include XMoCo (w/o fcl) that is identical to XMoCo except that we do not include the additional FC layers on top of the encoders. All these models are finetuned end-to-end on each downstream task using the same protocol.
Table 1 shows the top-1 accuracy of each downstream task. We observe that all the self-supervised models outperform Scratch on all downstream tasks, suggesting the effectiveness of pretraining with contrastive learning. We also see that our cross-modal objective outperforms the unimodal objective (∆( 4©- 3©)). The comparisons between XMoCo vs. XMoCo (w/o fcl) and CM-ACC vs. CM-ACC (w/o fcl) show the effectiveness of the additional FC layer on top of the encoders (∆( 5©- 4©), ∆( 6©- 7©)). When adding the FC layer, the performance further improves on all three benchmarks. This shows the importance of letting the gradients flow in a cross-modal fashion. Finally, the performance gap with the full-supervised case shows there is still room for improvement in the self-supervised approaches.
Next, we compare the number of unique categories the sampled instances originally belong to, using the ground-truth labels provided in the dataset. Our logic is that the more categories the samples come from, the more diverse and less redundant the samples are. We train these on UCF-101 over 300 iterations with different mini-batch sizes, M ∈ {32, 64, 128}. Fig. 2 shows that active sampling selects more categories than random sampling across all three mini-batch sizes. AtM = 128, active sampling (with gradient embedding) covers 60-70% of categories on UCF101, which is substantially more diverse than random sampling (30-40%). (A plot showing the probability of sampling unique negatives (instances from different categories) is shown in Appendix Figure 4.) While both sampling schemes perform similarly in early iterations, active sampling starts choosing more diverse instances as the training progresses; this is because the gradient embedding becomes more discriminative with respect to the uncertainty.
Random vs. active sampling. To validate the benefit of active sampling over random sampling, we compare models pretrained with different sampling approaches on downstream tasks. As shown in Table 1, our CM-ACC outperforms the XMoCo, which uses random sampling, by large margins, i.e. 3.1%, 1.9%, and 6.2% on UCF101, HMDB51, and ESC50, respectively (∆( 7©- 5©)).
Feature vs. gradient embedding. We compare two ways to do active sampling: using gradient embeddings (Eqn. 5) and feature embeddings (the outputs from ha and hv) when selecting the seed centroids with k-MEANS++. Fig. 2 shows that gradient embeddings produce a more diverse set of negative samples than feature embeddings; this is consistent across all three batch sizes. Table 2 shows that this diversity helps achieve better downstream performances across all three benchmarks. Fig. 5 (in Appendix) provides further insights, showing that the samples with high gradient magnitudes tend to be more informative negative samples.
From a theoretical aspect, the gradient norm induced by each candidate with computed pseudo labels estimates the candidates influence on the current model. The gradient embeddings convey information both about the model’s uncertainty and potential update direction upon receiving a candidate. However, such messages are missing from the feature embeddings. This shows the importance of considering both uncertainty and diversity when selecting random samples: the k-MEANS++ ensures the diversity in the sample set, but without the uncertainty measure we lose important discriminative information from the candidates.
Online hard example mining vs. active sampling. We compare our approach to online hard example mining (OHEM) (Shrivastava et al., 2016), which constructs negative samples by explicitly choosing the ones that incur high loss values. Specifically, we compute the pseudo-labels for all keys (negative sample candidates) with a given mini-batch of queries. We then compute the classification loss based on these pseudo labels and select the top M keys with the highest loss values. We pretrain the models on Kinetics-700 (Kay et al., 2017) and report the top-1 accuracy on UCF101 (Soomro et al., 2012) and HMDB51 (Kuehne et al., 2011). We use the same architecture and hyperparameters; the only difference is the sampling approach.
Table 3 shows OHEM is generally less effective than both random sampling and our active sampling. Intuitively, OHEM promotes the dictionary to contain the most challenging keys for a given minibatch of queries. Unfortunately, this causes OHEM to produce a redundant and biased dictionary, e.g., negative samples coming from a particular semantic category. Our results show that, when M (mini-batch size) is small, the performance of OHEM is even worse than random sampling, although the gap between OHEM and random sampling decreases as M increases. We believe this is because OHEM has a higher chance of selecting similar negative instances. When M is large, this issue can be mitigated to some extent, but the performance still falls behind ours by a large margin. This suggests the importance of having a diverse set of negative samples, which is unique in our approach.
Comparisons with SOTA. Table 4 shows our approach outperforms various self-supervised approaches on action recognition. For fair comparisons, we group the SOTA approaches by different pretraining dataset sizes, i.e. small-scale (UCF/HMDB), medium-scale (Kinetics), and large-scale (AudioSet). Our gains are calculated according to this grouping. As we can see, our approach outperforms SOTA approaches across all groups. Compared with GDT (Patrick et al., 2020), the current top performing model on cross-modal self-supervised learning, our model outperforms it by 1.6 % on UCF101 and 1.1 % on HMDB51. Table 5 shows audio classification transfer results. For Kinetics and AudioSet (240K), our model outperforms the current state-of-the-art, AVID (79.1%) by 0.1% and 1.8% on Kinetics and AudioSet 240K, respectively. Our approach also outperforms AVID (89.2%) pretrained on AudioSet (1.8M) by 1.6%.

heading: 6 CONCLUSION
text: We have shown that random sampling could be detrimental to contrastive learning due to the redundancy in negative samples, especially when the sample size is large, and have proposed an active sampling approach that yields diverse and informative negative samples. We demonstrated this on learning audio-visual representations from unlabeled videos. When pretrained on AudioSet, our approach outperforms previous state-of-the-art self-supervised approaches on various audio and visual downstream benchmarks. We also show that our active sampling approach significantly improves the performance of contrastive learning over random and online hard negative sampling approaches.

heading: A DETAILS ON DATA PROCESSING
text: We preprocess video frames by sampling at 10 FPS and applying random cropping, horizontal flipping, gray-scaling, and temporal jittering. We resize video frames to 3-channel images of 224×224; we set the clip length to 16 frames during pretraining, and 32 frames during finetuning on downstream tasks. For audio channel, we extract mel-spectrograms from the raw waveform using the LibROSA library and get a 80× T matrix with 80 frequency bands; T is proportionate to the length of an audio clip. We then segment the mel-spectrogram according to the corresponding video clips to ensure temporal synchrony. We treat the mel-spectrograms as an 80-channel 1D signal.
As for downstream tasks, we evaluate our models on action recognition using UCF101 (Soomro et al., 2012) and HMDB51 (Kuehne et al., 2011), and on sound classification using ESC50 (Piczak, 2015b). UCF101 contains 13K video clips from 101 action categories, HMDB51 contains 7K video clips from 51 categories, and ESC50 has 2K audio clips from 50 categories. UCF101 and HMDB51 have 3 official train/test splits, while ESC50 has 5 splits. We conduct our ablation study using split-1 of each dataset. We report our average performance over all splits when we compare with prior work.

heading: B ADDITIONAL EXPERIMENTS
text: Effect of mutual information. We investigate the impact of the amount of MI on contrastive learning using the Spatial-MultiOmniglot dataset (Ozair et al., 2019). It contains paired images (x, y) of Omniglot characters (Lake et al., 2015) with each image arranged in an m × n grid (each grid cell is 32 × 32 pixels). Let li be the alphabet size for the ith character in each image, then the MI I(x, y) = ∑mn i=1 logli. This way, we can easily control the MI by adding or removing characters. We follow the experimental protocol of Ozair et al. (Ozair et al., 2019), keeping the training dataset size fixed at 50K and using the same alphabet sets: Tifinagh (55 characters), Hiragana (52), Gujarati (48), Katakana (47), Bengali (46), Grantha (43), Sanskrit (42), Armenian (41), and Mkhedruli (41).
Fig. 3(a) shows the results as the number of characters (and thus the MI) increases. We see that all approaches achieve nearly 99% accuracy with less than 3 characters; this is the case when the exponent of the MI is smaller than the dataset size (50K), i.e., eI(x,y) = 55 with one character, eI(x,y) = 2, 860 with 2 characters. However, starting from 3 characters, the performance of the regular MoCo (SMoCo) drops significantly; this is because the exponent of the MI (=137,280 (55×52×48)) is much larger than the dataset size. Although our model also drops performance when the MI is increased, it outperforms the other approaches by a large margin. We also observe that XMoCo outperforms SMoCo in mild conditions (1-5 characters) but performs nearly the same as SMoCo with severe conditions (6-9 characters). This suggests that, while cross-modal prediction helps to learn good representations, it also suffers with the same issue when the MI is large, thus adopting active sampling is beneficial.
Effect of dictionary size. Fig. 3 (b) shows how the dictionary size affects downstream task performance. Here we pretrain our model on Kinetics-700 and finetune it on UCF-101. Overall, all
three approaches benefit from large dictionaries up to a threshold (at about 103), which is consistent with previous empirical findings (He et al., 2020). However, both XMoCo and SMoCo starts deteriorating performance after about 104 (which is consistent with previous theoretical claims of Arora et. al (Arora et al., 2019)), whereas ours do not suffer even after 104. This suggests that there are performance limits by simply increasing the size of a randomly-sampled dictionary, and also shows the benefit of our active sampling approach.
Effect of pretraining dataset sizes. We investigate the effects of the size of pretraining datasets, using Kinetics-Sound (22k), Kinetics (240K), and AudioSet (1.8M). We vary pretraining conditions while using the same protocol to finetune the models end-to-end on downstream tasks.
Table 6 shows that our model benefits from pretraining on video data, and that the performance improves as we use a large pretraining video dataset (Kinetics and AudioSet) than the relatively smaller dataset (Kinetics-Sound). Notably, our approach even outperforms the fully-supervised pretraining approaches by pretraining on a larger video dataset (1.0%, 3.6%, and 8.5% improvement on UCF101, HMDB51, and ESC50, respectively.)
Diversity of random vs. active sampling. To compare the diversity of the chosen negatives by random vs. active sampling, we plot the probability of them on sampling of unique negatives (instances from different categories). The more categories the samples come from, we get more diverse and less redundant samples. We train these on UCF-101 over 300 iterations with different mini-batch sizes, M ∈ {32, 64, 128}. As shown in Figure 4, the active sampling selects more categories than random sampling across all three mini-batch sizes. AtM = 128, active sampling (with gradient embedding) covers 60-70% of categories on UCF101, which is substantially more diverse than random sampling (30-40%).
C VISUALIZATION OF NEGATIVE INSTANCES
Figure 5 shows negative instances selected by active sampling and random sampling when we use audio clips as the query. We visualize the center frames of the selected video clips. We can see that our approach selects more challenging examples than the random sampling approach. For instance,
given a query opening bottle, our approach selected video clips from the same or similar semantic categories, e.g. drinking shot and opening bottle. Given snowboarding, our approach selected more video clips related to categories containing the snow scene, e.g. ice fishing, snow kiting, and tobogganing.
Furthermore, we also find that our approach selects more diverse negative samples. For example, given a query snowboarding, active sampling selected video clips from 4 different categories related to the snow scene: (ice fishing, playing ice hockey, snow kiting, and tobogganing). In comparison, the random sampling approach yields fewer semantic cate-
gories in general. This suggests that our active sampling approach produces more ‘challenging’ and ‘diverse’ negative instances than the random sampling approach.
To clearly investigate the relationship between negative samples and their gradient magnitudes, we show the gradient norm of each visualized sample in Figure 5. We can see that hard negatives tend to have larger gradient norms than easy negatives. Given a query Playing guitar, video clips containing the concept of “playing instruments” yield the higher gradient norms, i.e. playing violin (333.87) and tapping guitar (301.35), while concepts that are easy to discriminate, e.g., riding a camel yield a significantly smaller gradient norm (5.92). This provides evidence showing the gradient magnitude is effective in measuring the uncertainty of the current model, i.e., highly-uncertain samples (hard negatives) tend to yield gradients with larger magnitudes, while highly-confident samples (easy negatives) tend to have smaller gradient magnitudes.

heading: D WHEN WOULD CROSS-MODAL CONTRASTIVE LEARNING FAIL?
text: In general, cross-modal video representation learning is based on an assumption that the natural correspondence between audio and visual channels could serve as a useful source of supervision. While intuitive, this assumption may not hold for certain videos in-the-wild, which may cause the model to learn suboptimal representations. To investigate when our approach succeeds and fails, we conduct a post-hoc analysis by using thehttps://www.overleaf.com/project/5ded2abe1c17bc00011e5da8 ground-truth semantic category labels provided in Kinetics-700 (Carreira et al., 2019) (which is not used during pretraining). Specifically, we use our pretrained model to solve the audio-visual contrastive pretext task (Eqn.(7) in the main paper) and keep track of the prediction results (correct/incorrect). We then average the pretext task accuracy over 100 randomly chosen samples for each action category.
Figure 6 shows the top-10 and bottom-5 classes by using both audio (left) and video (right) as the query. We observe that, the top ranked classes for both audio and video are the activities that have highly correlated visual-audio signals. For instance, playing bass guitar, play piano, and play violin are all activities related to music. The correlation of audio-visual signals for these activities are obvious; such highly correlated signals are easier to be learned in a cross-modal manner. On the contrary, the bottom ranked classes are those that have subtle audio-visual correlation, e.g. tossing coin, shaking hand, looking at phone, and hugging. We also investigate the distribution of hard-easy classes with that reported in Kinetics-700 (Carreira et al., 2019) learned by the I3D-RGB model (Carreira & Zisserman, 2017). Interestingly, we find that some hard classes (e.g. karaoke and recording music) are listed in our top ranked classes. We suspect that, when only learned within visual modality, some classes with cluttered or complected spatial information will bring difficulties for classification. While, as our cross-modal approach can
leverage information from both auditory and visual information, so our model does not limited by such problems.
Algorithm 2 Cross-Modal Active Contrastive Coding (Detailed version of Algorithm 1) 1: Require: Audio-visual clips A, V ; encoders fv , fa, hv , ha; dictionary size K; pool size N ; batch size M 2: Initialize parameters, θvq , θvk, θ a q , θ a k v Uniform(0, 1)
3: Draw random dictionary, Dv ← {v1, · · · , vK} v Random(V ), Da ← {a1, · · · , aK} v Random(A) 4: Encode dictionary samples, kvi ← hv(vi), ∀vi ∈ Dv , kai ← ha(ai), ∀ai ∈ Da 5: for epoch = 1 to #epochs: do 6: Draw random pool, Uv ← {v1, · · · , vN} v Random(V ), Ua ← {a1, · · · , aN} v Random(A) 7: Encode pool samples, kvn ← hv(vn), ∀vn ∈ Uv , kan ← ha(an), ∀an ∈ Ua 8: for t = 1 to #mini-batches: do 9: Draw mini-batch, Bv ← {v1, · · · , vM} v V , Ba ← {a1, · · · , aM} v A
10: . Active sampling of negative video keys for Dv 11: Encode mini-batch samples, qai ← fa(ai), ∀ai ∈ Ba 12: for ∀vn ∈ Uv\Dv: do 13: Compute pseudo-posterior, p(ŷvn|vn, Ba)← exp(kvn·q a j )∑M
i=1 exp(k v n·qai ) ,∀j ∈ [1,M ] 14: Compute pseudo-label, ỹvn ← arg max p(ŷvn| · ) 15: end for 16: Compute gradient, gvn ← ∂∂θlastLCE (p(ŷ v n| · ), ỹvn) |θ=θaq ,∀n ∈ [1, N ] 17: Obtain Sv ← k-MEANS++INIT ({gvn : vn ∈ Uv\Dv},#seeds = M) 18: Update Dv ← ENQUEUE(DEQUEUE(Dv), Sv) 19: . Active sampling of negative audio keys for Da 20: Encode mini-batch samples, qvi ← fv(vi), ∀vi ∈ Bv 21: for ∀an ∈ Ua\Da: do 22: Compute pseudo-posterior, p(ŷan|an, Bv)← exp(kan·q v j )∑M
i=1 exp(k a n·qvi ) , ∀j ∈ [1,M ] 23: Compute pseudo-label, ỹan ← arg max p(ŷan| · ) 24: end for 25: Compute gradient, gan ← ∂∂θlastLCE (p(ŷ a n| · ), ỹan) |θ=θvq , ∀n ∈ [1, N ] 26: Obtain Sa ← k-MEANS++INIT ({gan : an ∈ Ua\Da},#seeds = M) 27: Update Da ← ENQUEUE(DEQUEUE(Da), Sa) 28: . Cross-modal contrastive predictive coding 29: Encode mini-batch samples, kvi ← hv(vi), ∀vi ∈ Bv , kai ← ha(ai),∀ai ∈ Ba 30: Compute p(yvi |·) = exp(qvi ·k a i /τ)∑K
j=0 exp(q v i ·k a j /τ)
, p(yai |·) = exp(qai ·k v i /τ)∑K
j=0 exp(q a i ·k v j /τ) ,∀i ∈ [1,M ] 31: . Update model parameters 32: Update θvq ← θvq − γ∇θLCE(p(yv| · ), yvgt)|θ=θvq , θ a q ← θaq − γ∇θLCE(p(ya| · ), yagt)|θ=θaq 33: Momentum update θvk ← mθvk + (1−m)θvq , θak ← mθak + (1−m)θaq 34: end for 35: end for 36: return Optimal solution θvq , θvk, θaq , θak
Algorithm 3 k-MEANS++INIT Seed Cluster Initialization 1: Require: Data X of N samples; number of centroids K 2: Choose one centroid uniformly at random, C[0]← x v Random(X) 3: for k = 1 to K − 1: do 4: . Compute a cumulative probability distribution with a probability in proportion to their squared dis-
tances from the nearest centroid that has already been chosen 5: for n = 0 to N − 1: do 6: Compute the squared distance, D[n]← (min dist(X[n], C))2 7: end for 8: Compute the cumulative probability distribution, P ← cumsum(D)
sum(D)
9: . The next centroid is chosen using P (X) as a weighted probability distribution 10: Choose one centroid at random, C[k]← x v P (X) 11: end for 12: return C containing K centroids
Algorithm 4 Cross-Modal Contrastive Coding without Active Sampling 1: Require: Audio-visual clips A, V ; dictionary Dv; encoders fv , fa, hv , ha;
dictionary size K; mini-batch size M ; learning rate γ; momentum m 2: Initialize parameters, θvq , θvk, θ a q , θ a k v Uniform(0, 1) 3: Load a dictionary at random, Da ← {v1, · · · , vK} v Random(V ) 4: Load a dictionary at random, Dv ← {a1, · · · , aK} v Random(A) 5: Encode dictionary samples, kvi ← hv(vi), ∀vi ∈ Da, kai ← ha(ai), ∀ai ∈ Dv 6: for epoch = 1 to #epochs: do 7: for t = 1 to #mini-batches: do 8: Load a mini-batch of visual clips, Bv ← {v1, · · · , vM} v V 9: Load a mini-batch of audio clips, Ba ← {a1, · · · , aM} v A
10: . Update dictionaries 11: Encode mini-batch samples, kvi ← hv(vi), ∀vi ∈ Bv 12: Encode mini-batch samples, kai ← ha(ai),∀ai ∈ Ba 13: Update Dv ← ENQUEUE(DEQUEUE(Dv), Bv) 14: Update Da ← ENQUEUE(DEQUEUE(Da), Ba) 15: . Cross-modal contrastive predictive coding 16: Encode mini-batch samples, qvi ← fv(vi), ∀vi ∈ Bv 17: Encode mini-batch samples, qai ← fa(ai),∀ai ∈ Ba 18: Compute the posterior, p(yvi |vi, ai, Dv) = exp(qvi ·k a i /τ)∑K
j=0 exp(q v i ·k a j /τ)
, ∀i ∈ [1,M ]
19: Compute the posterior, p(yai |ai, vi, Dv) = exp(qai ·k v i /τ)∑K
j=0 exp(q a i ·k v j /τ) , ∀i ∈ [1,M ] 20: . Update model parameters 21: Update θvq ← θvq − γ∇θLCE(− log p(yv| · ), yvgt)|θ=θvq 22: Update θaq ← θaq − γ∇θLCE(− log p(ya| · ), yagt)|θ=θaq 23: Momentum update θvk ← mθvk + (1−m)θvq 24: Momentum update θak ← mθak + (1−m)θaq 25: end for 26: end for 27: return Optimal solution θvq , θvk, θaq , θak

Review summaries:
In this paper, the authors propose a cross-modal (audio-video) self-supervised representation learning method with a contrastive learning framework. To overcome the high redundancy in the negative samples, they propose an active negative sampling method. They use a gradient with respect to the pseudo label to measure the uncertainty of a negative sample. They use K-means clustering to maximize the negative sample diversity when constructing a new negative set for queueing. They show their method's efficacy on the public benchmarks: Kinetics, AudioSet for retraining, and UCF-101, HMDB-51, ESC-50 for downstream tasks.