Active Contrastive Learning of Audio-Visual Video Representations

1 INTRODUCTION . Contrastive learning of audio and visual representations has delivered impressive results on various downstream scenarios ( Oord et al. , 2018 ; Hénaff et al. , 2019 ; Schneider et al. , 2019 ; Chen et al. , 2020 ) . This self-supervised training process can be understood as building a dynamic dictionary per mini-batch , where “ keys ” are typically randomly sampled from the data . The encoders are trained to perform dictionary look-up : an encoded “ query ” should be similar to the value of its matching key and dissimilar to others . This training objective maximizes a lower bound of mutual information ( MI ) between representations and the data ( Hjelm et al. , 2018 ; Arora et al. , 2019 ) . However , such lower bounds are tight only for sample sizes exponential in the MI ( McAllester & Stratos , 2020 ) , suggesting the importance of building a large and consistent dictionary across mini-batches . Recently , He et al . ( 2020 ) designed Momentum Contrast ( MoCo ) that builds a queue-based dictionary with momentum updates . It achieves a large and consistent dictionary by decoupling the dictionary size from the GPU/TPU memory capacity . However , Arora et al . ( 2019 ) showed that simply increasing the dictionary size beyond a threshold does not improve ( and sometimes can even harm ) the performance on downstream tasks . Furthermore , we find that MoCo can suffer when there is high redundancy in the data , because only relevant – and thus limited – parts of the dictionary are updated in each iteration , ultimately leading to a dictionary of redundant items ( we show this empirically in Fig . 3 ) . We argue that random negative sampling is much responsible for this : a randomly constructed dictionary will contain more “ biased keys ” ( similar keys that belong to the same class ) and “ ineffective keys ” ( keys that can be easily discriminated by the current model ) than a carefully constructed one . Furthermore , this issue can get aggravated when the dictionary size is large . In this paper , we focus on learning audio-visual representations of video data by leveraging the natural correspondence between the two modalities , which serves as a useful self-supervisory signal ( Owens & Efros , 2018 ; Owens et al. , 2016 ; Alwassel et al. , 2019 ) . Our starting point is contrastive learning ( Gutmann & Hyvärinen , 2010 ; Oord et al. , 2018 ) with momentum updates ( He et al. , 2020 ) . ∗Equal contribution 1Code is available at : https : //github.com/yunyikristy/CM-ACC However , as we discussed above , there are both practical challenges and theoretical limits to the dictionary size . This issue is common to all natural data but is especially severe in video ; successive frames contain highly redundant information , and from the information-theoretic perspective , audiovisual channels of video data contain higher MI than images because the higher dimensionality – i.e. , temporal and multimodal – reduces the uncertainty between successive video clips . Therefore , a dictionary of randomly sampled video clips would contain highly redundant information , causing the contrastive learning to be ineffective . Therefore , we propose an actively sampled dictionary to sample informative and diverse set of negative instances . Our approach is inspired by active learning ( Settles , 2009 ) that aims to identify and label only the maximally informative samples , so that one can train a high-performing classifier with minimal labeling effort . We adapt this idea to construct a non-redundant dictionary with informative negative samples . Our approach , Cross-Modal Active Contrastive Coding ( CM-ACC ) , learns discriminative audiovisual representations and achieves substantially better results on video data with a high amount of redundancy ( and thus high MI ) . We show that our actively sampled dictionary contains negative samples from a wider variety of semantic categories than a randomly sampled dictionary . As a result , our approach can benefit from large dictionaries even when randomly sampled dictionaries of the same size start to have a deleterious effect on model performance . When pretrained on AudioSet ( Gemmeke et al. , 2017 ) , our approach achieves new state-of-the-art classification performance on UCF101 ( Soomro et al. , 2012 ) , HMDB51 ( Kuehne et al. , 2011 ) , and ESC50 ( Piczak , 2015b ) . 2 BACKGROUND . Contrastive learning optimizes an objective that encourages similar samples to have similar representations than with dissimilar ones ( called negative samples ) ( Oord et al. , 2018 ) : min θf , θh ExvpX [ −log ( ef ( x ; θf ) ᵀh ( x+ ; θh ) ef ( x ; θf ) ᵀh ( x+ ; θh ) + ef ( x ; θf ) ᵀh ( x− ; θh ) ) ] ( 1 ) The samples x+ and x− are drawn from the same distribution as x ∈ X , and are assumed to be similar and dissimilar to x , respectively . The objective encourages f ( · ) and h ( · ) to learn representations of x such that ( x , x+ ) have a higher similarity than all the other pairs of ( x , x− ) . We can interpret it as a dynamic dictionary look-up process : Given a “ query ” x , it finds the correct “ key ” x+ among the other irrelevant keys x− in a dictionary . Denoting the query by q = f ( x ) , the correct key by k+ = h ( x+ ) , and the dictionary of K negative samples by { ki = h ( xi ) } , i ∈ [ 1 , K ] , we can express equation 1 in a softmax form , minθq , θk ExvpX [ −log e q·k+/τ∑K i=0 e q·ki/τ ] , where θq and θk are parameters of the query and key encoders , respectively , and τ is a temperature term that controls the shape of the probability distribution computed by the softmax function . Momentum Contrast ( MoCo ) decouples the dictionary size from the mini-batch size by implementing a queue-based dictionary , i.e. , current mini-batch samples are enqueued while the oldest are dequeued ( He et al. , 2020 ) . It then applies momentum updates to parameters of a key encoder θk with respect to parameters of a query encoder , θk ← mθk + ( 1 −m ) θq , where m ∈ [ 0 , 1 ) is a momentum coefficient . Only the parameters θq are updated by back-propagation , while the parameters θk are defined as a moving average of θq with exponential smoothing . These two modifications allow MoCo to build a large and slowly-changing ( and thus consistent ) dictionary . Theoretical Limitations of Contrastive Learning . Recent work provides theoretical analysis of the shortcomings of contrastive learning . McAllester & Stratos ( 2020 ) show that lower bounds to the MI are only tight for sample size exponential in the MI , suggesting that a large amount of data are required to achieve a tighter lower bound on MI . He et al . ( 2020 ) empirically showed that increasing negative samples has shown to improve the learned presentations . However , Arora et al . ( 2019 ) showed that such a phenomenon does not always hold : Excessive negative samples can sometimes hurt performance . Also , when the number of negative samples is large , the chance of sampling redundant instances increases , limiting the effectiveness of contrastive learning . One of our main contributions is to address this issue with active sampling of negative instances , which reduces redundancy and improves diversity , leading to improved performance on various downstream tasks . 3 APPROACH . 3.1 CROSS-MODAL CONTRASTIVE REPRESENTATION LEARNING . Our learning objective encourages the representations of audio and visual clips to be similar if they come from the same temporal block of a video . LetA = { a0 , · · · , aN−1 } and V = { v0 , · · · , vN−1 } be collections of audio and visual clips , where each pair ( ai , vi ) is from the same block of a video . We define query encoders fa and fv and key encoders ha and hv for audio and visual clips , respectively , with learnable parameters { θaq , θvq } for the query encoders and { θak , θvk } for the key encoders . These encoders compute representations of audio and visual clips as queries and keys , qv = fv ( v query ) , kv = hv ( v key ) , qa = fa ( a query ) , ka = ha ( a key ) ( 2 ) We train our encoders to perform cross-modal dictionary look-up , e.g. , given a query video clip vquery , we find the corresponding audio clip akey from a dictionary Da . Adapting MoCo ( He et al. , 2020 ) to our cross-modal setup , we implement a queue-based dictionaryDa that stores keys of audio clips { kai } Ki=1 , where K is the dictionary size . We compute the contrastive loss and backpropagate the gradients only to the visual query encoder fv and update the parameters θvq . For the audio encoder ha , we apply the momentum update ( He et al. , 2020 ) , θak ← mθak + ( 1−m ) θaq ( 3 ) The parameter θaq is not updated in this contrastive coding step ; we update it during the audio-tovisual step ( similar as above with the opposite modalities ) . Here we explain the visual-to-audio step only ; we perform bi-directional contrastive coding and train the whole model end-to-end . 3.2 ACTIVE SAMPLING OF NEGATIVE INSTANCES : UNCERTAINTY AND DIVERSITY . The quality of negative samples is crucial in contrastive learning . Existing work typically adopts random negative sampling . However , we want a diverse set of negative samples so that comparisons between positive and negative pairs are the most informative they can be . Motivated by active learning ( Settles , 2009 ) , we propose a gradient-based active sampling approach to improve the quality of negative samples . In active learning , the learner chooses samples that seem maximally informative and queries an oracle for labels to obtain an optimal solution with a minimal labeling budget . Adapting this to our setting , we can empower the learner to choose the maximally informative negative samples to construct a dictionary ; the main question is how to measure the informativeness of samples without labels . One way to measure informativeness is through the lens of uncertainty : If a model is highly uncertain about its prediction of a sample , we can ensure the maximum update to the model by including the sample in a mini-batch ( conversely , if the uncertainly is low for all samples in a mini-batch , the model update will be small ) . Ash et al . ( 2020 ) showed that gradients of a loss function with respect to the model ’ s most confident predictions can approximate the uncertainty of samples , demonstrating its effectiveness in active learning . They provide a theoretical justification by showing that gradient norms of the last layer of a neural network with respect to pseudo-labels provides a lower bound on gradient norms induced by any other labels . In this work , we use gradients of the last layer to measure the uncertainty and encourage our model to include samples that have the highest gradient magnitudes to constitute a dictionary . While the uncertainty of each individual samples is important , the diversity of samples is also a critical measure of informativeness . Intuitively , it is possible that a model is highly uncertain about samples from particular semantic categories , but constructing a mini-batch of samples from just those categories can severely bias gradients and ultimately lead to a bad local minima . There are several principled approaches to ensure diversity , e.g. , submodular optimization ( Fujishige , 2005 ) and Determinantal Point Processes ( DPP ) ( Macchi , 1975 ; Kulesza & Taskar , 2011 ) . Unfortunately , those methods are typically inefficient because of the combinatorial search space ( Nemhauser et al. , 1978 ; Gilks et al. , 1995 ) . In this work , instead of using the expensive solutions , we opt to the fast solution of Ash et al . ( 2020 ) and use the initialization scheme of the k-MEANS++ seeding algorithm ( Arthur & Vassilvitskii , 2007 ) to sample a diverse set of negative samples .

Summary: In this paper, the authors propose a cross-modal (audio-video) self-supervised representation learning method with a contrastive learning framework. To overcome the high redundancy in the negative samples, they propose an active negative sampling method. They use a gradient with respect to the pseudo label to measure the uncertainty of a negative sample. They use K-means clustering to maximize the negative sample diversity when constructing a new negative set for queueing. They show their method's efficacy on the public benchmarks: Kinetics, AudioSet for retraining, and UCF-101, HMDB-51, ESC-50 for downstream tasks.