Contrast and Mix: Temporal Contrastive Video Domain Adaptation with Background Mixing

1 Introduction . Unsupervised domain adaptation ( UDA ) , which alleviates the requirement of large amounts of annotated data by adapting a model learned on a labelled source domain to an unlabelled target domain , has drawn a great deal of attention in the last few years [ 12 , 80 ] . Much progress has been made in developing deep UDA methods by minimizing the cross-domain divergence [ 39 , 70 ] , adding adversarial domain discriminators [ 20 , 74 ] , and image-to-image translation techniques [ 26 , 51 ] . However , despite impressive results on commonly used benchmark datasets ( e.g. , [ 61 , 75 , 57 ] ) , most of the methods have been developed only for images and not for videos , where the annotation task is often more complicated requiring tedious human labor in comparison to images . More recently , very few works have attempted deep UDA for video action recognition by directly matching segment-level features [ 8 , 27 , 50 , 42 ] or with attention weights [ 11 , 53 ] . However , ( 1 ) trivially matching segment-level feature distributions by extending the image-specific approaches , without considering the rich temporal information may not alone be sufficient for video domain adaptation ; ( 2 ) prior methods often focus on aligning target features with source , rather than exploiting any action semantics shared across both domains ( e.g. , difference in background with the same action : videos in the top row of Figure 1 are from the source and target domain respectively , but both capture the same action walking ) ; ( 3 ) existing methods often rely on complex adversarial learning which is unwieldy to train , resulting in very fragile convergence . 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . Meanwhile , self-supervised pretext tasks like predicting rotation and translation have recently emerged as an alternative to adversarial learning for unsupervised domain adaptation in images [ 38 , 71 ] . While these works show the promising potential of self-supervised learning in aligning source and target domains , the more recent very successful contrastive representation learning [ 9 , 23 , 52 ] has never been used to adapt video action recognition models to target domains . Motivated by this , in this paper , we explore the following natural , yet important question : whether and how contrastive learning could be exploited for the challenging and practically important task of unsupervised video domain adaptation for human action recognition ? To this end , we introduce Contrast and Mix ( CoMix ) , a simple yet effective approach based on contrastive learning to adapt video action recognition models trained on a labeled source domain to unlabelled target domains . First , we propose to represent video as a graph and then utilize temporal contrastive self-supervised learning over the graph representations as a nexus between source and target domains to align features , without requiring any additional adversarial learning , as most prior works do in video domain adaptation [ 8 , 11 , 53 ] . Specifically , we maximize the similarity between encoded representations of the same video at two different speeds as well as minimize the similarity between different videos played at different speeds , leveraging the fact that changing video speed does not change an action on both domains . While minimization of contrastive self-supervised losses in both domains simultaneously helps in domain alignment , it ignores action semantics shared across them as the loss treats each domain individually . To alleviate this , we incorporate new synthetic videos into the temporal contrastive objective , which are obtained by mixing background of a video from one domain to a video from another domain , as shown in Figure 1 ( bottom ) . Importantly , since mixing background doesn ’ t change the temporal dynamics , we introduce pseudo-labels for the mixed videos to be same as the label of the original videos and consider additional positives per anchor ( see Figure 2 ) , which encourages the model to generalize to new samples that may not be covered by temporal contrastive learning in hand . In other words , mixed background video of an input sample in the embedding space act as small semantic perturbations that are not imaginary , i.e. , they are representative of the action semantics shared across source and target domains . Finally , rather than relying only on the supervision of source categories to learn a discriminative representation , we generate pseudo-labels for the target samples in every batch and then harness the label information using a temporal supervised contrastive term , that pushes the examples from the same class close and the examples from different classes further apart ( Figure 2 : right ) . While our modified contrastive losses are motivated by the supervised contrastive learning [ 30 ] , we use pseudo labels for exploiting shared action semantics and discriminative information from target domain , instead of using true labels as an alternative to supervised cross-entropy loss ( which is not present for target samples ) . To the best of our knowledge , ours is the first work that successfully leverages contrastive learning in an unified framework to align cross-domain features while enhancing discriminabilty of the latent space for unsupervised video domain adaptation . To summarize , the main contributions of our work are as follows : • We introduce Contrast and Mix ( CoMix ) , a new contrastive learning framework to learn discriminative invariant feature representations for unsupervised video domain adaptation . Overall , CoMix is simple and easy to implement which perfectly fits into modern mini-batch end-to-end training . • We propose a novel extension to temporal contrastive loss by using background mixing that allows additional positives per anchor , thus adapting contrastive learning to leverage action semantics shared across both domains . We also integrate a supervised contrastive learning objective using pseudo label information from the target domain to enhance discriminabilty of the latent space . • We conduct extensive experiments on several challenging benchmarks ( UCF-HMDB [ 8 ] , Jester [ 53 ] , and Epic-Kitchens [ 50 ] ) for video domain adaptation to demonstrate the superiority of our approach over state-of-the-art methods . Our experiments show that CoMix delivers a significant performance increase over the compared methods , e.g. , CoMix outperforms SAVA [ 11 ] ( ECCV ’ 20 ) by 3.6 % on UCF-HMDB [ 8 ] and TA3N [ 8 ] ( ICCV ’ 19 ) by 9.2 % on Jester [ 45 ] benchmark respectively ) . 2 Related Work . Action Recognition . Much progress has been made in developing a variety of ways to recognize video actions , by either applying 2D-CNNs [ 6 , 37 , 47 , 79 ] or 3D-CNNs [ 4 , 17 , 22 , 73 ] . Many successful architectures are usually based on the two-stream model [ 67 ] , processing RGB frames and optical-flow in two separate CNNs with a late fusion in the upper layers [ 29 ] . SlowFast network [ 18 ] employs two pathways for recognizing actions by processing a video at different frame rates . Mitigating background bias in action recognition has also been presented in [ 10 , 36 ] . Despite remarkable progress , these models critically depend on large labeled datasets which impose challenges for cross-domain action recognition . In contrast , our work focuses on unsupervised domain adaptation for action recognition , with labeled data in source domain , but only unlabeled data in target domain . Unsupervised Domain Adaptation . Unsupervised domain adaptation has been studied from multiple perspectives ( see reviews [ 12 , 80 ] ) . Representative works minimize some measurement of distributional discrepancy [ 21 , 39 , 65 , 70 ] or adopt adversarial learning [ 5 , 20 , 40 , 56 , 74 ] to generate domain-invariant features . Leveraging image translation [ 25 , 26 , 51 ] or style transfer [ 15 , 91 ] is also another popular trend in domain adaptation . Deep self-training that focus on iteratively training the model using both labeled source data and generated target pseudo labels have been proposed in [ 46 , 90 ] . Semi-supervised domain adaptation leveraging a few labeled samples from the target domain has also been proposed for many applications [ 14 , 33 , 63 ] . A very few methods have recently attempted video domain adaptation , using adversarial learning combined with temporal attention [ 8 , 42 , 53 ] , multi-modal cues [ 50 ] , and clip order prediction [ 11 ] . While existing video DA methods mainly rely on adversarial learning ( which is often complicated and hard to train ) in some form or other , they do not take any action semantics shared across domains into consideration . Our approach on the other hand , successfully leverages temporal contrastive learning to learn domain-invariant features while exploiting shared action semantics through background mixing for video domain adaptation . Recently , self-supervised tasks like predicting rotation and translation have been used for unsupervised domain adaptation and generalization , mainly for images [ 3 , 38 , 71 ] . By contrast , we focus on the more challenging problem of domain adaptation for human action recognition , where our goal is to align domains by learning consistent features representing different speeds of unlabeled videos . We further propose a temporal supervised contrastive loss to ensure discriminabilty by considering pseudo-labeling in an unified framework for video domain adaptation . Contrastive Learning . Contrastive representation learning is becoming increasingly attractive due to its great potential to leverage large amount of unlabeled images [ 9 , 16 , 23 , 48 , 24 , 52 ] and videos [ 19 , 31 , 54 , 60 , 59 , 78 ] . Speed of a video is investigated for self-supervised [ 1 , 28 , 77 , 86 ] and semi-supervised learning [ 68 , 94 ] unlike the problem we consider in this paper . Recent works [ 84 , 87 ] utilize contrastive learning with different augmentations for learning unsupervised representations of graph data . Contrastive learning has also been recently used in supervised settings , where labels are used to guide the choice of positive and negative pairs [ 30 ] . While our approach is inspired by these , we propose a novel temporal contrastive learning framework with background mixing for video domain adaptation , which to our best knowledge has not been explored in the literature . Image Mixtures . Mixup regularization [ 89 ] and its variants [ 2 , 76 , 88 ] that train models on virtual examples constructed as convex combinations of pairs of images and labels have been used to improve the generalization of neural networks . Very few methods apply Mixup in domain adaptation , but mainly to stabilize the domain discriminator [ 62 , 83 , 85 ] or to smoothen the predictions [ 44 ] . Several works have recently leveraged the idea of different image mixtures [ 34 , 66 ] for improving contrastive representation learning . Our proposed background mixing can be regarded as an extension of this line of research by adding background of a video from one domain to a video from another domain , to explore shared semantics while learning domain-invariant features for action recognition . 3 Proposed Method . Unsupervised video domain adaptation aims to improve the model generalization performance by transferring knowledge from a labeled source domain to an unlabeled target domain . Formally , we have a set of labelled source videos Dsource= { ( Vi { s } , yi ) } NSi=1 and a set of unlabelled target videos Dtarget= { Vi { t } } NTi=1 , with a common label space L. Given these data sets , our goal is to learn a single model for action recognition that performs well on previously unseen target domain videos . Approach Overview . Figure 3 illustrates an overview of CoMix . Our action recognition model consists of a feature encoder F with a temporal graph encoder G. Given a video , the feature encoder F first extracts clip-level features , and then a graph encoder G utilizes those features to model intrinsic temporal relations for providing a robust encoded representation for action recognition . CoMix adopts supervised learning on the source videos , as the labels are available , jointly with two novel temporal contrastive learning loss terms to align the features for domain adaptation . Specifically , we maximize the similarity of the encoded representation of the fast version of a video ( represented by f clips ) with that of the slow version of the same video ( represented by s clips , where s < f ) as well as minimize the similarity of the representations of different videos within each of the two domains . However , as temporal contrastive loss treats each domain individually , we further add two new sets of synthetic videos that contain source videos mixed with target background and vice versa , respectively for introducing the background variations among the videos while keeping the action semantics intact . Finally , we generate pseudo-labels for the target videos in every mini-batch and utilize them using another temporal supervised contrastive term . This term contrasts target videos with the same pseudo-label as positives to learn features discriminative for the target domain . We now describe each of our proposed components individually in detail in the following subsections . Video Representation . Capturing long-range temporal structure in videos is crucial for action recognition , which in turn affects the overall generalization performance of a model when adapting across domains . Thus , we adopt a graph convolutional neural network ( G ) on top of a 3D convolutional neural network ( F ) as our video feature encoder . Specifically , for a video V with n clips , the feature extractor F maps the clips into the corresponding sequence of features , which alone do not incorporate the rich temporal structure of the video . Therefore , we use the temporal graph encoder which constructs a fully connected graph on top of the clip-level features , with learnable edge weights through a parameterized adjacency matrix , as in [ 81 ] . With these graph representations , we apply a graph convolutional neural network with three layers and finally perform average pooling over all the node features to output the encoded representation of the video V. In summary , the end-to-end network G ( F ( . ) ) : V→ Rc takes a sequence of clips from a video as input and outputs confidence scores ( logits ) over the number of classes c for recognizing actions . Temporal Contrastive Learning . Given video representations , our goal is to leverage contrastive self-supervised learning in both domains for unsupervised domain adaptation . To this end , we use temporal speed invariance in videos as a proxy task and enforce this with a pairwise contrastive loss . Specifically , our key idea is to represent videos in two different temporal speeds ( fast and slow ) to obtain their encoded representations and then consider the fast and slow version representations of the same video to constitute positive pairs , while versions from different videos constitute negative pairs . Formally , let us consider a mini-batch of B videos { V1n , V2n , ... , VBn } with corresponding feature representations { z1n , z2n , ... , zBn } , where each of the videos Vin is represented using n number of sampled clips . Let f be the number of clips used to represent the fast version of the videos ( forwarded through the base branch ) , and s be that used for the slow version ( forwarded through the auxiliary branch ) , with s < f , as shown in Figure 4 . Given positive and negative pairs , the model is trained such that it learns to maximize agreement between positive pairs , while minimizing agreement between negative pairs . This is achieved by employing a temporal contrastive loss ( Ltcl ) as Ltcl ( Vif , Vis ) = − log h ( zif , z i s ) h ( zif , z i s ) + B∑ j=1 , j 6=i v∈ { s , f } h ( zif , z j v ) ( 1 ) where , h ( u , v ) = exp ( u ᵀv ‖u‖2‖v‖2 /τ ) is the exponential of cosine similarity measure and τ is the temperature hyperparameter [ 9 ] . We use f = 16 , and choose s from { 12 , 8 , 4 } following a random uniform distribution in every training iteration where randomness encourages the model to learn from a variety of temporal speed variations to learn robust representations . Background Mixing . As temporal contrastive loss treats each domain individually , it ignores shared action semantics which is vital for domain alignment . Thus , we propose a new perspective of temporal contrastive loss through background mixing , specifically to alleviate the crossdomain background shift , as seen in Figure 1 . The basic idea is to obtain the background frames for the videos in one domain and mix it with the frames of the videos from the other domain . More details on how we extract the backgrounds are provided in supplementary material . This introduces variation in each of the domains by adding new synthetic videos with the same action semantics as earlier , but possessing background from the other domain . Given two videos Vi { s } ∈ Dsource and Vi { t } ∈ Dtarget with corresponding background frames ( single image per video ) as BGi { s } and BGi { t } , we obtain the synthetic videos in both domains by a convex combination of the background with each of the frames in the videos as follows . V̂i { s } = ( 1− λ ) ·Vi { s } + λ ·BGi { t } V̂i { t } = ( 1− λ ) ·Vi { t } + λ ·BGi { s } ( 2 ) where , λ is sampled from the uniform distribution [ 0 , γ ] , V̂i { s } and V̂i { t } correspond to the video from source domain with target background and vice versa , respectively . The main operation in our proposed background mixing is to generate a synthetic video with background from the other domain while retaining the temporal action seman- tics intact . Since mixing background doesn ’ t change the motion pattern of a video which actually defines an action , we assume both the original and mixed video to be of the same action class and go beyond single instance positives in Eq . 1 by adding additional positives per anchor , as in supervised contrastive learning [ 30 ] ( see Figure 2 for an illustrative example ) . The modified temporal contrastive loss with background mixing ( Lbgm ) is defined as below : Lbgm ( Vif , Vis ) = − 1 |P ( zif ) | ∑ p∈P ( zif ) log h ( zif , p ) ∑ p∈P ( zif ) h ( zif , p ) + B∑ j=1 , j 6=i v∈ { s , f } { h ( zif , z j v ) + h ( zif , ẑ j v ) } ( 3 ) where , P ( zif ) ≡ { zis , ẑis , ẑif } is the set of positives for the anchor zif , and ẑis/f represent the feature representation of the corresponding background-mixed video depending on the domain to which Vi belongs . Note that for anchor zif , there are 3 positive pairs : ( a ) slow version of the mixed video ( ẑis ) , ( b ) fast version of the mixed video ( ẑ i f ) , and ( c ) slow version of the original video ( z i s ) . Also , the loss is computed for all positive pairs in the mini-batch , i.e. , ( Vif , V i s ) , ( V i s , V i f ) , ( V̂ i f , V̂ i s ) , and ( V̂is , V̂ i f ) . Simultaneous minimization of Lbgm in both source and target domains not only learns temporal dynamics but also helps to better align the features for video domain adaptation by leveraging action semantics shared across both domains . Our background mixing is especially effective in video domain adaptation as it enforces the model to be robust to domain changes ( i.e. , difference in background as shown in Figure 1 ) while leaving the action semantics intact . Further , it can also be adopted as a data augmentation strategy for improved generalization in standard video action recognition : we leave this as an interesting future work . Incorporating Target Pseudo Labels . While temporal contrastive loss with background mixing helps in aligning the learned representations across the two domains , we can not fully rely on source categories to learn features discriminative for target domain . Therefore , we propose to use a supervised contrastive loss [ 30 ] over pseudo-labeled target samples , an extended version of temporal contrastive loss in Eqn . 1 to enhance discriminabilty by allowing many samples per anchor to be positive , so that videos of the same pseudo-label can be attracted to each other in the embedding space . Let A be the subset of videos assigned pseudo-labels using a confidence threshold , from a mini-batch of B videos , the supervised temporal contrastive loss for incorporating target pseudo-labels ( Ltpl ) is defined as Ltpl ( Vif , Vis ) = − 1 |P ( zif ) | ∑ p∈P ( zif ) log h ( zif , p ) ∑ p∈P ( zif ) h ( zif , p ) + ∑ a∈A , a6=i v∈ { s , f } h ( zif , z a v ) ( 4 ) where , P ( zif ) ≡ { zps , z p f : p ∈ A & ỹp = ỹi } \ { zif } is the set of all positives for video Vif and ỹi represent the pseudo-label for target video Vi . Note that the set of positives ( P ( . ) ) includes all the target domain samples ( fast and slow ) classified as the same action class as that of the anchor ( zif ) through the pseudo labels . Following [ 95 ] , we leverage a temporal ensemble prediction for a given video Vi from the target domain to produce robust and better-calibrated version of pseudo-labels . Specifically , we obtain the encoded ( logits ) representations zif and z i s from the base and auxiliary branch respectively and then compute the pseudo-label as ỹi = argmaxk softmax ( z i fused ) , where zifused represents the mean of both logits . We consider the class index k on which the model is most confident among c classes , provided it is higher than a confidence threshold . Optimization . Besides the losses Lbgm and Ltpl , we minimize the standard supervised cross-entropy loss ( Lce ) on the labelled source videos as follows . Lce ( Vi { s } , yi ) = − c∑ k=1 ( yi ) k log ( G ( F ( Vi { s } ) ) ) k ( 5 ) Overall , the loss function for training our model involving both source and target domain data is , LCoMix = L { s } ce + λbgm ( L { s } bgm + L { t } bgm ) + λtplL { t } tpl ( 6 ) where λbgm and λtpl are weights to balance the impact of individual loss terms . To reduce the number of hyper-parameters , we use the same weight λbgm for both L { s } bgm and L { t } bgm . Notably , for the semi-supervised domain adaptation setting , we also use supervised cross-entropy loss for the few labeled target domain videos in addition to the source domain videos .

Summary: In this paper, the authors tackle the problem of video domain adaptation for the task of action recognition. To align source and target domain features, they propose to use contrastive learning instead of domain adversarial learning. For contrastive learning, two instances of different speeds from the same video constitute a positive pair. Two instances of different speed from the different video constitute a negative pair. In addition to the contrastive learning, they propose the background mixup in order to bridge the domain gap between source background and target background. They also incorporate target pseudo-labels to learn more discriminative target classifier. They validate the efficacy of the model on the publicly available benchmarks: UCF-HMDB, Jester and Epic-Kitchens.  The claimed contribution of the paper are as follows.  1) Introducing a new UDA framework, Contrast and Mix (CoMix): contrastive learning with speed, and background mixup for bridging the domain gap  2) Integrate the pseudo labeling for UDA  3) Extensive experiments on three datasets, and achieving state-of-the-art performance on UCF-HMDB and Jester.