Model Adaptation: Historical Contrastive Learning for Unsupervised Domain Adaptation without Source Data

1 Introduction . Deep neural networks ( DNNs ) [ 28 , 73 , 23 ] have achieved great success in various computer vision tasks [ 8 , 55 , 60 , 59 , 28 , 73 , 23 ] but often generalize poorly to new domains due to the inter-domain discrepancy [ 1 ] . Unsupervised domain adaptation ( UDA ) [ 78 , 51 , 76 , 64 , 66 , 79 , 77 , 103 , 102 , 25 , 71 , 44 , 26 , 86 ] addresses the inter-domain discrepancy by aligning the source and target data distributions , but it requires to access the source-domain data which often raises concerns in data privacy , data portability , and data transmission efficiency . In this work , we study unsupervised model adaptation ( UMA ) , an alternative setting that aims to adapt source-trained models to fit target data distribution without accessing the source-domain data . Under the UMA setting , the only information carried forward is a portable source-trained model which is usually much smaller than the source-domain data and can be transmitted more efficiently [ 45 , 42 , 43 , 72 , 48 ] as illustrated in Table 1 . Beyond that , the UMA setting also alleviates the concern of data privacy and intellectual property effectively . On the other hand , the absence of the labeled source-domain data makes domain adaptation much more challenging and susceptible to collapse . ∗Corresponding author . 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . To this end , we develop historical contrastive learning ( HCL ) that aims to make up for the absence of source data by adapting the source-trained model to fit target data distribution without forgetting source hypothesis , as illustrated in Fig . 1 . HCL addresses the UMA challenge from two perspectives . First , it introduces historical contrastive instance discrimination ( HCID ) that learns target samples by comparing their embeddings generated by the current model ( as queries ) and those generated by historical models ( as keys ) : a query is pulled close to its positive keys while pushed apart from its negative keys . HCID can thus be viewed as a new type of instance contrastive learning for the task of UMA with historical models , which learns instance-discriminative target representations without forgetting source-domain hypothesis . Second , it introduces historical contrastive category discrimination ( HCCD ) that pseudo-labels target samples for learning category-discriminative target representations . Specifically , HCCD re-weights the pseudo labels according to their consistency across the current and historical models . The proposed HCL tackles UMA with three desirable features : 1 ) It introduces historical contrast and achieves UMA without forgetting source hypothesis ; 2 ) The HCID works at instance level , which encourages to learn instance-discriminative target representations that generalize well to unseen data [ 98 ] ; 3 ) The HCCD works at category level ( i.e. , output space ) which encourages to learn category-discriminative target representation that is well aligned with the objective of down-stream tasks . The contributions of this work can be summarized in three aspects . First , we investigate memorybased learning for unsupervised model adaptation that learns discriminative representations for unlabeled target data without forgetting source hypothesis . To the best of our knowledge , this is the first work that explores memory-based learning for the task of UMA . Second , we design historical contrastive learning which introduces historical contrastive instance discrimination and category discrimination , the latter is naturally aligned with the objective of UMA . Third , extensive experiments show that the proposed historical contrastive learning outperforms state-of-the-art methods consistently across a variety of visual tasks and setups . 2 Related Works . Our work is closely related to several branches of research in unsupervised model adaptation , domain adaptation , memory-based learning and contrastive learning . Unsupervised model adaptation aims to adapt a source-trained model to fit target data distributions without accessing source-domain data . This problem has attracted increasing attention recently with a few pioneer studies each of which focuses on a specific visual task . For example , [ 45 , 46 ] freezes the classifier of source-trained model and performs information maximization on target data for classification model adaptation . [ 42 ] tackles classification model adaptation with a conditional GANs that generates training images with target-alike styles and source-alike semantics . [ 43 ] presents a self-entropy descent algorithm to improve model adaptation for object detection . [ 72 ] reduces the uncertainty of target predictions ( by source-trained model ) for segmentation model adaptation . [ 48 ] introduces data-free knowledge distillation to transfer source-domain knowledge for segmentation model adaptation . Despite the different designs for different tasks , the common motivation of these studies is to make up for the absence of source data in domain adaptation . [ 40 ] and [ 88 ] tackle source-free domain adaptation from a generative manner by generating samples from the source classes and generating reference distributions , respectively . We tackle the absence of source data by a memory mechanism that encourages to memorize source hypothesis during model adaptation . Specifically , we design historical contrastive learning that learns target representations by contrasting historical and currently evolved models . To the best of our knowledge , this is the first work that explores memory mechanism for UMA . Domain adaptation is related to UMA but it requires to access labeled source data in training . Most existing work handles UDA from three typical approaches . The first exploits adversarial training to align source and target distributions in the feature , output or latent space [ 78 , 51 , 76 , 14 , 96 , 64 , 66 , 79 , 34 , 77 , 35 , 21 , 94 , 29 , 10 , 9 , 80 , 20 ] . The second employs self-training to generate pseudo labels to learn from unlabeled target data iteratively [ 103 , 69 , 100 , 102 , 31 , 19 , 90 , 91 ] . The third leverages image translation to modify image styles to reduce domain gaps [ 25 , 71 , 44 , 95 , 26 , 86 , 32 , 33 , 93 , 92 ] . In addition , [ 30 ] proposes a categorical contrastive learning for domain adaptation . Memory-based learning has been studied extensively . Memory networks [ 81 ] as one of early efforts explores to use external modules to store memory for supervised learning . Temporal ensemble [ 41 ] , as well as a few following works [ 74 , 12 ] extend the memory mechanism to semi-supervised learning . It employs historical hypothesis/models to regularize the current model and produces stable and competitive predictions . Mean Teacher [ 74 ] leverages moving-average models as the memory model to regularize the training , and similar idea was extended for UDA [ 16 , 99 , 4 , 52 ] . Mutual learning [ 97 ] has also been proposed for learning among multiple peer student models . Most aforementioned methods require labeled data in training . They do not work very well for UMA due to the absence of supervision from the labeled source data , by either collapsing in training or helping little in model adaptation performance . We design innovative historical contrastive learning to make up for the absence of the labeled source data , more details to be presented in the ensuing subsections . Contrastive learning [ 82 , 87 , 22 , 54 , 101 , 24 , 56 , 75 , 36 , 11 ] learns discriminative representations from multiple views of the same instance . It works with certain dictionary look-up mechanism [ 22 ] , where a given image x is augmented into two views , query and key , and the query token q should match its designated key k+ over a set of negative keys k− from other images . Existing work can be broadly classified into three categories based on dictionary creation strategies . The first creates a memory bank [ 82 ] to store all the keys in the previous epoch . The second builds an end-to-end dictionary [ 87 , 75 , 36 , 11 ] that generates keys using samples from the current mini-batch . The third employs a momentum encoder [ 22 ] that generates keys on-the-fly by a momentum-updated encoder . Other related source-free adaptation works . [ 7 ] considers supervised continual learning from previously learned tasks to a new task , which learns representations using the contrastive learning objective and preserves learned representations using a self-supervised distillation step , where the contrastively learned representations are more robust against the catastrophic forgetting for supervised continual learning . [ 38 ] addresses a source-free universal domain adaptation problem that does not guarantee that the classes in the target domain are the same as in the source domain . [ 39 ] propose a simple yet effective solution to realize inheritable models suitable for open-set source-free DA problem . 3 Historical Contrastive Learning . This section presents the proposed historical contrastive learning that memorizes source hypothesis to make up for the absence of source data as illustrated in Fig . 2 . The proposed HCL consists of two key designs . The first is historical contrastive instance discrimination which encourages to learn instance-discriminative target representations that generalize well to unseen data [ 98 ] . The second is historical contrastive category discrimination that encourages to learn category-discriminative target representations which is well aligned with the objective of visual recognition tasks . More details to be described in the ensuring subsections . 3.1 Historical Contrastive Instance Discrimination . The proposed HCID learns from unlabeled target samples via contrastive learning over their embeddings generated from current and historical models : the positive pairs are pulled close while negative pairs are pushed apart . It is a new type of contrastive learning for UMA , which preserves sourcedomain hypothesis by generating positive keys from historical models . HCID works at instance level and encourages to learn instance-discriminative target representations that generalize well to unseen data [ 98 ] . HCID loss . Given a query sample xq and a set of key samples Xk = { xk0 , xk1 , xk2 , ... , xkN } , HCID employs current model Et to encode the query qt = Et ( xq ) , and historical encoders Et−m to encode the keys kt−mn = E t−m ( xkn ) , n = 0 , · · · , N . With the encoded embeddings , HCID is achieved via a historical contrastive loss LHisNCE , minimization of which pulls q close to its positive key kt−m+ while pushing it apart from all other ( negative ) keys : LHisNCE = ∑ xq∈Xtgt − log exp ( qt·kt−m+ /τ ) rt−m+∑N i=0 exp ( q t·kt−mi /τ ) r t−m i ( 1 ) where τ is a temperature parameter [ 82 ] , r indicates the reliability of each key kt−mn , with which we reweight the similarity loss of each key to encourage to memorize well-learnt instead of poorly-learnt historical embeddings . In this work , we use the classification entropy to estimate the reliability of each key . The positive key sample is the augmentation of the query sample [ 22 , 36 ] , and all the rest are negative keys . Remark 1 Note LHisNCE in Eq.1 has a similar form as the InfoNCE loss [ 56 , 22 ] . InfoNCE can actually be viewed as a special case of HisNCE , where all the query and keys are encoded by the current model ( m = 0 ) and the reliability is fixed ( rt−mi = 1 , ∀i ) . For HisNCE , we assign each key a reliability score to encourage to memorize the well-learnt historical embeddings only . It is also worth noting that Eq . 1 only shows historical contrast with one historical model for simplicity . In practice , we could employ multiple historical models to comprehensively distill ( memorize ) the well-learnt embeddings from them . It could be achieved by computing Eq.1 multiple times .

Summary: This paper proposes a generic method HCL to solve the source data-free domain adaptation problem across various computer vision tasks. Based on the memory mechanism, the method involves two designs to learn discriminative features while preserving the source hypothesis. The first design HCID leverages the contrastive instance discrimination where positive pairs are made of the feature from the current model and features from historical models including the original source model and model in previous epochs. The second design HCCD leverages the consistency between the current model and previous models to select confident pseudo labels for task-specific self-training. Experiments across various vision tasks and domain adaptation settings validate the effectiveness of both designs and the generic method.