Model Adaptation: Historical Contrastive Learning for Unsupervised Domain Adaptation without Source Data
Authors: Jiaxing Huang
Dayan Guan
Aoran Xiao
Shijian Lu
Venue: NeurIPS 2021 Poster (2021)
Keywords: Unsupervised Domain Adaptation
Model Adaptation
Contrastive Learning
Segmentation
Detection
Classification
Transfer Learning

Abstract: Unsupervised domain adaptation aims to align a labeled source domain and an unlabeled target domain, but it requires to access the source data which often raises concerns in data privacy, data portability and data transmission efficiency. We study unsupervised model adaptation (UMA), or called Unsupervised Domain Adaptation without Source Data, an alternative setting that aims to adapt source-trained models towards target distributions without accessing source data. To this end, we design an innovative historical contrastive learning (HCL) technique that exploits historical source hypothesis to make up for the absence of source data in UMA. HCL addresses the UMA challenge from two perspectives. First, it introduces historical contrastive instance discrimination (HCID) that learns from target samples by contrasting their embeddings which are generated by the currently adapted model and the historical models. With the historical models, HCID encourages UMA to learn instance-discriminative target representations while preserving the source hypothesis. Second, it introduces historical contrastive category discrimination (HCCD) that pseudo-labels target samples to learn category-discriminative target representations. Specifically, HCCD re-weights pseudo labels according to their prediction consistency across the current and historical models. Extensive experiments show that HCL outperforms and state-of-the-art methods consistently across a variety of visual tasks and setups.

Model Adaptation: Historical Contrastive Learning for Unsupervised Domain Adaptation without Source Data
Jiaxing Huang, Dayan Guan, Aoran Xiao, Shijian Lu
School of Computer Science Engineering, Nanyang Technological University {Jiaxing Huang, Duan,
Shijian. Lu) edu.
Abstract
Unsupervised domain adaptation aims to align labeled source domain and unlabeled target domain, but it requires to access the source data which often raises concerns in data privacy, data portability and data transmission efficiency. We study unsupervised model adaptation (UMA), or called Unsupervised Domain Adapta- tion without Source Data, an alternative setting that aims to adapt source-trained models towards target distributions without accessing source data. To this end, we design an innovative historical contrastive learning (HCL) technique that exploits historical source hypothesis to make up for the absence of source data in UMA. HCL addresses the UMA challenge from two perspectives. First, it introduces historical contrastive instance discrimination (HCID) that learns from target sam- ples by contrasting their embeddings which are generated by the currently adapted model and the historical models. With the historical models, HCID encourages UMA to learn instance-discriminatii target representations while preserving the source hypothesis. Second, it introduces historical contrastive category discrimi- nation (HCCD) that pseudo-labels target samples to learn disrrmmiiiiii target representations. Specifically, HCCD re-weights pseudo labels according their prediction consistency across the current and historical models. Extensive experiments show that HCL outperforms and state-of of- the-art methods consistently across a variety of visual tasks and setups.
Introduction
Deep neural networks (DNNs)
have achieved great success in various computer vision 23 but often generalize poorly to new domains due to the inter-domain
28 73
tasks 8 55
60
59 28
73
discrepancy D. Unsupervised domain adaptation (UDA) 778 51 76 64 66 79
[03 J02 25 71
24 86 addresses the inter-domain discrepancy by aligning the source and target data distributions, but requires to access the source-domain data which often raises concerns in data privacy, data portability, and data transmission efficiency.
In this work, we study unsupervised model adaptation (UMA), an alternative setting that aims to adapt source-trained models to fit target data distribution without accessing the source-domain data, Under the UMA setting, the only information carried forward a portable source-trained model which is usually much smaller than the source-domain data and can be transmitted more efficiently 45 42 43 72 (48) as illustrated Table Beyond that, the UMA setting also alleviates the concern of data privacy and intellectual property effectively. On the other hand, the absence of the labeled source-domain data makes domain adaptation much more challenging and susceptible to collapse.
"Corresponding author.
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

Table 1: Source data have much larger sizes than source- trained models.
Semantic segmentation
Object detection
Image classification
Storage size (MB)
GTAS
SYNTHIA
Cityscapes
VisDA17
Source dataset
62, 873.6
22, 323.2
12, 697.99. 6
7, 78884.8
Source-trained model
179.1
179.1
563.4
172.6
To this end, we develop historical contrastive learning (HCL) that aims to make up for the absence of source data by adapting the source-trained model to fit target data distribution without forgetting source hypothesis, as illustrated in Fig. D HCL addresses the UMA challenge from two perspectives. First, it introduces historical contrastive instance discrimination (HCID) that learns target samples by comparing their embeddings generated by the current model (as queries) and those generated by bistorical models (as keys): a query is pulled close to its positive keys while pushed apart from its negative keys. HCID can thus be viewed task of UMA with historical models, which learns instance-discriminatiy target representations without forgetting source-domain hypothesis. Second, it introduces historical contrastive category discrimination (HCCD) that pseudo-labels target samples for learning category-discrminntte target representations. Specifically, HCCD re-weights the pseudo labels according to their consistency across the current and historical models.
type of instance contrastive learning for the
The proposed HCL tackles UMA with three desirable features: 1) It introduces historical contrast and achieves UMA without forgetting source hypothesis: 2) The HCID works at instance level, which encourages to learn sestance-discrmmngg
target representations that generalize well to unseen data [98]: 3) The HCCD works at category level (i.e., output space) which encourages to learn diteorry- target representation that is well aligned with the objective of down-stream tasks.
The contributions of this work can be summarized in three aspects. First, we investigate memory- based learning for unsupervised model adaptation that learns discriminative representations for unlabeled target data without forgetting source hypothesis. To the best of our knowledge, this the first work that explores memory-based learning for the task of UMA. Second, we design historical contrastive learning which introduces historical contrastive instance discrimination and category discrimination, the latter is naturally aligned with the objective of UMA. Third, extensive experiments show that the proposed historical contrastive learning outperforms state-of-ther methods consistently across a variety of visual tasks and setups.
2
Related Works
Our work is closely related several branches of research in unsupervised model adaptation, domain adaptation, memory-based learning and contrastive learning.
Unsupervised model adaptation aims to adapt a source-trained model to fit target data distributions without accessing source-domain data. This problem has attracted increasing attention recently with a few pioneer studies each of which focuses on specific visual task. For example, [45] 46 freezes the classifier of source- saurce-trained model and performs information maximization on target data for [42] tackles classification model adaptation with a conditional GANs that generates training images with target-alike styles and source-alike semanties. (43 presents a
classification model adaptation.
self-entropy descent algorithm to improve model adaptation for object detection. 772 reduces the
uncertainty of target predictions (by source-traine model) for segmentation model adaptation. [48]
introduces data-free knowledge distillation transfer source-domain knowledge for segmentation model adaptation. Despite the different designs for different tasks, the common motivation of these studies is to make up for the absence of source data in domain adaptation. [40] and (88) tackle
source-fin free domain adaptation from a generative manner by generating samples from the source classes and generating reference distributions, respectively.
We tackle the absence of source data by a memory mechanism that encourages to memorize source hypothesis during model adaptation. Specifically, we design historical contrastive learning that learns target representations by contrasting historical and currently evolved models. To the best of our knowledge, this is the first work that explores memory mechanism for UMA.

---

Xete/Xige:
Souree/taget data
G+16 Current/tisoocical modal
Huswet Historical contrastive instance discrimination loss
Sege Sourestarget feature
f/ Cureen/tistorical feature
E Historical contrastie category discrimination loss
Unavailable in unsupervised model adaptation
Memory for the absence of source supervision
Xare
fave
sup
G
figtt
Shared
LyDA
+
EasST
Xagt
G
fige
Xegt
G
Frge
Unsupervised Domain ddapaation
Historical Contrastive Leaming for UMA
Figure 1: Illustration of unsupervised domain adaptation, unsupervised model adaptation and the proposed historical contrastive learning which exploits historical source hypothesis (or memorized knowledge) to make up for the absence of source supervision in the process of UMA. Here the historical source hypothesis could be the original source hypothesis G (i.e. t=m, trained using the labeled source data only), the adapted source hypothesis G (i.e. m < I, trained in the last m epoch), or other types of previous models.
Domain adaptation is related to UMA but it requires to access labeled source data in training. Most existing work handles UDA from three typical approaches. The first exploits adversarial training to align source and target distributions in the feature, output latent space [7] 51 76 4 96
64 66
(79 34 77 (35 21 94] (2 10,
|9 80, 20). The second employs self-training to generate pseudo labels
learn from unlabeled target data iteratively [103] 69 [00 [02
31
19
90 911 The third leverages
image translation to modify image styles to reduce domain gaps (2 71 44 95 26
86
32
33 93 92
In addition,
30 proposes categorical contrastive learning for domain adaptation.
Memory-based learning has been studied extensively. Memory networks [81] as one of carly efforts explores to external modules to store memory for supervised learning. Temporal ensemble 41]. as well as a few following works [74, 12] extend the memory mechanism to semi-supervised learning. It employs historical hypothesismmodels to regularize the current model and produces stable and competitive predictions. Mean Teacher 774 leverages moving-average models as the memory
model to regularize the training, and similar idea was extended for UDA [16 99 4 [2]
Mutual
learning 977 has also been proposed for learning among multiple peer student models.
Most aforementioned methods require labeled data in training. They do not work very well for UMA due to the absence of supervision from the labeled source data, by either collapsing training or helping little in model adaptation performance. We design innovative historical contrastive learning to make up for the absence of the labeled source data, more details to be presented in the ensuing subsections.
learns discriminative representations from multiple views of the same instance. It works with certain dictionary look-up mechanism 22] where a given image z is augmented into two views, query and key, and the query token q should match its designated key li over set of negative keys k_ from other images. Existing work can be broadly classified into three categories based on dictionary creation strategies. The first creates memory bank [82) to store all the keys in the previous epoch. The second builds an end-to-end dictionary [87 75 36 II that generates keys using samples from the current bini-batch. The third employs momentum encoder [22 that generates keys on-the-fly by a momentum-updee uuddateed encoder.
Contrastive learning 82
Other related source-free adaptation works. considers supervised continual learning from previously learned tasks to a new task, which learns representations using the contrastive learning objective and preserves learned representations using self superviseed distillation step, where the contrastively learned representations are more robust against the catastrophic forgetting for supervised continual learning. [38] addresses a source-free universal domain adaptation problem that does not
guarantee that the classes in the target domain are the same as in the source domain. [39 propose a simple yet effective solution to realize inheritable models suitable for open-set source-free DA problem.
3

---

E1: Current eacoder E
f:; Current feature Historical features
ct Cureett classifier
pf: Curent prediction
Historical eacodars
ff
c= Historical classifiers
p0 ///2" /pfm Historical predictions
Keys f
Keys
E 54-m
Ct-m
Data A Augeetattio
HUNNCE
E
fe
C
9
Query
Query
Historical Contrastive Instance Discrimination
Historical Contrastive Category Discrimination
Figure The proposed historical contrastive learning consists of two key designs including historical contrastive instance discrimination (HCID) and historical contrastive category discrimination (HCCD). HCID learns from target samples by contrasting their embeddings generated by the current model (as queries) and historical models (as keys), which learns instance-disen discriminative target representations. HCCD pseudo-labels target samples to learn category- -iigory-discrmminaiiee target representations, where the pseudo labels are re-wed weighted adaptively according to the prediction consistency across the current and historical models.
3 Historical Contrastive Learning
This section presents the proposed historical contrastive learning that memorizes source hypothesis to make up for the absence of source data as illustrated Fig. The proposed HCL consists of two key designs. The first is historical contrastive instance discrimination which encourages to learn eastane-ddisciminativ target representations that generalize well to unseen data (98). The second
historical contrastive category discrimination that encourages to learn integory-discriminee target representations which well aligned with the objective of visual recognition tasks. More details to be described in the ensuring subsections.
3.1 Historical Contrastive Instance Discrimination
The proposed HCID learns from unlabeled target samples via contrastive learning over their embed- dings generated from current and historical models: the positive pairs pulled close while negative pairs are pushed apart. is a new type of contrastive learning for UMA, which preserves source- domain hypothesis by generating positive keys from historical models. HCID works at instance level and encourages to learn eestance-discrminnad target representations that generalize well to unseen data 98
HCID loss. Given query sample Igg and set of key samples X₂ = (ri, Tky Tky Fx }. HCID employs eurrent model E' to encode the query o' E' (zx), and historical encoders Et-m encode the keys K4,- = gg--m(xx,))) n = 0, ... N. With the encoded embeddings. HCID achieved via a historical contrastive loss CHiANCE, minimization of which pulls q close to its positive key k. while pushing it apart from all other (negative) keys:
explq" -Kt /r)r4=
LHINNCE
Σ E
log
(1)
Σ === 1xppy"-h15-1 /T)
where r is a temperature parameter [82]. indicates the reliability of each key k4-m, with which we reweight the similarity loss of each key to encourage to memorize well-learnt instead of poorly-learn historical embeddings. In this work, we use the classification entropy to estimate the reliability of each key. The positive key sample is the augmentation of the query sample 22 36), and all the rest are negative keys.
Remark 1 Note LIINNE in E has similar form as the InfoNCE loss 56
22
InfoNCE can
actually be viewed as special case of HisNCE, where all the query and keys are encoded by the current model (m 0) and the reliability is fixed (r) 4-m 1, Vi). For HisNCE, assign each key

---

reliability score to encourage to memorize the well-learnt historical embeddings only. It is also worth noting that Eq. only shows historical contrast with one historical model for simplicity. practice, we could employ multiple historical models to comprehensively distill (memorize) the well-learnt embeddings from them. could be achieved by computing Eq/ multiple times.
3.2 Historical contrastive category discrimination
We design HCCD that generates pseudo labels and learns them conditioned historical consistency, i.e. the prediction consistency across the current and historical models. HCCD can be viewed as new type of self-training, where pseudo labels are re-weighted by the historical consistency. works at category level and encourages to learn category-discrimnnate target representations that are aligned with the objective of visual recognition tasks in UMA.
Historical contrastive pseudo label generation. Given an unlabeled sample æ, the current and historical models predict p (a:) (as the query) and p²-m Gt (x:) (as the keys). The pseudo label and the historical consistency the sample are computed by:
i = r(p',, hcon = 1 - Sigmoid([Ip' - pp' 111))
where p K-class probability vector, I is the pseudo label generation function [03 ÿ (1), g(3).... .... g())) is the predicted category label.
102 and
HCCD loss. Given the unlabeled data æ and its historical contrastive pseudo label (g, heon). HCCD performs self-training on target data via weighted cross-entropy loss:
LHiST =
Σ x
Iean x ÿ log Pav
(2)
where heon is the per-sample historical consistency and the predictions of a sample across the current and historical models are consistent, we consider well-learnt sample and increase its influence in self-training. Otherwise, we consider it as pooly-learnt sample and decrease its influence in self-training.
re-weight the self-training loss.
3.3 Theoretical Insights
The two designs in Historical Contrastive Learning (HCL) are inherently connected with some probabilistic models and convergent under certain conditions:
Proposition 1 The historical contrastive instance discrimination (IICID) can be modelled as maximum likelihood problem optimized via Expectation Maximization.
Proposition 2 The HCID is convergent under certain conditions.
Proposition 3
The historical contrastive category discrimination (HCCD) can be modelled as
classification maximum likelihood problem optimized via Classification Expectation Maximization.
Proposition 4 The HCCD is convergent under certain conditions.
The proofs of Proposition 1, Proposition 2, Proposition 3 and Proposition 4 are provided in the appendix, respectively.
Experiments
This section presents experiments including datasets, implementation details, evaluations of the proposed HCL in semantic segmentation, object detection and image classification tasks as well as the discussion of its desirable features.
5

---

Cityscapes Table 2: Experiments semantic segmentation task GTA5
Cityscapes ("SF" denotes source-data
free, i.c., adaptation without source data).
Mathel
Road
SW
Fzre:
Vez
Ternnn
C5r
Far
KaaptSoz
M.3
36.0
15.9
253
23:3
14.8
$5.4
15.3
15.6
58.5
2775
75.7
323
35.5
39
30.1
25.1
42.4
AdvEal 120
x
19.4
33.1
81.0
25.6
25.8
27.2
33.5
24.7
$3.9
36.7
78.8
58.7
30.5
84.8
38.5
44.5
1.7
31.6
32.4
43.5
IDA [3]
x
9066
17.1
87.6
93.1
19.1
79.5
12.4
20.6
85.7
40.5
79.7
56.
14.1
66.3
51.5
88.3
0.0
16.3
85.8
46.1
CRST [00
x
91.0
15.4
80.0
15.7
21.4
12.3
52.8
24.5
85.0
14.1
90.
59.5
24.6
66.1
27.3
90.1
26.9
36.
43.1
49.1
OCED E5
x
92.4
25.3
X2,3
31.2
29.1
32.5
5322
35.6
555
34.8
54.2
58.9
32.2
54.7
40.6
46.1
2.1
31.1
32.7
45.6
UK [23
7
123
35.2
81.6
3335
13.3
37.1
173
12.1
$4.2
35.9
838
57.7
24.1
11.7
27.3
44.3
6.9
29.1
00.4
43.1
+H0
96.1
25.4
37.
16.0
K34
86.0
41.9
44.7
49.1
2033
45.3
4HCL
49.3
HCID
21.9
36.5
13.7
BCCD
910
23.1
82.8
37.2
59.5
12.5
46.7
HCI.
92.0
24.5
85.0
37.6
83.3
99.4
85.6
96.5
41.0
Table 3: Experiments on semantic segmentation task SYNTHIA + Cityscapes ("SF" denotes
source-data free, i.e., adaptation without source data).
build
Fole"
AdapoSop
76
84.3
42.7
77.5
4.7
7.0
77.9
82.5
54.3
21.0
72.3
32.2
15.5
32.3
26.7
AdvEm (0
85.6
42.2
79.7
8.7
0.4
25.9
5.4
8.1
80.4
84.1
57.9
23.8
75.3
36.4
14.2
33.0
41.2
$8.0
IDA SSl
64.3
37.7
79.5
5.3
24.9
92
8.4
80.0
84.1
57.2
23.0
76.0
36.1
20.3
36.5
41.7
48.9
CRST
19.4
CCCDAISS
44.9
79.5
94
11.8
26,1
76.8
39.
21.5
32.1
43.9
50.0
(2]
59.3
24.6
77.0
14.0
18
31.5
18.3
32.0
R31
80.4
46.3
17.8
76.7
1700
18.5
34.6
30.6
45.0
+HCL
78.7
23.2
79.5
24.8
44.1
51.1
[S)
#HCL
782
25.3
79.6
7.3
0.2
37.7
21
30.9
N0.4
83.3
59.8
29.4
79.2
34.2
24.5
38.9
45.0
51.9
HCI
809
34.9
767
36.1
20.1
28.2
79.1
83.1
55.6
25.6
78.8
32.7
24.1
32.7
43.5
90.2
4.1
Datasets
UMA for semantic segmentation is evaluated challenging tasks GTAS 661
and SYNTHIA 62) - Cityscapes. GTA5 has 24, 966 synthetic images and shares 19 cate- gories with Cityscapes. SYNTHIA contains 9, 400 synthetic images and shares 16 categories with Cityscapes. Cityscapes has 2975 and 500 real-world images for training and validation, respectively.
15
UMA for object detection is evaluated on tasks Cityscapes - Foggy Cityscapes [68] and Cityscapes + BDD100 100k (89] Foggy Cityscapes is derived by applying simulated fog to the 2.5 Cityscapes images. BDD100K has 70k training images and 10k validation images, and shares 7 categories with Cityscapes. We evaluate a subset of BDD100k (i.c., daytime) as 884 65 [3]
for fair comparisons.
UMA for image classification is evaluated on benchmarks VisDA17 (58
and Office-31 [63
VisDA17 has 152, 409 synthetic images as the source domain and 55, 400 real images of 12 shared categories as the target domain. Office-31 has 4110 images of three sources including 2817 from Amazon, 795 from Webcam and 498 from DSLR (with 31 shared categories). Following [02 63 the evaluation is six adaptation tasks A-*, D-W, W DD AD D and W-+A
4.2 Implementation Details
Semantic segmentation: Following 79
103 we employ DeepLab-V2 [8] as the segmentation
model. We adopt SGD 3 with momentum 0.9, weight decay 1e and learning rate 2.5e
where the learning rate is decayed by polynomial annealing policy g
adopt Faster R-CNN (60 the detection model. We use SGD (3] with momentum 0.9 and weight decay 5e - 4. The learning rate is le 3 for first 50k iterations and then decreased to
Object detection:
Following 84 65 [3),
for another 20k iterations.
63 70), we adopt ResNet-101 and ResNet-50 23 as the classification models for VisDA17 and Office-31. respectively. We use SGD 3 with momentum 0.9, weight decay 5e 4, learning rate le and batch size 32.
Image classification; Following [02
4.3 Unsupervised Domain Adaption for Semantic Segmentation
We evaluated the proposed HCL UMA-based semantic segmentation tasks GTA5 -+ Cityscapes and SYNTHIA Cityscapes. Tables 2 and 3 show experimental results in mean Intersection- over-Union (mloU). We can see that HCL. outperforms state-of-the-art UMA methods by large margins. In addition, HCL is complementary to existing UMA methods and incorporating it as

---

Table 4: Experiments object detection task Cityscapes - Foggy Cityscapes ("SF" denotes
source-data free, i.c., adaptation without source data).
Method
SF
persen
rider
car
truck
bus
train
meyele
bicycle
mAF
DA [33
X
25.0
31.0
40.3
22.1
35.3
20.2
20.0
27.1
27.6
MLDA 83
X
33.2
44.8
28.2
41.8
30.5
36.5
DMA 37
X
30.8
40.5
44.3
27.2
38.4
34.5
28.4
32.2
34,6
CAFA 27
x
41.9
38.7
56.7
22.6
41.5
26.8
24.6
35.5
36.0
SWDA 65
x
36.2
35.3
43.5
30.0
29.9
42.3
32.6
24.5
34.3
CRDA
84
x
32.9
43.8
49.2
27.2
45.1
36.4
30,3
34.6
37.4
STOD 43
25.5
44.5
40.7
33.2
22.2
28.4
34.1
39.0
33.5
+HCL
/
HCL
38.7
46.0
47.9
33.0
45.7
38.9
32.8
34,9
39.7
Table Experiments on object detection task Cityscapes + BDD100k ("SF" denotes source-data
free, i.c., adaptation without source data).
person
rider
car
truck
bus
meycle
bicycle
mAP
DA 113
29.4
26.5
44.6
14.3
16.8
20.6
24.0
SWDA 65
30.2
29.5
45.7
15.2
18.4
17.1
21.2
25.3
CRDA 84
31.4
31.3
46.3
19.5
18.9
17.3
23.8
26,9
SFOD
32.4
32.6
50.4
20.6
23.4
18.9
25.0
29.0
+HCL
V
33,9
34.4
52.8
22.1
25.3
22.6
26.7
31.1
HCL
32.7
33.2
52.0
21,3
25.6
21.5
26.0
30,3
denoted by "+HCL" improves the existing UMA methods clearly and consistently, Furthermore, HCL even achieves competitive performance as compared with state-of-the-arr UDA methods (labeled by Xin the column SF) which require to access the labeled source data in training. Further, We conduct ablation studies of the proposed HCL over the UMA-based semantic segmentation task GTA5 Cityscapes. As the bottom of Table 2 shows, either HCID HCCD achieves comparable performance. In addition, HCID and HCCD offer orthogonal self-supervision signals where HCID focuses instance-level discrimination between queries and historical keys and HCCD focuses on category-level discrimination among samples with different pseudo category labels. The two designs thus complementary and the combination of them in HCL produces the best segmentation.
4.4 Unsupervised Domain Adaptation for Object Detection
We evaluated the proposed HCL over the UMA-based object detection tasks Cityscapes - Foggy Cityscapes and Cityscapes -> BDD100K. Tables 4 and 5 show experimental results. We can observe that HCL outperforms state-of-the-art UMA method SFOD clearly. Similar to the semantic segmentation experiments, HCL achieves competitive performance compared with stateooftth-aan UDA methods (labeled by Xin column SF) which require to access labeled source data in training.
4.5 Unsupervised Domain Adaptation for Image Classification
We evaluate the proposed HCL over the UMA-based image classificat tasks VisDA17 and Office-31 31. Tables (n show experimental results. We can observe that HCL outperforms state-of-the- UMA methods clearly. Similar to the semantic segmentation and object detection experiments, HCL achieves competitive performance as compared with state-of-the-art UDA methods (labeled by X) which require to access labeled source data in training.
4.6 Discussion
Generalization across computer vision tasks: We study how HCL generalizes across computer vision tasks by evaluating it over three representative tasks semantic segmentation, object detection and image classification. Experiments in Tables 7 show that HCL achieves competitive performance consistently across all three visual tasks, demonstrating the generalization ability of HCL across computer vision tasks.
Complementarity studies: We study the complementarity of our proposed HCL by combining it with existing UMA methods. Experiments in Table (the row highlighted by "+HCL") shows that incorporating HCL boosts the existing UMA methods consistently.

---

Table 6: Experiments image classification benchmark VisDA17 ("SF" denotes source-date free,
i.c., adaptation without source data).
Method
SF
Acro
Bike
Bus
Car
Horse
Knifc
Motor
Person
Plant
Skatcheard
Train
Truck
Mcan
17
77.7
65.1
51.9
82.8
57.4
ENT 1B
X
$0.3
15.5
75.8
48.3
77.9
27.3
69.7
40.2
46.5
46.6
79.3
16.0
57.0
60.9
79.6
88.6
83.0
71.9
CBST [03
87.2
78.8
56.5
55.4
85.1
79.2
83.8
77.7
82.8
88.8
69.0
72.0
76.4
CRST [122
88.0
79.2
N2.5
81.4
85.6
73.9
68.8
78.
3C-GAN (22
94,8
73.4
68.8
74.8
93.1
95.4
$8.6
84.7
89.1
84.7
83.5
48.1
81.6
.HCI.
93.8
86.6
84,1
74.3
93.2
95.0
88.4
85.0
90.4
85.2
84.5
49.8
84.2
SHOT
86.4
51.1
80.4
-HCL
94.3
87.0
82.6
70.6
92.0
93.2
87.0
80.6
89.6
86.8
84.6
58.7
83.9
HCL
93.3
85.4
80.7
88.1
86.6
80.0
74.7
83.5
Table Experiments on image classification benchmark Office-31 ("SF" denotes source-data free,
i.e., adaptation without source data).
Method
SF
A-W
D-#
W-+D
A-D
D-A
W-+A
Mean
DAN
(49
99.6
63.6
62.8
DANN
17
82.0
99.1 1
79.7
68.2
67.4
82.2
ADDA
78
X
86.2
96.2
98.4
77.8
69.5
68.9
82.9
JAN 50
X
85.4
97.4
99.8
84.7
68.6
70.0
84.3
CBST 103
X
87.8
98.5
100
86.5
71.2
70.9
85.8
CRST 102
89.4
98.9
100
88.7
72.6
70.9
3C-GAN 42
93.7
98.5
99.8
92.7
75.3
77.8
89.6
+HCL
100.0
77.1
79.0
SHOT [55
72.5
71.4
87.3
+HCL
92.8
99.0
100.0
94.4
76.1
78.3
90.1
HCL
92.5
98.2
100.0
94.7
75.9
77.7
89.8
visualization of feature representation on GTA Cityscapes model adaptation task. We compare HCL. with two state-of-the-ard UMA methods, i.e., "UR" (72) and "SFDA" " [48], and Fig the visualization. We can observe that HCL can learn desirable cestance-iiccimmiatiie yet -ategory-diserrminaatve representations because it incorporates two key designs that work in a complementary manner: 1) HCID works at instance level, which encourages to learn instance-dises -iraane-diserimiaatiee target representations that generalize well to unseen data [98] 2) HCCD works at category level which encourages to learn category- category-discrimg discriminatiiee target representations that are well aligned with the objective of down-stream stream visual tasks. addition, qualitative illustrations are provided in 4ig It can be observed that our proposed HCL clearly outperforms UR and SFDA.
Feature visualization: This paragraph presents the t-SNE 53
Generalization across learning setups: We study how HCL generalizes across learning setups by adapting into two adaptation setups, i.e., partial-ser adaptation and open-set adaptation. Experiments in Table show that HCL achieves competitive performance consistently across both setups.
13
UR (R
SFDA 48
(Cur()rs)
Figure 3: The t-SNE [53 visualization of feature representation on GTA + Cityscapes unsupervsied model adaptation task: Each color in the graphs stands for a category samples (image pixels) with a digit representing the center of category of samples. It can be observed that the proposed HCL outperforms "UR" and "SFDA" qualitatively, by generating instance- dintane-discrimmnaaive and category- discriminative representations for unlabeled target data.

---

Table 8: Experiments on image classification benchmark Office-Hooee under the setup of partial-set DA (domain adaptation) and open-set DA ("SF" denotes source-date free, i.c., adaptation without source data).
Partial-sct DA
SF
A-+C
A-PP
A-RR
C-+A
C-+P
C-R
P--A
P-4C
P--
R-+A
R-+C
R-PP
Mean
SAN
5
X
44.4
68.7
74,6
67.5
65.0
77.8
59.8
44.7
80.1
72.2
50.2
78.7
65.3
59.2
79.5
75.0
68.3
75.7
57.7
84,5
SAPN
38.9
76.3
81.4
70.4
77.8
72.4
75.8
60.4
79.9
71.8
SHOT 45
/
$7.9
83.6
88.8
72.4
74.0
79.0
76.1
60.6
90.1
81,9
68.3
88.5
76.8
+HCL
V
66.9
85.5
92.5
78.3
71.2
87.1
78.3
65.1 1
90.7
82.4
68.7
88.4
80.1
HCL
65.6
85.2
92.7
77.3
76.2
87.2
78.2
66.0
89.1
81.5
68.4
87.3
79.6
Open-set DA
SF
A-C
A-PP
A-RR R
C-+A
C-+P
C-R
P-A
P-+C
P--R
R-+A
R-C
R-P
Mean
OSBP 67
x
56.7
51.5
49.2
67.5
65.5
74.0
62.5
64.8
69.3
80.6
74.7
71.5
65.7
OpeaMax
x
56.5
52.9
53.7
69.1
74.5
64.1
64.0
71.2
80.3
73.0
76.9
66.7
STA 4Y
58.1
54.4
71.6
69.3
81.9
63.4
65.2
74,9
85.0
75.8
80.8
SHOT
45
V
62.5
77.8
83,9
60.9
73.4
79.4
64.7
58.7
83.1
69.1
62.0
82.1
71.5
+HCL
/
64.2
78.3
83.0
61.1
72.2
79.6
65.5
59.3
80.6
80.1
72.0
82.8
73.2
HCL
64.0
78.6
82.4
64.5
73.1
80.1
64.8
59.8
75.3
78.1
69.3
81.5
72.6
UR 72
SFDA 48
HCL(Ours)
Ground Truth
Qualitative illustrations and comparison domain adaptive semantic segmentation task GTA5 - Cityscapes. Our historical contrastive learning (HCL) exploits historical source hypothesis to make up for the absence of source data in UMA, which produces better qualitative results (i.c., semantic segmentation) by preserving the source hypothesis. It can be observed that HCL generates better segmentation results, for example, the sidewalk in the first row, the road in the second and the sky and sidewalk in the third row.
Figure 4:
5 Conclusion
In this work, studied historical contrastive learning, innovative UMA technique that exploits historical source hypothesis to make up for the absence of source data UMA. We achieve historical contrastive learning by novel designs of historical contrastive instance discrimination and historical contrastive category discrimination which learn discriminative representations for target data while preserving source hypothesis simultaneously. Extensive experiments over variety of visual tasks and learning setups show that HCL outperforms tfate-oftthe-ee techniques consistently. Moving forward, we will explore memory-based learning in other transfer learning tasks.
Acknowledgement
This research was conducted at Singtel Cognitive and Artificial Intelligence Lab for Enterprises (SCALE@NTU), which is collaboration between Singapore Telecommunications Limited (Singtel) and Nanyang Technological University (NTU) that is supported by * under its Industry Alignment Fund (LOA Award number: I1701E0013).

---

References
[1] Shai Ben-David, John Blitzer, Koby Crammer, and Fernando Percira. Analysis of representations for domain adaptation. In Advances neural information processing systems, pages 137-144, 2007.
[2] Abhijit Bendale and Terrance E Boult. Towards open set deep networks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 1563-1572, 2016.
[3] Léon Bottou. Large-scale machine learning with stochastic gradient descent. In Proceedings of COMP- STAT"2010, pages 177-186. Springer, 2010.
[4] Qi Cai, Yingwei Pan, Chong-Wor Wah Ngo, Xinmci Tian, Lingyu Duan, and Ting Yao. Exploring object relation in mean teacher for cross-domain detection. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 1457--11466, 2019.
[5] Zhangjie Cao, Mingsheng Long, Jianmin Wang. and Michael I Jordan. Partial transfer learning with selective adversarial networks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 2724-2732, 2018.
[6] Zhangjie Cao, Kaichao You, Mingsheng Long, Jianmin Wang, and Qiang Yang. Learning to transfer examples for partial domain adaptation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pottern Recognition, pages 2985-2999999 2994,,994.25 2019.
[7] Hyuntak Cha, Jaeho Lee, and Jinwoo Shin. Co2l: Contrastive continual learning. arXiv preprint arXiv:2106. 14413, 2021.
[8] Liang-Chich Chen, George Papandreou. Iasonas Kokkinos, Kevin Murphy, and Alan L Yuille. Deeplab: Semantic image segmentation with deep convolutional nets, atrous convolution, and fully connected crfs. IEEE transactions on pattern analysis and machine intelligence, 40(4):834 848, 2017.
[9] Shiming Chen, Wenjie Wang. Beihao Xia, Qinmu Peng. Xinge You. Feng Zheng, and Ling Shao. Free: Feature refinement for generalized zero-shot learning. In Proceedings of the IEEE/CVF International Conference Computer Vision (ICCV), pages 122 131,, 2021.
[10] Shiming Chen, GuoSen Xie, Yang Liu, Qinmu Peng, Baigui Sun, Hao Li, Xinge You, and Ling Shao. Hsva: Hierarchical semantic-visual adaptation for zero-shot learning. In 35th Conference on Neural Information Processing Systems (NeurIPS), 2021.
[11] Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. In International conference machine learning, pages 1.597-1607. PMLR, 2020.
[12] Yanbei Chen, Xiatian Zhu and Shaogang Gong. Semi-supervised deep learning with memory. In Proceedings the European Conference on Computer Vision (ECCV), pages 268 068-283,44 2018.
[13] Yuhua Chen, Wen Li, Christos Sakaridis, Dengxin Dai, and Luc Van Gool. Domain adaptive faster r-cnn for object detection in the wild. In Proceedings of the IEEE conference computer vision and pattern recognition, pages 3339-3348, 2018.
[14] Yuhua Chen, Wen Li, and Lue Van Gool. Road: Reality oriented adaptation for semantic segmentation of urban scenes. In Proceedings of the IEFE Conference Computer Vision and Pattern Recognition, pages 7892-7901, 2018
[15] Marius Cordts, Mohamed Omran, Sebastian Ramos, Timo Rehfeld, Markus Enzweiler, Rodrigo Be- nenson, Uwe Franke, Stefan Roth, and Bernt Schiele. The cityscapes dataset for semantic urban scene understanding. In Proceedings of the IEEE conference computer vision and pottern recognition, pages 3213-3223, 2016.
[16] Geoffrey French, Michal Mackiewicz, and Mark Fisher. Self-ensembling for visual domain adaptation. arXiv preprint 1.XXv:1700.0555 2017.
[17] Yaroslav Ganin, Evgeniya Ustinova, Hana Ajakan, Pascal Germain, Hugo Larochelle, François Laviolette. Mario Marchand, and Victor Lempitsky. Domain-adversarial training of neural networks. The Journal of Machine Learning Research, 17(1):2096 2202000--000,,, 2016.
[18] Yves Grandvalet and Yoshua Bengio. Semi-supervised learning by entropy minimization. In Advances in neural information processing systems, pages 529-536. 2005.
[19] Dayan Guan, Jiaxing Huang, Shijian Lu, and Aoran Xiao. Scale variance minimization for unsupervised domain adaptation in image segmentation. Pattern Recognition, 112:107764, 2021.
10

---

[20] Dayan Guan. Jiaxing Huang. Aoran Xiao, and Shijian Lu. Domain adaptive video segmentation via temporal consistency regularization. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 8053-8064, 2021.
[21] Dayan Guan, Jiaxing Huang. Aoran Xiao, Shijian Lu, and Yanpeng Cao. Uncertainty- aware unsupervised domain adaptation object detection. IEEF Transactions on Multimedia, 2021.
[22] Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. Momentum contrast for unsupervised visual representation learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 9729-9738, 2020.
[23] Kaiming He, Xiangyu Zhang. Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770-778, 2016.
[24] R Devon Hjelm, Alex Fedorov, Samuel and Yoshua Bengio. Learning deep representations by mutual information estimation and maximization. arXiv preprint arXiv:1808.0667 2018.
Karan Grewal, Phil Bachman, Adam Trischler,
[25] Judy Hoffman, Eric Tzeng, Taesung Park, Jun-Yan Zhu, Phillip Isola, Kate Saenko, Alexei Efros, and Trevor Darrell. Cycada: Cycle-consistent adversarial domain adaptation. In International Conference on Machine Learning, pages 1989-1998, 2018.
[26] Weixiang Hong, Zhenzhen Wang, Ming Yang, and Junsong Yuan. Conditional generative adversarial network for structured domain adaptation. In Proceedings of the IELE Conference on Computer Vision and Pattern Recognition, pages 1335-1344, 2018.
[27] Cheng-Chun Hsu, Yi-Hsuan Tsai, Yen- Yu Lin, and Ming-Hsuan Yang. Every pixel matters: Center-aware feature alignment for domain adaptive object detector. In Eurupean Conference Compuder Vision, pages 338. Springer, 2020.
[28] Gao Huang, Zhuang Liu, Laurens Van Der Maaten, and Kilian Q Weinberger. Densely connected convo- lutional networks. In Proceedings of the IEFE conference computer vision and pattern recognition, pages 4700-4708, 2017.
[29] Jiaxing Huang, Dayan Guan, Shijian Lu, and Aoran Xiao. Mlan: Multi-level adversarial network for domain adaptive semantic segmentation. Pattern Recognition, 2021.
[30] Jiaxing Huang, Dayan Guan, Aoran Xiao, and Shijian Lu, Category contrast for unsupervised domain adaptation in visual tasks. arXiv preprint arXiv:2106.0286 2021.
[31] Jiaxing Huang, Dayan Guan, Aoran Xiao, and Shijian Lu. Cross-view regularization for domain adaptive panoptic segmentation. In Proceedings the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 10133-10144, 2021.
[32] Jiaxing Huang, Dayan Guan, Aoran Xiao, and Shijian Lu. Fsdr: Frequency space domain randomization for domain generalization. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 6891-6902, 2021.
[33] Jiaxing Huang, Dayan Guan, Aoran Xiao, and Shijian Lu. Rda: Robust domain adaptation via fourier adversarial attacking. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), 2021.
[34] Jiaxing Huang, Dayan Guan, Aoran Xiao, and Shijian Lu. Semi-supervised domain adaptation via adaptive and progressive feature alignment. arXiv preprint arXiv:2106.02884 2021.
[35] Jiaxing Huang, Shijian Lu, Dayan Guan, and Xiaobing Zhang. Contextual- consistent domain adaptation for semantic segmentation. In European Conference on Computer Vision, pages 705-722. Springer, 2020.
[36] Prannay Khosla, Piotr Teterwak, Chen Wang, Aaron Sama, Yonglong Tian, Phillip Isola, Aaron Maschinot, Ce Liu, and Dilip Krishnan. Supervised contrastive learning. arXiv preprint 2000: 1362, 2020.
[37] Tackyung Kim, Minki Jeong, Seunghyeon Kim, Scokeon Choi, and Changick Kim. Diversify and match: A domain adaptive representation learning paradigm for object detection. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pages 12456-12465, 2019.
[38] Jogendra Nath Kundu, Naveen Venkat, R Venkatesh Babu, et al. Universal source- free domain adaptation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 4544
4553, 2020.
11

---

[39] Jogendra Nath Kundu, Naveen Venkat, Ambareesh Revanur, R Venkatesh Babu, et al. Towards inheritable models for open-set domain adaptation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pages 12376-12385, 2020.
[40] Vinod K Kurmi, Venkatesh K Subramanian, and Vinay P Namboodiri. Domain impression: A source data free domain adaptation method. In Proceedings of the IEFE//VF Winter Conference on Applications of Computer Vision, pages 615-625, 2021.
[41] Samuli Laine and Timo Aila. Temporal ensembling for semi-supervised learning. arXiv preprint arXiv: 1610.0022, 2016.
[42] Rui Li, Qianfen Jiao, Wenming Cao, Hau-San Wong, and Si Wu. Model adaptation: Unsupervised domain adaptation without source data. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 9641-9650, 2020. 2020.
[43] Xianfeng Li. Weijie Chen, Di Xie, Shicai Yang. Peng Yuan, Shiliang Pu, and Yueting Zhuang. free lunch for unsupervised domain adaptive object detection without source data. arXiv preprint arXiv:2012.05400 2020.
[44] Yunsheng Li, Lu Yuan, and Nuno Vasconcelos. Bidirectional learning for domain adaptation of semantic segmentation. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 6936-6945, 2019.
[45] Jian Liang, Dapeng Hu, and Jiashi Feng. Do we really need to access the source data? source hypothesis transfer for unsupervised domain adaptation. In International Conference on Machine Learning, pages 6028-6039, PMLR, 2020.
[46] Jian Liang, Dapeng Hu, Yunbo Wang, Ran He, and Jiashi Feng. Source data-absent unsupervised domain adaptation through hypothesis transfer and labeling transfer. IEEE Transactions Pattern Analysis and Machine Intelligence, 2021.
[47] Hong Liu, Zhangjie Cao, Mingsheng Long, Jianmin Wang, and Qiang Yang. Separate adapt: Open set domain adaptation via progressive separation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 2927-2936, 2019.
[48] Yuang Liu, Wei Zhang, and Jun Wang. Source-free domain adaptation for semantic segmentation. arXiv preprint arXiv:2103. 163777 2021.
[49] Mingsheng Long, Yue Cao, Jianmin Wang, and Michael Jordan. Learning transferable features with deep adaptation networks. In International Conference on Machine Learning, pages 97-105, 2015.
[50] Mingsheng Long, Han Zhu, Jianmin Wang, and Michael Jordan. Deep transfer learning with joint adaptation networks. International conference on machine learning, pages 2208-2211 PMLR, 2017.
[51] Yawei Luo, Liang Zheng, Tao Guan, Junqing Yu, and Yi Yang. Taking closer look at domain shift: Category-level adversaries for semantics consistent domain adaptation. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 2507-2516, 2019.
[52] Zhipeng Luo, Zhongang Cai, Changging Zhou, Gongjie Zhang, Haiyu Zhao, Shuai Yi, Shijian Lu, Hongsheng Li, Shanghang Zhang, and Ziwei Liu. Unsupervised domain adaptive 3d detection with multi-level consistency. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 8866 8875, 2021.
[53] Laurens van der Maaten and Geoffrey Hinton. Visualizing data using t-sne. Journal of muchine learning research, (NNov):25792200) 2008
[54] Ishan Misra and Laurens van der Maaten. Self-supervised learning of pretext- invariant representations. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pages 6707-6717, 2020.
[55] Hyeonwoo Noh, Seunghoon Hong, and Bohyung Han. Learning deconvolution network for semantic segmentation. In Proceedings of the IEEE international conference computer vision, pages 1520-1528. 2015.
[56] Aaron van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predictive coding. arXiv preprint 107.0:1807.03748. 2018
[57] Fei Pan, Inkyu Shin, Francois Rameau, Seokju Lee, and In So Kweon. Unsupervised intra- domain adaptation for semantic segmentation through self superviine arXiv preprint arXiv:2004.07733. 2020.
12

---

[58] Xingchao Peng. Ben Usman, Neela Kaushik, Dequan Wang, Judy Hoffman, and Kate Saenko. Visda: A synthetic-to-real benchmark for visual domain adaptation. In Proceedings of the IEEE Conference on Computer Vision and Puttern Recognition Workshops, pages 2021-2026, 2018.
[59] Joseph Redmon, Santosh Diyvala, Ross Girshick, and Ali Farhadi. You only look once: Unified, real-time object detection. In Proceedings of the IEEE conference on computer vision and pattern recognition. pages 779-788, 2016.
[60] Shaoqing Ren, Kaiming He, Ross Girshick, and Jian Sun. Faster r-cnn: Towards veal-time object detection with region proposal networks. In Advances in neural information processing systems, pages 91-99, 2015.
[61] Stephan R Richter, Vibhav Vineet, Stefan Roth, and Vladlen Koltun. Playing for data: Ground truth from computer games. In European conference on computer vision. pages 102- 118. Springer, 2016.
[62] German Ros, Laura Sellart, Joanna Materzynska, David Vazquez, and Antonio M Lopez. The synthia dataset: A large collection of synthetic images for semantic segmentation of urban scenes. In Proceedings of the IEEE conference computer vision and pattern recognition, pages 3234 3243, 2016.
[63] Kate Saenko, Brian Kulis, Mario Fritz, and Trevor Darrell. Adapting visual category models to new domains. In European conference computer vision, pages 213-226. Springer, 2010.
[64] Kuniaki Saito, Yoshitaka Ushiku, Tatsuya Harada, and Kate Saenko. Adversarial dropout regularization. arXiv preprint arXiv:1711.01577 2017.
[65] Kuniaki Saito, Yoshitaka Ushiku, Tatsuya Harada, and Kate Sacnko. Strong-weak distribution alignment for adaptive object detection. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 6956-69666 6665,6965722 2019.
[66] Kuniaki Saito, Kohei Watanabe, Yoshitaka Ushiku, and Tatsuya Harada. Maximum classifier discrepancy for unsupervised domain adaptation. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 3723-3322333 -7322, 2018.
[67] Kuniaki Saito, Shohei Yamamoto, Yoshitaka Ushiku, and Tatsuya Harada. Open domain adaptation by backpropagation, In Proceedings of the European Conference on Computer Vision (ECCV), pages 153-168, 2018.
[68] Christos Sakaridis. Dengxin Dai, and Luc Van Gool. Semantic foggy scene understanding with synthetic data. International Journal of Computer Vision, 126(9);933-99,, 2018
[69] Fatemeh Sadat Saleh, Mohammad Sadegh Aliakbarian, Mathieu Salzmann, Lars Petersson, and Jose M Alvarez. Effective use of synthetic data for urban scene semantic segmentation. In European Conference on Computer Vision, pages 86-103. Springer, 2018
[70] Swami Sankaranarayanan, Yogesh Balaji, Carlos D Castillo, and Rama Chellappa. Generate to adapt: Aligning domains using generative adversarial networks. In Proceedings of the IEEE Conference on Computer Vision and Puttern Recognition, pages 8503-850 8512, 2018
[71] Swami Sankaranarayanan, Yogesh Balaji. Arpit Jain, Ser Nam Lim, and Rama Chellappa. Learning from synthetic data: Addressing domain shift for semantic segmentation. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 7752-3761, 2018.
[72] Prabhu Teja Sivaprasad and Francois Fleuret. Uncertainty reduction for model adaptation in semantic segmentation. In Proceedings the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), number CONF. JEEE, 2021.
[73] Christian Szegedy. Wei Liu. Yangqing Jia, Pierre Sermanet, Scott Reed, Dragomir Anguelov, Dumitru Erhan, Vincent Vanhoucke, and Andrew Rabinovich. Going deeper with convolutions. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 1-9, 2015.
[74] Antti Tarvainen and Harri Valpola. Mean teachers are better role models: Weight-averagel consistency targets improve semi-supervised deep learning results. In Advances in neural information processing systems, pages 1 1195-1204 2017.
[75] Yonglong Tian. Dilip Krishnan, and Phillip Isola. Contrastive multiview coding, arXiv preprint arXiv: (XXi::906600849 2019.
[76] Yi-Hsuan Tsai, Wei-Chih Hung. Samuel Schulter, Kihyuk Sohn, Ming Hasun Yang, and Manmohan Chandraker. Learning to adapt structured output space for semantic segmentation. In Proceedings of the IEEE Conference Computer Vision and Pattern Recognition, pages 1472-7483 2018
13

---

[77] Yi-Hsuan Tsai, Kihyuk Sohn, Samuel Schulter, and Manmohan Chandraker. Domain adaptation for structured oulput via discriminative patch representations. In Proceedings of the IEEE International Conference on Computer Vision, pages 1456-1465, 2019.
[78] Fric Treng, Judy Hoffman, Kate Saenko, and Trevor Darrell. Adversarial discriminative domain adap- tation. In Proceedings of the IEEF Conference on Computer Vision and Pattern Recognition, pages 7167-7176, 2017.
[79] Tuan-Hung Vu, Himalaya Jain, Maxime Bucher, Matthicu Cord, and Patrick Pérez. Advent: Adversarial entropy minimization for domain adaptation semantic segmentation. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 2517-2526, 2019.
[80] Guoqiang Wei, Cuiling Lan, Wenjun Zeng. Zhizheng Zhang, and Zhibo Chen. Toalign: Task-oriented alignment for unsupervised domain adaptation. 35th Advances in Neural Information Processing Systems, 2021.
[81] Jason Weston, Sumit Chopra, and Antoine Bordes. Memory networks. arXiv preprint 1rXiv:1410.3916. 2014.
[82] Zhirong Wu, Yuanjun Xiong, Stella X Yu. and Dahua Lin. Unsupervised feature learning via non- parametric instance discrimination. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 3733-3742, 2018.
[83] Rongchang Xie, Fei Yu, Jiachao Wang, Yizhou Wang. and Li Zhang. Multi-level domain adaptive learning for cross-domain detection. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV) Workshops, Oct 2019.
[84] Chang- Dong Xu, Xing-Rea Ran Zhao, Xin Jin, and Xiu-Shen Wei. Exploring categorical regularization for domain adaptive object detection. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 11724- 11733, 2020.
[85] Ruijia Xu, Guanbin Li, Jihan Yang, and Liang Lin. Larger norm more transferable: An adaptive feature norm approach for unsupervised domain adaptation. In Proceedings of the IEEE International Conference on Computer Vision, pages 1426-1435, 2019.
[86] Yanchao Yang and Stefano Soatto. Fda: Fourier domain adaptation for semantic segmentation. In Proceedings of the IEEF/CVF Conference Computer Vision and Pattern Recognition, pages 4085- 4095, 2020.
[87] Mang Ye, Xu Zhang, Pong C Yuen, and Shih-Fu Chang. Unsupervised embedding learning via invariant and spreading instance feature. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 6210 6219, 2019.
(88] Hao-Wa Ych, Baoyao Yang, Pong C Yuen, and Tatsuya Harada. Sofa: Source-data-fiee feature alignment for unsupervised domain adaptation. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pages 774-483, 2021.
[89] Fisher Yu, Wenqi Xian, Yingying Chen, Fangchen Liu, Mike Liao, Vashisht Madhavan, and Trevor Darrell. Bdd100k: diverse driving video database with scalable annotation tooling. arXiv preprint Seli:1805.04653 2(5):6, 2018
[90] Yunpeng Zhai, Shijian Lu, Qixiang Ye. Xuebo Shan, Jie Chen, Rongrong Ji, and Yonghong Tian. Ad- cluster: Augmented discriminative clustering for domain adaptive person re-identification. Proceedings of the IEEE/VVF Conference on Computer Vision and Pottern Recognition, pages 9021-9030, 2020.
[91] Yunpeng Zhai, Qixiang Ye, Shijian Lu, Mengxi Jia, Rongrong Ji, and Yonghong Tian. Multiple expert brainstorming for domain adaptive person re-identification. In Computer Vision-ECCV 2020: 16th European Conference, Glasgow, UK, August 28,8 2020, Proceedings, Part VII 16, pages 594 61 Springer, 2020.
[92] Fangneng Zhan, Chuhui Xue, and Shijian Lu. Ga-dan: Geometry- domain adaptation network for scene text detection and recognition. In Proceedings of the IEÉE/CVF International Conference on Computer Vision, pages 9105-9115, 2019.
[93] Jingyi Zhang, Jiaxing Huang, and Shijian Lu. Spectral unsupervised domain adaptation for visual recognition. arXiv preprint arXiv:2106.061 2021.
[94] Jingyi Zhang. Jiaxing Huang, Zhipeng Luo, Gongjie Zhang, and Shijian Lu. Da-detr: Domain adaptive detection transformer by hybrid attention. arXiv preprint arXiv:2103.1008 2021.
14

---

[95] Xiaobing Zhang, Haigang Gong, Xili Dai, Fan Yang, Nianbo Liu, and Ming Liu. Understanding pictograph with facial features: End-to-end sentence-level reading of chinese. In AAAI, pages 9211-9218, 2019.
[96] Yang Zhang, Philip David, and Boqing Gong. Curriculum domain adaptation for semantic segmentation of urban scenes. In Proceedings the IEEE International Conference Computer Vision, pages 2020-2030, 2017.
[97] Ying Zhang, Tao Xiang, Timothy M Hospedales, and Huchuan Lu. Deep mutual learning. In Proceedings the IEEE Conference on Computer Vision and Pattern Recognition, pages 4320 4328, 2018
[98] Nanxuan Zhao, Zhirong Wu, Rynson W. H. Lau, and Stephen Lin. What makes instance discrimination good transfer learning? International Conference on Leurning Representations, 2021.
[99] Zhedong Zheng and Yi Yang. Unsupervised scene adaptation with memory regularization in vivo, arXiv preprint 1rXiv:1912.11164 2019.
[100] Zhun Zhong. Liang Zheng. Zhiming Luo, Shaozi Li, and Yi Yang. Invariance matters: Exemplar memory for domain adaptive person re-identification. Proceedings of IEEE Conference on Computer Vision and Pattern Recognition, pages 598-607, 2019.
[101] Chengxu Zhuang, Alex Lin Zhai, Daniel Yamins. Local aggregation unsupervised learning of visual embeddings. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 6002-6012, 2019.
[102] Yang Zou, Zhiding Yu, Xiaofeng Liu, BVK Kumar, and Jinsong Wang. Confidence regularized self- training. In Proceedings of the IEEE International Conference on Computer Vision, pages 1982-5997) 2019.
[103] Yang Zou, Zhiding Yu, BVK Vijaya Kumar. and Jinsong Wang. Unsupervised domain adaptation for semantic segmentation via class-balanced self- self-training. In Proceedings of the European Conference Computer Vision (ECCV), pages 289 305,005,0000 2018.
15


Reviews:
[{"review_id": "wZwuP6jzqiF", "reviewer": "Reviewer_8f8p", "summary": "This paper proposes a generic method HCL to solve the source data-free domain adaptation problem across various computer vision tasks. Based on the memory mechanism, the method involves two designs to learn discriminative features while preserving the source hypothesis. The first design HCID leverages the contrastive instance discrimination where positive pairs are made of the feature from the current model and features from historical models including the original source model and model in previous epochs. The second design HCCD leverages the consistency between the current model and previous models to select confident pseudo labels for task-specific self-training. Experiments across various vision tasks and domain adaptation settings validate the effectiveness of both designs and the generic method.", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "## Pros\n1. Originality: The HCID design and the HCL method are novel for source data-free domain adaptation. \n2. Quality: The proposed method is technically sound and supported by theoretical analysis. \n3. Clarity: The writing is clear and easy to follow. The paper is well-organized.\n4. Significance: This paper targets the generic unsupervised model adaptation (UMA) problem, which is significant. Demonstratable results are achieved by HCL on various UMA benchmarks.\n\n## Cons\n1. Originality: Missing related works [A, B] on the recent popular UMA problem, although some results in these works may be better than the results in this paper.\n2. Clarity: How HCL works with the two designs (HCID and HCCD) is unclear. $L_{HCL} = L_{HisNCE} + L_{HisST}$? Is there any tradeoff hyper-parameter between two objectives or do the two designs work in the meantime or in a two-stage fashion? If there is a tradeoff, how does the tradeoff hyper-parameter affect the adaptation performance? If there are other hyper-parameters, please clarify their effects similarly. In addition, the implementation or training details of the HCL applied to various computer vision tasks or domain adaptation settings are not provided adequately.\n\nReferences\n\n[A] Kundu et al., Universal source-free domain adaptation, CVPR'20 \n\n[B] Liang et al., Source Data-absent Unsupervised Domain Adaptation through Hypothesis Transfer and Labeling Transfer, arxiv'20\n\n-----------------------------------------------------------------------------------\n## Post-rebuttal\nThe response has addressed most of my concerns on the training details of the proposed method. I decided to keep my original score of 7 because of the strong performance and extensive experiments in the submission.\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Yes.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "5h"}, "extra_text": {}}, {"review_id": "tmchCH5JCEh", "reviewer": "Reviewer_acZE", "summary": "The authors address unsupervised model adaptation (UMA). While unsupervised domain adaptation (UDA) lets the model access the source domain data during an adaptation to a target domain, such access could raise multiple concerns such as privacy and portability. UMA does not allow the model to access the source domain data during adaptation. The proposed method, historical contrastive learning (HCL), utilizes the historical source hypothesis to remedy the lack of source data. The experimental results show that the proposed method achieves state-of-the-art performance for multiple visual recognition tasks.", "questions": "", "limitations": "", "rating": 8, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Originality:\n- UMA has two original components; one is historical contrastive instance discrimination (HCID), and the other is historical contrastive category discrimination (HCCD).\n- The idea of contrastive learning between two models can be found in another paper for continual learning [a]. It is not a problem since [a] is released after the deadline of NeurIPS 2021.\n\n[a] Cha et al., Co2L: Contrastive Continual Learning. arXiv, 2021.\n\nQuality: \n- While there are already many references, the authors should refer to the following papers.\n- [b] addresses source-free domain adaptation under universal domain adaptation settings, which means there is no guarantee that the classes in the target domain are the same as in the source domain. Since UMA in this paper is a variant of closed domain adaptation, [b] can be considered a more complex problem.\n- [c] and [d] address source free domain adaptation for image classification. [c] does not compare the performance to those of existing UDA methods. [d] reports some results on VisDA, Office-31, and Office-Home. [d] uses AlexNet for Office-31; therefore, the performances are worse than those in Table 7.\n- By the way, [37] is already published in CVPR 2021.\n\n[b] Kundu et al., Universal Source-Free Domain Adaptation. CVPR, 2020.\n\n[c] Kurmi et al., Domain Impression: A Source Data Free Domain Adaptation Method. WACV, 2021.\n\n[d] Yeh et al., SoFA: Source-data-free Feature Alignment for Unsupervised Domain Adaptation. WACV, 2021.\n\nClarity: \n- The motivation and formulation are clearly described.\n- The theoretical insights in Section 3.5 clarifies the use of the proposed method for UMA.\n\nSignificance:\n- This paper's major strength is throughout experiments for various visual recognition tasks such as segmentation, detection, and classification. HCL itself shows the state-of-the-art performance; moreover, the combination with other state-of-the-art methods shows that HCL is complementary to each.\n\nPost-rebuttal:\n- I have read the other reviews and all responses from the authors. There are no concerns to recommend this paper for acceptance. Therefore, I would like to keep my first score.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "In the checklists, there are descriptions of the limitations and the societal impacts. The societal impact here is a positive one, but the reviewer agrees with the authors.", "review_text": "", "extra_scores": {"ethical_concerns": "There are no ethical concerns.\n", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "2 hours"}, "extra_text": {}}, {"review_id": "fo7vpc2fy12", "reviewer": "Reviewer_DhdD", "summary": "In this paper, the authors propose to solve the problem of unsupervised model adaptation. In this task, the aim is that given a classifier trained on a source dataset and a set of unlabelled target examples, an unsupervised model adaptation is performed to improve the performance of the model on the target dataset without requiring access to source examples. The method is evaluated on standard domain adaptation datasets for classification, detection and segmentation. The approach is related to EM and some theoretical insights provided.\n", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Main review\nThe main insight adopted in this paper is that as the source samples are not provided, the model embeddings for the target dataset for the initial model (before adaptation) could serve to provide embeddings that are closer to the source domain. Then using these embeddings in a contrastive learning setting, instance-wise and class-wise contrastive learning with these set of models is used to guide the model adaptation procedure. \n\nPros:\n1) The proposed approach is interesting\n 2) It is thoroughly  validated with state of the art results for this relatively new task of model adaptation for classification, detection and segmentation. \n3) Thorough analysis is provided for the method along with theoretical insights by relating it to EM algorithm\n\n\nCons:\n\nWhile the paper is well written, thoroughly evaluated and with good insights, it has a few issues. \n\nSeveral aspects of the paper are not very clear and require clarification\n1) How are the negative and positive sets identified? One possibility is that for an instance, all historical embeddings for that particular instance form the positive and all other samples would be negative.\n2)  If this is the case, negative samples would far outweigh positive samples. How is this imbalance handled.\n3) Around line 148 it is mentioned: \"For HisNCE, we assign each key a reliability score to encourage to memorize the well-learnt historical embeddings only.” How is this reliability score computed. Later on, for the categorical case, it is mentioned that the samples that are consistent in prediction across current and historical models are termed as well-learnt sample. Is this same criterion used for assigning a reliability score?\n4) Similar issue with positive and negative sets for the HCCD learning is also present.\n\nOn the whole, the main issues with the paper relate to clarity in writing. There are minor grammatical mistakes also that could be addressed. To conclude, based on the method, analysis and results I retain a positive view on the paper. \n\nPost rebuttal\n\nThe rebuttal addresses the concerns raised and provides the clarifications required. I continue to retain my original rating of accept.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Yes, the limitations and impact are discussed.", "review_text": "", "extra_scores": {"ethical_concerns": "The paper utilizes standard adaptation settings. Thus no new potential ethical concerns are expected.", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "4"}, "extra_text": {}}, {"review_id": "0UrdNTqvB1k", "reviewer": "Reviewer_UV3u", "summary": "This paper studies the unsupervised model adaptation problem, where only the trained model, but not the source data, is available to use for target domain adaptation. The authors propose historical contrastive learning (HCL) technique to help this source-free setting. HCL is composed of two components, HCID and HCCD. HCID learns from target instances by contrasting the features generated from the adapted model and historical models. HCCD explores self-training to pseudo label target instances to learn category discriminative representations. Experiments on three different vision applications are conducted. The paper is generally easy to follow, despite some unclear points. The major concern is the novelty.  ", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strong points\n1.\tModel adaptation without access to source data is an interesting and practical problem.\n2.\tThe experiments include three computer vision applications, segmentation, detection and classification. The performances on different datasets are consistently better than baselines, although some improvements are marginal (see week points for detail)\n\nWeak points\n1.\tLimited novelty is my major concern. I recognize the authors apply contrastive learning to this new problem, and I also recognize the proposed modifications, e.g. the reliability scaler in Eq 3 compared to the traditional InfoNCE. But these do not meet the NeruIPS bar, especially the proposed modifications are not fully justified by experiments. (see below)\n2.\tFor $r$, to support the claim that this soft version (compared to traditional $r_i^{t-m} =1$), the analytical experiment of $r=1$ is missing. Similarly, in Eq 4, a baseline by setting $h_{con} =1$ (no reweighting) is missing.\n3.\tMissing key related references [ref-1]. \n[ref-1]Vinod K Kurmi, Venkatesh K Subramanian, and Vinay P Namboodiri. Domain impression: A source data free domain adaptation method. WACV 2021\n[ref-2] Jogendra Nath Kundu, et al., Towards Inheritable Models for Open-Set Domain Adaptation, CVPR 2020\nPlease consider making a discussion/comparison.\n4.\tThe equation above line 164, $p^t$ is a scalar, l-1 norm on a scalar does not seem correct to me. One more right bracket.\n5.\tA completed ablation study on different applications is highly recommended. \n\nOther minors: \n6.     For the baselines, I suggest removing all the non-source-free ones, which are out of this paper’s focus. \n7.\tLine 121, the reference [20] seems wrong.\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "No known concern on limitations and potential negative societal impact.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "4"}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper tackles a challenging problem of unsupervised domain adaptation (UDA) without access to source data while assuming access to a pretrained source model. The proposed method takes inspiration from contrastive losses (such as InfoNCE) and tailors these for the UDA setting. Reviewers appreciated the significance of the problem tackled by the paper (source-free UDA) and the experiments on a variety of vision tasks that show clear improvement over existing methods. Authors should revise the paper taking into account the reviewers' comments, including a discussion of the prior work pointed out in the reviews on the source-free UDA and universal DA problems. 
Decision: Accept (Poster)