PROTOTYPE-ASSISTED ADVERSARIAL LEARNING FOR UNSUPERVISED DOMAIN ADAPTATION
Authors: Dapeng Hu
Jian Liang*
Qibin Hou
Hanshu Yan
Jiashi Feng
Venue:  (2020)
Keywords: Domain Adaptation
Transfer Learning
Adversarial Learning

Abstract: This paper presents a generic framework to tackle the crucial class mismatch problem in unsupervised domain adaptation (UDA) for multi-class distributions.  Previous adversarial learning methods condition domain alignment only on pseudo labels, but noisy and inaccurate pseudo labels may perturb the multi-class distribution embedded in probabilistic predictions, hence bringing insufficient alleviation to the latent mismatch problem.  Compared with pseudo labels, class prototypes are more accurate and reliable since they summarize over all the instances and are  able  to  represent  the  inherent  semantic  distribution  shared  across  domains. Therefore, we propose a novel Prototype-Assisted Adversarial Learning (PAAL) scheme, which incorporates instance probabilistic predictions and class prototypes together  to  provide  reliable  indicators  for  adversarial  domain  alignment.   With the PAAL scheme,  we align both the instance feature representations and class prototype  representations  to  alleviate  the  mismatch  among  semantically  different classes.   Also,  we exploit the class prototypes as proxy to minimize the within-class variance in the target domain to mitigate the mismatch among semantically similar classes.  With these novelties, we constitute a Prototype-Assisted Conditional Domain Adaptation (PACDA) framework which well tackles the class mismatch problem. We demonstrate the good performance and generalization ability of the PAAL scheme and also PACDA framework on two UDA tasks, i.e., object recognition (Office-Home,ImageCLEF-DA, andOffice) and synthetic-to-real semantic segmentation (GTA5→CityscapesandSynthia→Cityscapes).

Under review as a conference paper at ICLR 2020
PROTOTYPE-ASSISTER ADVERSARIAL LEARNING FOR UNSUPERVISED DOMAIN ADAPTATION
Anonymous authors Paper under double-blind review
ABSTRACT
This paper presents generic framework remedy the misalignment in unsuper- vised domain adaptation (UDA). Previous adversarial learning methods for UDA condition domain alignment only on pseudo labels, but pseudo labels may be inac- curate, hence bringing insufficient alleviation to the misalignment. Compared with pseudo labels, class prototypes are more accurate and reliable since they are sum- marized over all the instances and are able to represent the inherent semantic struc- tures shared across domains. Therefore, we propose novel Prototype-Asistted Adversarial Learning (PAAL) scheme, which incorporates instance probabilistic predictions and class prototypes together to provide reliable indicators for adver- sarial domain adaptation. With the PAAL scheme, align both the instance feature representations and class prototypes to alleviate the misalignment among semantically different instances. Also, exploit the class prototypes as proxy to minimize the intra-class variance in the target domain to mitigate the misalign- ment among semantically similar instances. With these novelties, constitute a Prototype-Assisted Conditional Domain Adaptation (PACDA) framework which achieves state-of of-he-tf results on two UDA tasks, i.c., object recognition (Office- Home, ImageCLEF-DD -A, and Office) and synthetic-to-real semantic segmentation (GTAS--Ciyycene and
INTRODUCTION
Unsupervised domain adaptation (UDA) aims to leverage the knowledge of labeled data set (source domain) to help train a predictive model for unlabeled data set (target domain). Deep UDA meth- ods bring noticeable performance gain many tasks (Long et al., 2015; Saito et al., 2017: Richter et al., 2016; Tsai et al., 2018; Lee et al., 2019; Vu et al., 201 90) by exploiting supervision from heterogeneous sources. Some methods exploit maximum mean discrepancy (MMD) (Gretton et al., 2008; Long al., 2015) or other distribution statistics like central moments (Sun & Saenko, 2016; Zellinger et al., 2017; Koniusz et al., 2017) for domain adaptation. Recently, generative adversarial learning (Goodfellow et al., 2014) provides promising alternative solution to UDA problem.
Since the labels of the target instances are not given in UDA, adversarial learning scheme for adap- tation (Ganin & Lempitsky, 2015) suffers from the cross-domain misalignment, where the target instances from a class A are potentially misaligned with source instances from another class B. Inspired by the pseudo-labeling strategy from semi-supervised learning, previous methods either used the pseudo labels in the target domain to perform joint distribution discrepancy minimization (Long al., 2013; 2015) or developed conditional adversarial learning methods that involve one high-dimensional domain discriminator (Long et al., 2018) or multiple discriminators (Chen et al., 2017b; Pei al., 2018). Though effective, these conditional domain adversarial learning methods align different instances from different domains relying only on their own predictions. Simple prob- abilistic predictions or pseudo labels may not accurately represent the semantic information of input instances, misleading the alignment. A toy example is given in Fig. 1(a). The pseudo label of the chosen instance æ is inclined to be class 'square" while the ground truth label is class 'circle'. Only guided by the instance prediction, the 'circle" class in the target domain and the 'square' class in the source domain are easily confused, causing the misalignment in the adversarial domain adaptation.
To remedy the misalignment, alignment, instead of using only the possibly inaccurate predictions. Prototypes are global feature representations of different classes and are relevant to the inherent semantic structures shared across
propose to exploit the class prototypes for adversarial domain

---

Under review as a conference paper at ICLR 2020
p= 1.3 8.4
p=|0.1 0.2 0.7]
prototypes
p-
p- 0.5 0.1
(a) conditional adversarial leamning
(b) prototype- assisted adversarial learning
Figure I: Illustration of two adversarial learning schemes. Different from class-agnostic adversarial learning that pursues the marginal distribution alignment but ignores the semantic consistency, (a) conditional adversarial learning relies heavily on the instance-level pseudo labels to perform con- ditional distribution alignment, while (b) our prototype-assisted adversarial learning integrates the instance-level pseudo labels and global class prototypes to make the conditional indicators more reliable. Class information is denoted in different shapes with source in solid and target in hollow.
domains. As shown in Fig. 1(b), class prototypes are expected remedy the negative effects of inaccurate probabilistic predictions. Motivated by this, we propose a Prototype-Assisted Adversarial Learning (PAAL) scheme which complements instance predictions with class prototypes to obtain more reliable conditional information for guiding the source-target feature representation alignment.
Specifically, we summarize the class prototypes from all instances according to their predictions. In this way, one hand, we lower the dependence of class prototypes on instance predictions which may be inaccurate, and on the other hand, we encourage the instances with greater certainty to contribute more to their corresponding class prototypes. The prototypes are updated dynamically through a moving average strategy to make them more accurate and reliable. Then by broadcast- ing class prototypes to each instance according to its probability prediction, the inaccurate semantic distribution depicted by instance predictions can be alleviated. Based on reliable prototype-base conditional information, through the proposed PAAL scheme to relieve the alignment among semantically dissimilar in- stances. However, such conditional domain alignment may promote the confusion among seman- tically similar instances across domains to some degree. To further alleviate it, we introduce an intra-class objective in the target domain pursue the class compactness. Built on the proposed PAAL scheme and this intra-class compactness objective, we develop PrototypeAssister Condi- tional Domain Adaptation (PACDA) framework for solving UDA problems. Extensive experimental evaluations on both object recognition and semantic segmentation tasks clearly demonstrate the ad- vantages of our approaches over previous otate-of-the-arts (Long et al., 2018; Xu et al., 2019; Luo et al., 2019; Tsai et al., 2019).
align both the instance feature representations and the class prototypes
The contributions of this work can be summarized into three folds: 1) To the best our knowledge, we are the first to leverage the class prototypes in conditional adversarial learning to prevent the misalignment in UDA; 2) We propose simple yet effective domain adversarial learning framework PACDA to remedy the misalignment among semantically similar instances well as semantically dissimilar instances; 3) The proposed PAAL scheme and PACDA framework are generic, and our framework achieves the state-of-the-ee results several unsupervised domain adaptation tasks including object recognition and semantic segmentation.
RELATED WORK
Unsupervised Domain Adaptation. UDA is first modeled as the covariate shift problem (Shi- modaira, 2000) where marginal distributions of different domains are different but their conditional distributions are the same. To address it. (Dudik et al., 2006; Huang et al., 2007) exploit non- parametric instance re-weighting scheme. Another prevailing paradigm (Pan et al., 2010; Long et al., 2013; Herath et al., 2017) aims to learn feature transformation with some popular cross-domain met- rics, e.g., the empirical maximum mean discrepancy (MMD) statistics. Recently, a large number of deep UDA works (Long et al., 2015; Haeusser et al., 2017; Saito et al., 2018; Tsai et al., 2018) have been developed and boosted the performance of various vision tasks. Generally, they can be divided into discrepancy-baat basedd and betesarial-basee methods. Discrepancy-basee methods (Tzeng al., 2014: Long et al., 2017) address the dataset shift by mitigating specific discrepancies defined on different layers of shared model between domains, e.g. resembling shallow feature transforma-
2

---

Under review as a conference paper at ICLR 2020
tion by matching higher moment statistics of features from different domains (Zellinger al., 2017; Koniusz et al., 2017). Recently, adversarial learning has become a dominantly popular solution to domain adaptation problems. It leverages extra domain discriminator to promote domain confu- sion. (Ganin & Lempitsky, 2015) designs a gradient reversal layer inside the classification network and (Tzeng et al., 2017) utilizes inverted label GAN loss to fool the discriminator.
Pseudo-labeling. UDA can be regarded as a semi-supervvied learning (SSL) task where unlabeled data are replaced by the target instances. Therefore, some popular SSL strategies, e.g., entropy minimization (Grandvalet & Bengio, 2005; Vu et al., 2019b), mean-teacher (Tarvainen & Valpola, 2017: French et al.. 2018), and virtual adversarial training (Miyato et al., 2018; Shu et al., 2018), have been successfully applied to UDA. Pseudo-labeling! is favored by most UDA methods due to its convenience. For example, (Saito et al., 2017: Li et al., 2019) exploit the intermediate pseudo-labels with tri-training and self-training, respectively. (Pan et al., 2019) obtains target-specific prototypes with the help of pseudo labels and aligns prototypes across domains at different levels. Recently, curriculum learning (Choi et al., 2019), self-paced learning (Zou et al., 2018) and re-weighting schemes (Long et al., 2018) are further leveraged to tackle possible false pseudo-labels.
Conditional Domain Adaptation. Apart from the explicit integration with the last classifier layer, pseudo-labels can also be incorporated into adversarial learning to enhance the feature-level domain alignment. Concerning shallow methods (Long et al., 2013; Zhang et al., 2017), pscudo-labels can help mitigate the joint distribution discrepancy via minimizing multiple class-wise MMD measures. (Long et al., 2017) proposes align the joint distributions of multiple domain-specifie layers across domains based on a joint maximum mean discrepancy criterion. Recently, (Chen et al., 2017b; Pei al., 2018) leverages the probabilities with multiple domain discriminators enable fine-grained alignment of different data distributions in an end-to-end manner. In contrast, (Long et al.. 2018) conditions the adversarial domain adaptation on discriminative information via the outer product of feature representation and classifier prediction. Motivated by the cemantically-consisse (Cicek& & Soatto, 2019) imposes multi-way adversarial loss instead of a binary one on the domain alignment. However, these methods all highly rely on the localized pseudo-labels to align label- conditional feature distributions and ignore the global class-level semantics. As far as we know, we are the first to exploit class prototypes to guide the domain adversarial learning. Compared with (Pei et al., 2018; Long et al., 2018), our PACDA framework complements the original feature representations with reliable semantic features and merely involves two lowddimensional domain discriminators, making the domain alignment process simple, conditional, and reliable.
GAN,
METHOD
In this section, we first begin with the basic settings of UDA and then give detailed descriptions the proposed PAAL scheme and the PACDA framework. Though proposed for image classification, they can also be easily applied to semantic segmentation.
3.1 PROBLEM SETTINGS
In a vanilla UDA task, we are given label-rich source domain data {(x},, y:) sampled from the joint distribution (x ya) and unlabeled target domain data {=ih sampled from the joint distribution (æ, By). where x E X's and y E Yss denote an image and its corresponding label from the source domain dataset, æt E Xr denotes an image from the target domain dataset and P. * Qt. The goal of UDA is to learn a discriminative model from X's Vss and Xr to predict labels for unlabeled target samples Xr.
As described in (Ganin et al.. 2016), vanilla domain adversarial learning framework consists of a feature extractor network G, a classifier network F, and a discriminator network D. Given an image æ, we denote the feature representation vector extracted by G as f = G(x) E I2d and the probability prediction obtained by F as p = F(f)= E IR where d means the feature dimension and mcans the number of classes. The vanilla domain adversarial learning method in (Ganin et al., 2016) can be formulated as optimizing the following minimax optimization problem:
min max (G, F) >adto Lade (G, , D),
G,F D
Ladv (G, D) -B log log ((F)) E Ex/~Q. log(1 D(Fi),
(2)
3

---

Under review as a conference paper at ICLR 2020
Peature Extrattor (G)
fu
Clessifer (F)
Ps M P
Shard
Festurn V octtor
M emmaa
f.
Prediction Voctor
Sourse
Feekure Discriminator
Protolype D Dactminator (D,)
C cdly
(Dy)
Protorype V Vecior
M Moma
Ccurdl
ft
Mf ps
Loss Function
Extractor (G)
Feature
Classifeer (P)
Indicanor
Target
Figure 2: Overview of the proposed PACDA framework which consists of a shared feature extractor G. shared classifier F, and two domain discriminators (D), Dp). Mema represents the global class prototype matrix while Mg, e is computed by source or target instances within current batch.
C, F) = - -B
NP) wi)
log(p"), p. F(G(()))
(3)
where the binary domain classifier D R [0, 1] predicts the domain assignment probability the input features, (G F) is the cross-entropy loss of source domain data as for the classification task, and Aadv is the trade-off parameter.
3.2 PROTOTYPE-ASSISTE ADVERSARIAL LEARNING (PAAL) SCHEME
The misalignment UDA multi-class distributions challenges the popular vanilla adversarial learning. In previous works (Long et al., 2017; Pei et al., 2018; Long et al., 2018), target domain data are conditioned only on corresponding pseudo labels predicted by the model for adversarial domain alignment. The general optimization process of these methods is the same aforementioned vanilla domain adversarial learning, except that feature representations jointly with predictions are considered by the discriminator D:
(4) the conditional adversarial loss that leverages the classification predictions P. and pt. A classic previous work (Long et al., 2018) implicitly conditions the feature representation on the prediction through the outer product f @ p. and uses one shared discriminator align the conditioned feature representations. (Long et al., 2018) further proves that using the outer product can perform much better than simple concatenation f p. Different from (Long al., 8), (Chen et al., 2017b; Pei et al., 2018) explicitly utilize multiple class-wise domain discriminators to align the feature representations relying on the corresponding predictions.
goond (G, D) -Ea log[D(fi.pp) -E _. log(1 D(F p!)..
However, the pseudo labels may be inaccurate due to the domain shift. Therefore, only conditioning the alignment on pseudo labels can not safely remedy the misalignment. Compared with the pseudo labels, the class prototypes are more robust and reliable in terms of representing the shared semantic structures (Yang et al., 2018). To acquire more reliable and accurate conditional information for domain adversarial learning, we propose to complement instance predictions with class prototypes and reformulate the adversarial loss to:
Cpaal (G, D) EP log M T p')] Eal Exl-q. log[1 D(F M. p(). (5)
Here Meme Rexd denotes the global class prototype matrix in our prototype-assin assisted adversar- ial learning loss Lreat (G, D). In reality, the reliable conditional information is obtained through broadcasting the global class prototypes to each independent instance according to its prediction p.
We propose to summarize feature representations the instances within the same class as the corre- sponding prototype. Then the probability prediction is leveraged to obtain accurate class prototypes. Using predictions as weights can adaptively control the contributions of typical and non-typical in- first gather
stances to the class prototype, making class prototypes the feature representation each instance relying on its prediction to generate the batch-level class prototypes. Then the global class prototypes can be obtained by virtue of an averaging strategy such
reliable. Specifically,

---

Under review as a conference paper at ICLR 2020
exponential moving average (ema) on the batch ones. This process can be formulated as "AA,A, phd.
(6) Mema Aema-Mema + (1 Xema) M, where [m] mee], my = Σ, p". /E;
Here n means the batch size, pr represents the probability of the i-th instance belonging to the k-th semantic class, Xema is empirical weight, M E [2 is the batch-level class prototype matrix and Mema is the global one computed by certain source domain data and contributes to more reliable conditional information exploited by discriminators. Similarly, batch-level class prototypes are broadcast each instance in this batch through M Pa which can be denoted fa E {s.t).
3.3 PROTOTYPE-ASSIITER CONDITIONAL DOMAIN ADAPTATION (PACDA) FRAMEWORK
With our prototype-basee conditional information, we further propose Prototype-Assisted Con- ditional Domain Adaptation (PACDA) framework. This framework aligns both instance-level and prototype-level feature representations through PAAL and promotes the intra-class compactness in target domain such that the misalignment can be substantially alleviated even though no supervision available in the target domain. Its overall architecture is shown in Fig. 2.
Besides the backbone feature extractor G and the task classifier F, there are two discriminators in our framework PACDA, i.e., the instance-level feature discriminator D and the prototype-leve feature discriminator Dp- We can formulate our general objective function as (w.l.o.g., Lado + -Fade) )
C(X's D's, Xr)
Lg X C'dt Nado Ca + Xehe,
where À denotes balance factors among differrnnt loss functions, Cy is the supervised classification loss on source domain data described by Eq. (3), C'dy is the adversarial loss to align instance feature representations across domains, Cudv is the adversarial loss to align class prototype representations across domains, and L, the loss to promote the intra-class compactness in target domain.
Instance- Alignment Conditioning the instance feature representation on our prototype-base based conditional information, we seek to align feature representations across domains at the instance-level through discriminator Dy. With the assistance of the accurate semantic structures embedded in class prototypes, misalignment among semantically dissimilar instances can be effectively alleviated. We can define the instance-level adversarial loss Lado' as
C.d FG Dy) = -En;~r, (f M - B.i D + &MaaPil (8)
Prototype-Level Alignment Instance-level alignment only implicitly aligns the multi-class distribu- tion across domains, which may not ensure the semantic consistency between two domains. Besides, since in practice global class prototypes are collected from only source domain data, which possibly cannot accurately represent inherent semantic structures in the target domain due to the domain shift. Taking into account these two causes, we perform the prototype-level alignment with discriminator Dp to explicitly align the class prototype representations across domains. The specific loss function cf adv is defined as
(G, -Bgj-P, log(D, M am p3)] - log[1-10 DDD(P e e Em p()]. (9)
Intra-Class Compactness Although adversarial alignment based on PAAL can relieve the mis- alignment among obviously semantically different instances, it cannot well handle the misalignment among semantically similar instances. Specifically, incorporating class prototypes into instance pre- dictions would confuse semantically similar instances during domain alignment and result in the misalignment among them. To solve this problem, our framework further promotes the intra-clas compactness in the target domain enlarge the margin between instances of semantically similar classes. Taking the prototypes as proxy, we minimize the following loss for target domain samples to encourage the intra-class compactness:
(G, F) EB Eq_ IIF!
fillz
(10)
Thus, the complete minimax optimization problem of our PACDA framework can be formulated as
min max (G X Cl (G, Nato (G, (G,
(11)
G.F Dy.D,
With only two diwddieeniional (2 x d) discriminators added, effectively remedy the misalign- ment in domain adversarial learning. Some theoretical insights with the help of domain adaptation theory (Ben-David al., 2010) is discussed in the Appendix.
5

---

Under review as a conference paper at ICLR 2020
4 EXPERIMENTS
4.1 EXPERIMENTAL SETUP
We conduct experiments to verify the effectiveness and generalization ability of our methods, i.e., PACDA (full) in Eq. (11) and PAAL (Nadn = X = 0) on two different UDA tasks, including cross-domain object recognition on ImageCLEF-DA', Office31 (Saenko et al., 2010) and Office- Home (Venkateswara et al., 2017), and synthetic-to-real semantic segmentation for GTA5 (Richter (Cordts et al., 2016) and Synthia (Ros et al., 2016) Cityseapes.
et al., (016))+Ciryscooo
Datasets. Office-Homet is new challenging dataset that consists of 65 different object categories found typically in 4 different Office and Home settings, i... Artistic (Ar) images, Clip Art (Ca), Product images (Pr), and Real-World (Re) images. ImageCLEFF-DA is standard dataset built for the
competition. We follow (Long et al., 2015) to select subsets, i.e., C, I, and P. which share 12 common classes. Office31 is popular dataset that includes 31 object categories taken from 3 domains, i.e., Amazon (A), DSLR (D), and Webcam (W).
Citscapes is realistic dataset of pixel-level annotated urban street scenes. We use its original training split and validation split as the training target data and testing target data respectively. GTAS consists of 24,966 densely labeled synthetic road scenes annotated with the same 19 classes as Cityscapes. For Synthia, we take the WWNTHIA-RADD-CIYSCAPPA which is composed of 9,400 synthetic images compatible with annotated classes of Cityscapes.
set as the source domain,
Implementation Details. For object recognition, we follow the standard protocol (Ganin & Lem- pitsky, 2015), i.e. using all the labeled source instances and all the unlabeled target instances for UDA, and report the average accuracy based on three random trials for fair comparisons. Following (Long et al., 2018; Xu et al., 2019), we experiment with ResNet-50 model pretrained on ImageNet. Specifically, we follow (Long et al., 2018) to choose the network parameters, and all convolutional layers and the classifier layer are trained through backpropagation, where ===5e-3, A5ma=55-11 X and Xgde increase from 0 to 1 with the same strategy as (Ganin & Lempitsky, 2015). Regarding the domain discriminator, we design a simple two-layer classifier 225-+1024--27 for both Dy and Dp- Empirically, we fix the batch size to 36 with the initial learning rate being 1c-4.
For semantic segmentation, we adopt DeepLab-Va V2 (Chen et al., 2017a) based on ResNet-101 (He et al., 2016) as done in (Tsai et al., 2018; Vu et al., 2019b; Luo et al., 2019; Tsai et al., 2019). Following DCGAN (Radford et al., 2015), the discriminator network consists of three lutional layers with stride 2 and channel numbers {256, 512, 1}. In training, we use SGD (Bottou, 2010) to optimize the network with momentum (0.9), weight decay (Se-4), and initial learning rate (2.5e-4). We use the same learning rate policy as in (Chen et al., 2017 Discriminators are opti- mized by Adam (Kingma Ba, 2015) with momentum (3, = 0.9, 32 = 0.99), initial learning rate (1e-4) along with the same decreasing strategy as above. For both tasks, N is set to 1e-3 follow- ing (Tsai et al., 2018) 18) and Xema is set to 0.7. For GTAS G7A5--Cinyscapes, X2 Nad =le-3 and x For Synthia->Ciyyccaps, =le-4 and >.=le-4.
4 convo-
All experiments are implemented via PyTorrh on single Titan X GPU. The total iteration number is set as 10k for object recognition and 100k for semantic segmentation. For objection recogni- tion tasks, we choose the (Morerio et al., 2018) on Ar-Cl for convenience. For semantic segmentation tasks, training split of Cityscapes used for the hyper-parament selection. Data augmentation skills like random scale or random flip and ten-crop ensemble evaluation are not adopted.
which have the minimal mean entropy of target data
4.2 COMPARISON RESULTS
Cross-Domain Object Recognition. The comparison results between our methods (i.e., PAAL and PACDA) and state-of-thh-an (SOTA) approaches (Xu et al., 2019; Long et al., 2018; Zhang et al., 2018) on Office-Home, Office31, and ImageCLEF-DA are shown in Tables 1 and 2. respectively. As indicated in these tables, PACDA improves previous approaches in the average accuracy for all three benchmarks (c.g.. 67.3%-468. for Sifice-Home, 88.1% 18.1%%+88 +88.8 for ImageCLEF-DA. and for Office3l). Generally, PACDA performs the best for most transfer tasks. Taking
"https://wwe wwwp://imaaelfffooommm /ttpa///eeooo/////aaaaaaaaaioooo

---

Under review as a conference paper at ICLR 2020
Table 1: Accuracy (%) on Office- for UDA under ResNet-50, Red: Best, Blue: Second best.
Methods
Ar-:Cl Ar-Pt Ar-Re Cl+AA Cl-Pt Cl-RRe
Pr-+Ar
Pr-CI
Pr-Re
Re-AAt
Re-+CI
Re-Pr Avg.
RosNet-50 (Hc et 2016)
34,9
50.0
58.0
37.4
41.9
46.2
38.5
31.2
60.4
53.9
41.2
59.9
46.1
DANN (Ganio & Lompitsky, 2015)
45.6
59.3
70.1
47.0
58.5
60.9
46.1
43.7
68.5
63.2
76.8
57.6
CDAN (Long et al., 2018)
49.0
69.3
74.5
54.4
66.0
68.4
55.6
48.3
75.9
68.4
55.4
80.5
63.8
CDAN+E (Long al. 2018)
50.7
70.6
76.0
57.6
70.0
70.0
574
50.9
77.3
70.9
56.7
81,6
65.8
DWT-MECC (Ray et al., 2019)
50.3
72.1
77.0
59.6
70.2
58.3
48.1
77.3
69.3
53.6
82.0
65.6
SAFN (Xu et al., 2019)
52.0
73.3
77.9
65.2
71.5
73.2
63.6
52.6
78.2
72.3
57.1
81.5
67.3
PAAL
50.7
69.2
73.2
58.2
66.4
68.3
55.5
48.3
68.5
55.6
79.2
64.0
+0.2
+0.5
+0022
+0.5
+0.6
+0.5
+0.3
+0.4
=0.1
+0.2
=0.3
+0.2
PACDA
53,5
73.5
78.4
64.1
73.2
74.4
64.1
50.8
79.9
73.6
56.6
82.6
68.7
=0.2
#0.1
=0.2
+04
20.3
=02
=0.5
+0.9
=0.1
10.1
=0.1
#0.2
Table 2: Accuracy (%) on Image CLEFFD and Office31 for UDA under ResNet-50.
Datasets
ImageCLEF-RR
Avg.
Offices1
Avg.
Methods
C-11
C-P
I-C
I-P
P-C
P-1
A-D A W
D-
W
W-+
W-+
ResNet-50 50000000000 (He et al., 2016)
78.0
65.5
91.5
74.8
91.2
81.9
80.7
68.9
68.4
62.5
96.7
60.7
99.3
76.
DANN (Ganin Lempitsky, 2015)
87.0
74.3
96.2
75.0
91.5
86.0
85.0
79.7
82.0
96.9
67.4
99.1
82.2
CDAN (Long et al., 2018)
90.5
74.5
97.0
76.7
93.5
90.6
87.1
89.8
93.1
70.1
98.2
68.0
100.
86.6
CDAN+E ono et al., 2018)
91.3
74.2
97.7
77.7
94.3
90.7
87.7
92.9
94,1
71.0
98.6
69.3
100.
87.7
ICAN (Zhang al. 2018)
89.9
78.5
94.7
79.5
92.0
89.7
87.4
90.1
92.5
72.1
98.8
69.9
100.
87.2
CAT (Deng et al., 2019)
91.3
75.3
95.5
77.2
93.6
91.0
87.3
90.8
94.4
72.2
98.0
70,2
100.
87.6
SAFN (Xu al., 2019)
91.1
77.0
96.2
78.0
94.7
91.7
88.1
87.7
69.8
69.7
85.7
PAAL
91.4
74.6
97.4
77.6
95.2
90.7
87.8
93.4
94.6
70.2
97.7
69.9
100.
87.6
0.1
10.66
10.1
10.4
10.1
0.4
1.0.0
..0.7
1.088
1.0.1
10.6
1.0.0
PACDA
92.2
76.2
97.7
78.1
96.3
92.4
88.8
94.8
95.7
72.8
74.4
89.3
0.2
+0.2
40.1
+-0.2
+0.2
+0.2
+0.00
+.0.3
+ 1.5
+0.2
+0.5
+-0.0
Table 3: Comparison results of synthetic-to-reat semantic segmentation using the same architecture with NonAdapt and AdaptSeg (Tsai et al., 2018), AdvEnt (Vu et. 2019b). CLAN (Luo al., 2019)
and AdaptPatch (Tsai et al., 2019). Top: GTA5 + Cityscapes. Bottom: Synthia ~ Cityscapes.
road
sdwk
bldng
wall
Fence
pole
light
sign
v4g
10r.
sky
per.
rider
car
truck
bus
train
mbike
bike
mioU
Methods
NonAdapt
75.8
16.8
77.2
12.5
21.0
25.5
30.1
20.1
81.3
24.6
70.3
53.8
26.4
49.9
17.2
25.9
6.5
25.3
36.0
36.6
snurce: GTAS
AdaptSeg
86.5
25.9
79.8
22.1
20.0
23.6
33.1
21.8
81.8
25.9
75.9
57.3
26.2
76.3
29.8
32
7.2
29.5
32.5
41.4
AdvEnt
89.9
36.5
81.6
29.2
25.2
28.5
32.5
22,4
83,9
34.0
77.1
57.4
27.9
83.7
29
39.1
1.5
28.4
23.3
43.8
CLAN
87.0
27.1
27.3
23.3
28.3
35.5
24.2
83.6
27.4
74.2
58.6
28.0
76.2
33.1
36.7
6.7
31,9
31.4
43.2
AdaptPatch
89.2
38.4 80.4
24.4
21.0
27.7
32.9
16.1
83.1
34.1
77.8
57.4
27.6
78.6
31.2
40.2
4.7
27.6
27.6
43.2
PAAL
89.7 30.3 81.7
30.7
24.8
29.4
36.3
22.6
83.4
36.1
76.3
60.4
28.4
83.8
32.7
41.7
4.4
27.3
31.0
44.8
PACDA
92.4 50,3 83.3 33,2 24.6
32,9
36.1
22.7
84.4
38.0
79.9
60.3
24.7
85.2
37.8
46.6
7.2
26.4
18.6
46.6
source: Synthia
NonAdapt
55.6 23.8 74.6
6.1
12.1 74.8
79.0 55.3
19.1 39.6
23.3
13.7 25.0
38.6
AdaptSeg
79.2 37.2 78.8
9.9
10.5
78.2
80.5 53.5 19.6 67.0
29.5
21.6
31.3
45.9
AdvEnt
87.0 44.1 79.7
7.2 80.1
83.6 56.4
23.7 72.7
32.6
12.8
33.7
47.6
CLAN
81.3
80.1
16.1
13.7 78.2
81.5 53.4 21.2 73.0
32.9
22.6
30.7
47.8
AdaptPatch
82.2 39.4 79.4
6.5
10.8
77.8
82.0 54.9
21.1 67.7
30.7
17.8
32.2
46.3
PAAL
87.2 43.2 80.0
8.2 9.6
79.2
82.3 56. 20.3 81.1
33.5
16.5
32.3 48.4
PACDA
87.0 42.4 80.8
9.7
10.9 80.0
83.6 50.1 20.7 82.1
34.4
19.4 38.6 49.2
careful look PAAL, we find that it always beats CDAN and achieves competitive performance with SOTA methods like CAT (Deng et al., 2019).
Synthetic-to-real Semantic Segmentation. We compare PAAL and PACDA with SOTA meth- ods (Tsai et al., 2018; Vu et al., 2019h; Luo et al., 2019; Tsai et al., 2019) on synthetic-to-real semantic segmentation. Following (Chen al., 2017b), we evaluate models on all 19 classes for GTA5-Cittscape while on only 13 classes for Sanuhia--Cütyccappe As shown in Table 3, without bells and whistles, our PAAL method outperforms all of those methods and our PACDA framework further achieves new SOTA results on both tasks, i.e., 43.8%->46.6 for GTA5 CTAS->Ciryscppss and 47.8% 47.8%~449.2% +49.2% for Synthia-+Ciyssaapp
terms of the mean IoU (mloU) value.
4.3 FURTHER ANALYSIS
Quantitative Analysis. To verify the effectiveness of each component in Eq. (11), we introduce a variant named PA that merely ignores the intra-class objective ()2 0). The empirical con- vergence curves about Ar-CC in Fig. (3)(a) imply that all of our variants tend to converge after 10k iterations, and the second term can help accelerate the convergence. Fig. 3(b) shows that all terms
7

---

Under review as a conference paper at ICLR 2020
Lartice repadas
Ner femder 500 (a) Convergence
Natwork
(b) Ablation study
(c) Proxy A-distance
(d) Network architecture
Figure Quantitative analysis of our methods.
(a) Source only
(b) DANN
(c) CDAN
(d) PAAL
(e) PACDA
Figure 4: t-SNE (Maaten Hinton, 2008) embedding visualization of UDA methods for C-+I on ImageCLEF-Dá (class information is denoted by different colors with source in A with target in *).
in the PACDA framework, i.e., PAAL alignment different levels and the intra-class objective, can bring evident improvement on both tasks. As shown in Fig. (3)(c), we provide the proxy A-distances (Ganin al., 2016) of different methods for Ar-SCI and C-+I. The A-distance Disf 2c) is popular measure for domain discrepancy, where is the test error of binary classifier trained on learned features. All the UDA methods have smaller distances than 'source only' by aligning different domains. Besides, our PACDA has the minimum distance for both tasks, implying that it can learn better features to bridge the domain gap between domains. To testify the sensitivity of report the accuracies of DANN, CDAN and PACDA for C-+I on the ImageCLEF- DA with 3 different backbone architectures, i.e., VGG-16, ResNet-18, 18, and ResNet-50. Obviously, PACDA is the best- pest-performing method that shows desirable robustness when the network changing from VGG-16 to ResNet-18.
PACDA. in Fig. (d)
Qualitative Analysis. For object recognition, we study the t-SNE visualizations of aligned features generated by different UDA methods in Fig. 4. As expected, conditional methods including CDAN and PAAL can semantically align multi-class distributions much better than DANN. Besides, PAAL learns slightly better features than CDAN due to less misalignment. Once considering the intra-class objective, PACDA further enhances PAAL by pushing away semantically confusing classes, which achieves the best adaptation performance. For semantic segmentation, we present some qualitative results in Fig. 5. Similarly, PAAL effectively improves the adaptation performance and PAAL as well as PACDA can further improve the segmentation results.
(a) Target image (b) Ground truth
(f) PACDA Figure 5: Qualitative results of synthetic-to-real semantic segmentation for GTA5 Cilyscpess
(c) Source only
(d) PAAL
(c) PAAL
CONCLUSION
In this work, we developed the prototype-assisted adversarial learning scheme to remedy the mis- alignment for UDA tasks. Unlike previous conditional ones whose performance is vulnerable to inaccurate instance predictions, our proposed scheme leverages the reliable and accurate class pro- totypes for aligning multi-class distributions across domains and is demonstrated to be more effec- tive to prevent the misalignment. Then we further augment this scheme by imposing the intra-class compactness with the prototypes proxy. Extensive evaluations both object recognition and semantic segmentation tasks clearly justify the effectiveness and superiority of our UDA methods well-established baselines.
8

---

Under review as a conference paper at ICLR 2020
REFERENCES
Shai Ben-David, John Blitzer, Koby Crammer, Alex Kulesza, Fernando Pereira, and Jennifer Wort- man Vaughan. A theory of learning from different domains. Machine Learning, 79(1-2):151-175 2010.
Léon Bottou. Large-scale machine learning with stochastic gradient descent. In International Con-
ference on Computational Statistics. Springer, 2010.
Liang-Chies Chen, George Papandreou, Iasonas Kokkinos, Kevin Murphy, and Alan Yuille. Deeplab: Semantic image segmentation with deep convolutional nets, atrous convolution, and fully connected crfs. IEEE Transactions on Pattern Analysis and Machine Intelligence, 40(4): 834-848, 2017a.
Yi-Hsin Chen, Wei- Yu Chen, Yu-Ting Chen, Bo-Cheng Tsai, Yu-Chiang Frank Wang, and Min Sun. No more discrimination: Cross city adaptation of road scene segmenters. In IEEE International Conference on Computer Vision (ICCV), 2017b.
Jaehoon Choi, Minki Jeong, Taekyung Kim, and Changick Kim. Pseudo-labeling curriculum for unsupervised domain adaptation. British Machine Vision Conference (BMVC), 2019.
Safa Cicek and Stefano Soatto. Unsupervised domain adaptation via regularized conditional align- ment. In IEFE Conference Computer Vision (ICCV), 2019.
Marius Cordts, Mohamed Omran, Sebastian Ramos, Timo Rehfeld, Markus Enzweiler, Rodrigo Benenson, Uwe Franke, Stefan Roth, and Bernt Schiele. The cityscapes dataset for semantic urban scene understanding. In IEEE Conference Computer Vision and Pattern Recognition (CVPR), 2016.
Zhijie Deng, Yucen Luo, and Jun Zhu. Cluster alignment with a teacher for unsupervised domain adaptation. In IEEE Conference Computer Vision (ICCV), 2019.
Miroslav Dudík, Steven J Phillips, and Robert E Schapire. Correcting sample selection bias in maximum entropy density estimation. In Advances in Neural Information Processing Systems (NeurIPS), 2006.
Geoffrey French, Michal Mackiewicz, and Mark Fisher. Self-ensembling for visual domain adapta- tion. In International Conference on Learning Representations (ICLR), 2018
Yaroslav Ganin and Victor Lempitsky. Unsupervised domain adaptation by backpropagation. In International Conference Machine Learning (ICML). 2015.
Yaroslav Ganin, Evgeniya Ustinova, Hana Ajakan, Pascal Germain, Hugo Larochelle, François Laviolette, Mario Marchand, and Victor Lempitsky. Domain-adversarial training of neural net- works. Journal of Machine Learning Research, 17(1):2096 (71):2099-2030, 2016.
Ian Goodfellow, Jean Pouget-Abadie, Mchdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. Advances in Neural infor- mation processing systems (NeurIPS), 2014.
Yves Grandvalet and Yoshua Bengio. Semi-supervised learning by entropy minimization. In Ad- vances neural information processing systems, pp. 529-536, 2005.
Arthur Gretton, Karsten M Borgwardt, Malte Rasch, Bernhard Schölkopf, and Alexander Smola. A kernel method for the two-sample problem. Journal of Machine Learning Research, 1:1-10, 2008.
Philip Haeusser, Thomas Frerix, Alexander Mordvintsev, and Daniel Cremers. Associative domain adaptation. In IEEE International Conference Computer Vision (ICCV), 2017.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. JEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2016.
Samitha Herath, Mehrtash Harandi, and Fatih Porikli. Learning an invariant hilbert space for domain adaptation. In IEFE Conference Computer Vision and Pattern Recognition (CVPR), 2017.

---

Under review as a conference paper at ICLR 2020
Jiayuan Huang, Arthur Gretton, Karsten Borgwardt, Bernhard Schölkopf, and Alex J Smola. Cor- recting sample selection bias by unlabeled data. In Advances in Neural Information Processing Systems (NeurIPS), 2007.
Diederik Kingma and Jimmy Ba. Adam: method for stochastic optimization. In International Conference on Learning Representations (ICLR), 2015.
Piotr Koniusz, Yusuf Tas, and Fatih Porikli. Domain adaptation by mixture of alignments of second- or higher-order scatter tensors. In IEEE Conference Computer Vision and Patlern Recognition (CVPR), 2017.
Kuan-Hui Lee, German Ros, Jie Li, and Adrien Gaidon. SPIGAN: Privileged adversarial learning from simulation. In International Conference on Learning Representatiioss 2019. URL https: //openzeview......ooo.e/ooooo/./ tttes///oummaa http:t/eee.mmmmmmmmcoooocco
Yunsheng Li, Lu Yuan, and Nuno Vasconcelos. Bidirectional learning for domain adaptation of se- mantic segmentation. IEEE Conference on Computer Vision and Puttern Recognition (CVPR), 2019.
Mingsheng Long, Jianmin Wang, Guiguang Ding, Jiaguang Sun, and Philip S Yu. Transfer feature learning with joint distribution adaptation. In IEEE Conference on Computer Vision (ICCV), 2013.
Mingsheng Long, Yue Cao, Jianmin Wang, and Michael Jordan. Learning transferable features with deep adaptation networks. In International Conference on Machine Learning (ICML), 2015.
Mingsheng Long, Han Zhu, Jianmin Wang, and Michael Jordan. Deep transfer learning with joint adaptation networks. In International Conference on Machine Learning (ICML), 2017.
Mingsheng Long, Zhangjie Cao, Jianmin Wang, and Michael Jordan. Conditional adversarial domain adaptation. In Advances in Neural Information Processing Systems (NeurIPS), 2018.
Yawei Luo, Liang Zheng. Tao Guan, Junqing Yu, and Yi Yang. Taking a closer look domain shift: -ategory-level adversaries for semantics consistent domain adaptation. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2019.
Laurens van der Maaten and Geoffrey Hinton. Visualizing data using t-sne. Journal of Machine Learning Research, (Nov))2579-2265, 2008
Takeru Miyato, Shin-ichi Maeda, Masanori Koyama, and Shin Ishii Virtual adversarial training: a regularization method for supervised and semi-supervised learning. IEEE transactions on pattern analysis and machine intelligence, 1179):1979-199 1993, 2018.
Pietro Morerio, Jacopo Cavazza, and Vittorio Murino. Minimal-entropy correlation alignment for unsupervised deep domain adaptation. In International Conference on Learning Representations, 2018. URL https: / www.oopeevvieeeeee net
Sinno Jialin Pan, Ivor W Tsang, James T Kwok, and Qiang Yang. Domain adaptation via transfer component analysis. IEEE Transactions on Neural Networks, 22(():199-2100 2010.
Yingwei Pan, Ting Yao, Yehao Li, Yu Wang, Chong- Chong-Wah Ngo. and Tao Mei. Transferrable proto- typical networks for unsupervised domain adaptation. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 2239-2247, 2019.
Zhongyi Pei, Zhangjie Cao, Mingsheng Long, and Jianmin Wang. Multi adversarial domain adap- tation. In AAAI Conference on Artificial Intelligence (AAAI), 2018.
Alec Radford, Luke Metz, and Soumith Chintala. Unsupervised representation learning with deep convolutional generative adversarial networks. arXiv preprint 15/1.:/5/1 2015.
Stephan R Richter, Vibhav Vineet, Stefan Roth, and Vladlen Koltun. Playing for data: Ground truth from computer games. In European Conference on Computer Vision (ECCV), 2016.
10

---

Under review as a conference paper at ICLR 2020
Genman Ros, Laura Sellart, Joanna Materzynska, David Vazquez, and Antonio M Lopez. The synthia dataset: large collection of synthetic images for semantic segmentation of urban scenes. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2016.
Subhankar Roy, Aliaksandr Siarohin, Enver Sangineto, Samuel Rota Bulo, Nicu Sebe, and Elisa Ricci. Unsupervised domain adaptation using feature-whitening and consensus loss. In IEEE Conference on Computer Vision and Puttern Recognition (CVPR), 2019.
Kate Saenko. Brian Kulis, Mario Fritz, and Trevor Darrell. Adapting visual category models to new domains. In European Conference on Computer Vision (ECCV), 2010.
Kuniaki Saito, Yoshitaka Ushiku, and Tatsuya Harada. Asymmetric tri-training for unsupervised domain adaptation. In International Conference on Machine Learning (ICML), 2017.
Kuniaki Saito, Kohei Watanabe, Yoshitaka Ushiku, and Tatsuya Harada. Maximum classifier dis- crepancy for unsupervised domain adaptation. In IEEE Conference Computer Vision and Pattern Recognition (CVPR), 2018.
Hidetoshi Shimodaira. Improving predictive inference under covariate shift by weighting the log- likelihood function. Journal of Statistical Planning and Inference, 90(2):227-244, 2000.
Rui Shu, Hung H Bui, Hirokazu Narui, and Stefano Ermon. A dirt-t approach to unsupervised domain adaptation. In International Conference on Learning Representations (ICLR), 2018.
Baochen Sun and Kate Saenko. Deep coral: Correlation alignment for deep domain adaptation. In
European Conference Computer Vision (ECCV). Springer, 2016.
Antti Tarvainen and Harri Valpola. Mean teachers are better role models: Weight-average consis- tency targets improve semi-supervised deep learning results. In Advances in neural information processing systems, pp. 1195-1204, 2017.
Yi-Hsuan Tsai, Wei-Chih Hung, Samuel Schulter, Kihyuk Sohn, Ming-Hsuan Yang, and Manmohan Chandraker. Learning to adapt structured output space for semantic segmentation. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2018.
Yi-Hsuan Tsai, Kihyuk Sohn, Samuel Schulter, and Manmohan Chandraker. Domain adaptation for structured output via discriminative representations. IEEE International Conference on Computer Vision (ICCV), 2019.
Eric Tzeng, Judy Hoffman, Ning Zhang, Kate Saenko, and Trevor Darrell. Deep domain confusion: Maximizing for domain invariance. arXiv preprint arXiv:1412.3474, 2014.
Eric Tzeng, Judy Hoffman, Kate Saenko, and Trevor Darrell. Adversarial discriminative domain adaptation. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2017.
Hemanth Venkateswara, Jose Eusebio, Shayok Chakraborty, and Sethuraman Panchanathan. Deep hashing network for unsupervised domain adaptation. IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2017.
Tuan-Hung Vu, Himalaya Jain, Maxime Bucher, Matthieu Cord, and Patrick Perez. Dada: Depth- aware domain adaptation in semantic segmentation. In The IEEE International Conference Computer Vision (ICCV), October 2019a.
Tuan-Hung Haun-Hun Vu, Himalaya Jain, Maxime Bucher, Matthieu Cord, and Patrick Pérez. Advent: Adver- sarial entropy minimization for domain adaptation in semantic segmentation. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2019b.
Ruijia Xu, Guanbin Li, Jihan Yang, and Liang Lin. Unsupervised domain adaptation: An adaptive feature norm approach. In IEEE Conference on Computer Vision (ICCV), 2019.
Hong Ming Yang, Xu- Yao Zhang, Fei Yin, and Cheng-Lin Liu. Robust classification with convolu- tional prototype learning. In IEEE Conference on Computer Vision and Pattern Recognition, pp- 3474-3482, 2018.
11

---

Under review as a conference paper at ICLR 2020
Werner Zellinger, Thomas Grubinger, Edwin Lughofer, Thomas Natschläger, and Susanne Saminger- Platz. Central moment discrepancy (cmd) for domain-inwe inniriiniv representation learn- ing. In International Conference Learning Representations (ICLR), 2017.
Jing Zhang, Wanqing Li, and Philip Ogunbona. Joint geometrical and statistical alignment for visual domain adaptation. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2017.
Weichen Zhang, Wanli Ouyang, Wen Li, and Dong Xu. Collaborative and adversarial network for unsupervised domain adaptation. In IEEE Conference on Computer Vision and Pattern Recogni- tion (CVPR), 2018.
Yang Zou, Zhiding Yu, BVK Vijaya Kumar, and Jinsong Wang. Unsupervised domain adaptation for semantic segmentation via ballncbdl self self-training. In European Conference on Computer Vision (ECCV), 2018.
12

---

Under review as a conference paper at ICLR 2020
A APPENDIX
A.1 THEORETICAL INSIGHTS
We try explain why our PAAL works well for UDA according to the domain adaptation theory proposed in (Ben-David et al., 2010). Denote by (p(F) P * y) the risk of classifier model F E H. w.r.t. the distribution P. and by e e F2) [ (F) o [2( the disagreement between hypotheses Fi, F2 H. Particularly, (Ben- Devid al., 2010) gives well-known upper bound on the target risk (F) of classifier F in the following,
(
(Q(F) & (p(F) + [cr((")) + +c -
(12)
where F* is the ideal classifier induced from F arg (i and the last term is related to the classical H-divergence dHaH( (P.Q) = F (r( F*) eo (F, F*)] Be- sides, according to (Ben-David al., 2010), the empirical H-ddivergecce calculated by m respective samples from distributions P and Q converges uniformly to the true H-divergence for classifier classes H of finite VC dimension d, which is expressed as
d (20g(2m) (2m) 12g(2/))
duau (P, S duan(P.Q)) + +
(13)
m7
The work (Ganin Lempitsky, 2015) introduces binary domain discriminator to minimize the empirical H-divergence d Q). which aligns the marginal distributions well. However, two multi-class distributions P and Q not semantically aligned, there may not be any classifier with low risk both domains, which means the second term of the upper bound in Eq. (12) is very large. The proposed PAAL scheme leverages reliable conditional information in the adversarial learning module so that semantically similar samples from different domains are implicitly aligned, thus has a high possibility of decreasing the second term. Compared with (Long al., 2018), the input to domain the adversarial learning module is much more compact (2 < c x d), which helps decrease the second term in Eq. (13).
13


Reviews:
[{"review_id": "Skl3YsvOdH", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes to leverage prototypes to solve the mismatch problem in unsupervised domain adaptation. It further imposes intra-class compactness to help ambiguous classes. Experiments show it achieves new state-of-the-art results in several datasets.\n\npros:\n+ intra-class compactness to help ambiguous classes\n\nconcerns:\n-- Prototypes does not come from nowhere. They come from predictions. If you worry about the quality of target predictions (pseudo labels), then Eq. 8 and Eq. 9 are questionable. The intra-class compactness relies on p_t, too. The authors should explain why prototypes are superior than pseudo labels in [1].\n-- How does the authors select hyper-parameters? There are lots of magic numbers in Section 4.1 about hyper-parameters but no clues about how to tune them. Recently there is a paper [2] about model selection for UDA, maybe the authors should try it.\n\ndetails:\n- terminology: \"intra-class\" is better than \"within class\"\n- separate citations: e.g. entropy minimization, mean-teacher, and virtual adversarial training, have been successfully applied to UDA (Vu et al., 2019; French et al., 2018; Shu et al., 2018) -> entropy minimization (Vu et al., 2019), mean-teacher (French et al., 2018), and virtual adversarial training (Shu et al., 2018), have been successfully applied to UDA\n- confusion: At the last of Section 3.2, it says \\hat{f}=M^{T}p. But in Eq. 9, \\hat{f} and M^{T}p are concatenated, which is confusing: why do you concatenate two identical vectors?\n- Implementation Details: Section 4.1, paragraph 4: \\lambda^{f}_{adv} =5e-3, \\lambda^{f}_{adv} and \\lambda^{p}_{adv} increase from 0 to 1. It is confusing that \\lambda^{f}_{adv} both is a constant and changes continuously. \n\n[1] Conditional adversarial domain adaptation, Long et.al, in NeurIPS 2018\n[2] Towards Accurate Model Selection in Deep Unsupervised Domain Adaptation, You et.al, in ICML 2019", "extra_scores": {"experience_assessment": "I have published in this field for several years.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I did not assess the derivations or theory.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly."}, "extra_text": {}}, {"review_id": "BJg0OCVRtS", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary:\n- key problem: address \"class mismatch\" in adversarial learning methods for unsupervised domain adaptation (UDA);\n- contributions: 1) extension of the domain adversarial learning objective to leverage class prototypes (exponential moving average of features weighted by predicted class probabilities) in addition to pseudo-labels and intermediate representations (cf. eqs.5-11), 2) state-of-the-art results on several UDA tasks (Office-Home, ImageCLEF-DA, sim2real on Cityscapes).\n\nRecommendation: weak accept (with some reservations below).\n\nKey reason: interesting and effective use of prototypes for UDA.\n- The formulation of the prototypes and additional learning objectives for UDA are clear and seem novel, although I would like to see a discussion of additional related works:\n-- \"Mean teachers are better role models: Weight-averaged consistency targets improve semi-supervised deep learning results\", Tarvainen and Valpola, NeurIPS'17;\n-- \"Unsupervised Domain Adaptation with Similarity Learning\", Pinheiro, CVPR'18;\n-- \"Transferable Prototypical Networks for Unsupervised Domain Adaptation\", Pan et al, CVPR'19.\n- The effectiveness of the contributions is validated on multiple UDA tasks, and the ablative analysis supports the claims (that prototype-level alignment and within-class compactness helps).\n\nMain reservation: the specific problem is not clearly formalized.\n- What is the often mentioned but not clearly described \"class mismatch\" problem in UDA? To the best of my knowledge, this not a standard problem (could not find any mention in the previous literature, no citations or definitions in the submission). Is it that the target label space is different than the source label space (e.g., different ontologies)? In this case, what is the information on the target label space that enables unsupervised adaptation from the source one? What is the inductive bias / prior / assumptions? \n- Alternatively, is the tackled problem only the noise in the pseudo-labels?\n- In any case, the submission would greatly benefit from a clearer mathematical formalism and experimental characterization of the specific problem tackled here, especially in light of claims like \"conditioning the alignment on pseudo labels can not well address the mismatch problem. Compared with the pseudo labels, the class prototypes are more robust and reliable in terms of representing the distribution of different semantic classes.\"\n\nAdditional Feedback:\n- missing references on sim2real UDA: \"DADA: Depth-aware Domain Adaptation in Semantic Segmentation\" (Vu et al, ICCV'19), \"SPIGAN: Privileged Adversarial Learning from Simulation\" (Lee et al, ICLR'19)\n\n## Post rebuttal update\n\nI would like to thank the authors for replying to our questions. The clarifications with respect to related works and missing references is helpful, although a bit high-level (i.e. not necessarily describing the relative advantages of the proposed method). Nonetheless, the expected benefits of prototypes is still not entirely clear enough here, for instance regarding the main statistical assumptions that the method needs to make to get robust prototypes (e.g., in the presence of outliers or specific forms of \"inaccuracies\" in the pseudo-labels or \"domain misalignment\"). Therefore, due to the overall lack of mathematical clarity in the text and rebuttal, my main reservation remains, and I will change my \"weak accept / borderline\" score to weak reject. I encourage the authors to formalize the problem in a clearer, non-ambiguous way, discussing more explicitly the limitations of the proposed method.", "extra_scores": {"experience_assessment": "I have published in this field for several years.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper focuses on adversarial domain adaptation, and proposes an approach inspired from the DANN. The contribution lies in additional terms in the loss, aimed to i) align the source and target prototypes  in each class (using pseudo labels for target examples); ii) minimize the variance of the latent representations for each class in the target domain. 

Reviews point out that the expected benefits of target prototypes might be ruined if the pseudo-labels are too noisy; they note that the specific problem needs be more clearly formalized and they regret the lack of clarity of the text. The sensitivity w.r.t. the hyper-parameter values needs be assessed more thoroughly. 

One also notes that SAFN is one of the baseline methods; but its best variant (with entropic regularization) is not considered, while the performance thereof is on par or greater than that of PACFA for ImageCLEF-Da; idem for AdapSeg (consider its multi-level variant) or AdvEnt with MinEnt. 

For these reasons, the paper seems premature for publication at ICLR 2020. 
Decision: Reject