Unsupervised Domain Adaptation through Self-Supervision
Authors: Yu Sun
Eric Tzeng
Trevor Darrell
Alexei A. Efros
Venue:  (2020)
Keywords: unsupervised domain adaptation

Abstract: This paper addresses unsupervised domain adaptation, the setting where labeled training data is available on a source domain, but the goal is to have good performance on a target domain with only unlabeled data. Like much of previous work, we seek to align the learned representations of the source and target domains while preserving discriminability. The way we accomplish alignment is by learning to perform auxiliary self-supervised task(s) on both domains simultaneously.  Each self-supervised task brings the two domains closer together along the direction relevant to that task. Training this jointly with the main task classifier on the source domain is shown to successfully generalize to the unlabeled target domain.  The presented objective is straightforward to implement and easy to optimize. We achieve state-of-the-art results on four out of seven standard benchmarks, and competitive results on segmentation adaptation. We also demonstrate that our method composes well with another popular pixel-level adaptation method.

Under review as a conference paper at ICLR 2020
UNSUPERVISED DOMAIN ADAPTATION THROUGH SELF-SUPERVISION
Anonymous authors Paper under double-blind review
ABSTRACT
This paper addresses unsupervised domain adaptation, the setting where labeled training data is available on a source domain, but the goal is to have good perfor- mance on a target domain with only unlabeled data. Like much of previous work, we seek to align the learned representations of the source and target domains while preserving discriminability. The way we accomplish alignment by learning to perform auxiliary self-superrvsed task(s) on both domains simultaneously. Each self-supervised task brings the two domains closer together along the direction relevant to that task. Training this jointly with the main task classifier on the source domain shown to successfully generalize to the unlabeled target domain. The presented objective is straightforward to implement and easy to optimize. We achieve state-of-the-an results on four out of seven standard benchmarks, and competitive results segmentation adaptation. We also demonstrate that method composes well with another popular pixel-level adaptation method.
INTRODUCTION
Visual distribution shifts are fundamental our constantly evolving world. We humans face them all the time, e.g. when we navigate a foreign city, read text in a new font, or recognize objects in an environment we have never encountered before. These real-world challenges to the human visual perception have direct parallels in computer vision. Formally, a distribution shift happens when model is trained on data from one distribution (source), but the goal is to make good predictions on some other distribution (target) that shares the label space with the source. Often computational models struggle even for pairs of distributions that humans find intuitively similar.
Our paper studies the setting of unsupervised domain adaptation, with labeled data in the source domain, but only unlabeled data in the target domain. The general philosophy of the field is to induce alignment of the source and target domains through some transformation. In the context of deep learning, convolutional neural network maps images to learned representations in some feature space, so inducing alignment is done by making the distribution shifts small between the source and target in this shared feature space (Csurka [017] Wang & Deng. 2018; Gopalan et al.) 2011). If, in addition, such representations preserve discriminability can learn a good classifier on the source, which now generalizes to the target under the reasonable assumption that the representations of the two domains have the same ground truth.
the source domain, then
Most existing approaches implement this philosophy of alignment by minimizing a measurement of distributional discrepancy in the feature space (details in section 2). often some form of maximum mean discrepancy (MMD) e.g. Long et al. (2017), a learned discriminator of the source and target as an approximation the total variation distance e.g. Ganin al. (2016). Both measurements lead to the formulation of the training objective as minimax optimization problem a.k.a. adversarial learning, which known to be very difficult to solve. Unless carefully balanced, the push and pull in opposite directions divergence (details in |section 2). Therefore, we propose to avoid minimax optimization altogether through very different approach.
often cause wild fluctuations in the discrepancy loss and lead to sudden
Our main idea is to achieve aligmment between the source and target domains by training a model on the same task in both domains simultancously. Indeed, if we had labels in both domains, could simply use our original classification task for this. However, since we lack labels in the target domain, propose to use superviue auxiliary task, which creates its own labels directly from

---

Under review as a conference paper at ICLR 2020
Self-superiied classifer #1
Self-aupervised classilir #1
Source classifee
Source clasaitior
classillir
Baattsupervised classilier 42
(a) Source classifier only
(b) adding self-supervised task
(c) adding more tasks
Figure 1: We propose method for unsupervised domain adaptation that uses self-supervision to align the learned representations of two domains in shared feature space. Here we visualize how these representations might be aligned in the feature space. a) Without domain, and source classifier cannot generalize to target. b) Training shared representation to support one self-supervised task on both domains can align the source and target along one direction. e) Using multiple self-sun suff-uueeviied tasks can further align the domains along multiple directions. Now the source and target are close in this shared feature space, and the source classifier hope generalize to the target.
method, the source domain is far away from the target
fact, we can use multiple self-supervised tasks, each one aligning the two domains along direction of variation relevant to that task. Jointly training all the self-supervised tasks on both domains together with the original task on the source domain produces well-aligned representations as shown Figure
data (see sectiintoon 3p.
Like all of deep learning, we can only empirically verify that at least in our experiments, the model does not overfit by internally creating different decision boundary for each domain along different dimensions, which would then yield bad results. Recent research suggests that stochastic gradient descent is indeed unlikely to find such overfitting solutions with costly decision boundary of high complexity (implicit regularization), even though the models have enough capacity ZZhang et al., 2016a Neyshabur et al. 2017b; Arora et al. 2018).
The key contribution of our work is to draw a connection between unsupervised domain adapta- tion and self-supervised learning. While we do not propose any fundamentally new self-supervised tasks, we offer insights in section 3) on how to select the right ones for adaptation, and propose in section 4 a novel training algorithm on those tasks, using batches of samples from both domains. Additionally, demonstrate that domain alignment could be achieved with simple and stable algorithm, without need for adversarial learning. In section we report sfate-of-the-art results on several standard benchmarks.
2 RELATED WORK
In this section we provide a brief overview the two fields that our work would like to bridge.
2.1 UNSUPERVISED DOMAIN ADAPTATION
Methods for unsupervised domain adaptation in computer vision can be divided into three broad classes. The dominant class, which our work belongs to, aims induce alignment between the source and the target domains in some feature space. This has been done by optimizing for some measurement of distributional discrepancy. One popular measurement the maximum mean dis- crepancy (MMD) the distance between the mean of the two domains in some reproducing kernel Hilbert space, where the kernel chosen to maximize the distance (Bousmalis et al., 2016 Long et al. 2015, 2017). Another way to obtain measurement of discrepancy is to train an adversarial discriminator that distinguishes between the two domains (Ganin & Lempitsky 2014 Ganin et al.) 2016 Tzeng al. 2017). However, both MMD and adversarial training are formulated as minimax optimization problems, which are widely known, both in theory and practice, to be very difficult (Fedus et al. 2017 Duchi al. 2016 Liang 2017: Jin et al.) 2019). Since the optimization land- scape is much more complex than in standard supervised learning. training often does not converge converges bad local minimum (Goodfellow 2016
Nagarajan & Kolter 2017
Li al. 2017)
2

---

Under review as a conference paper at ICLR 2020
and requires carefully balancing the two sets of parameters (for minimization and maximization) so one does not dominate the other (Salimans et al. 2016 Neyshabur et al. 2017a).
To make minimax optimization casier, researchers have proposed numerous modifications to the loss function, network design, and training procedure (Arjovsky et al., 2017; Gulrajani et al. 2017
Karras et al. 2017 Courty al., 2017| Sun & Saenko [016] (hhu al. 2018 Sener et al. 2016) Over the years, these modifications have yielded practical improvements on many standard bench- marks, but have also made the state-of-the-art algorithms very complicated. Often practitioners are not sure which tricks are necessary for which applications, and implementing these tricks can be bug-prone and frustrating. To make matters worse, since there is no labeled target data available for a validation set, practitioners have no way to perform hyper-
tuning or early stopping.
The second class of methods directly transforms the source images to resemble the target images with generative models (Taigman et al., 2016 Hoffmmn et al. 2017, Bousmalis et al., 2017). While similar to the first class in the philosophy of alignment, these methods operate on image pixels directly instead of an intermediate representation space, and therefore can benefit from an additional round of adaptation in some representation space, In (ssubeection 5.2) we demonstrate that composing our method with popular pixel-level method yields stronger performance than either alone.
The third class of methods uses a model trained on the labeled source data to estimate labels on the target data, then trains on some of those estimated pscudo-labels (c.g. the most confident ones), therefore bootstrapping through the unlabeled target data. Sometimes called self-ensembling (French et al. 2017). this technique is borrowed from semi-supervised learning, where it is called co-training (Saito et al. 2017] Zou et al. 2018 Chen et al.) 2018) (011). In contrast, our method uses joint training (of the main and self-supervised tasks), different from co-training in every aspect except the name.
2.2 SELF-SUPERVISED FEATURE LEARNING
The emerging field of self-supervised learning uses the machinery of supervised learning on prob- lems where external supervision is not available. The idea is use data itself supervision for auxiliary (also called "pretext") tasks that learn deep feature representations which will hopefully be informative for downstream "real" tasks. Many such auxiliary tasks have been proposed in the literature, including colorization (predicting the chrominance channels of an image given its lumi- nance) Zhang et al.) 2b Larsson al. 20017
Zhang al. 2017). image inpainting Pathak et al. 2015). solving jigsaw puzzles (Noroozi & Favaro 2016), image rotation prediction (Gidaris et al. 2018). predicting audio from video (Owens et al.
20166) spatial context prediction (Doersch et al.
2016). contrastive predictive coding (Oord et al. [2018). etc. Researchers have also experimented
with sus--superviiion on videos (Wang & Gupta 2015 Wang et al. 2019). and combining multiple self-supervised tasks together (Doersch & Zisserman 2017).
Typically, self-supervision is used as a pre-training step on unlaheled data (e.g. the ImageNet train- ing set without labels) to initialize deep learning model, followed by fine-tuning on a labeled training set (e.g. PASCAL VOC) and evaluating on the corresponding test set. Instead, in this paper, we train the self-supervised tasks together with the main supervised task, encouraging a consistent representation that both aligns the two domains and does well on the main task]]
Recently, self-supervision has also been used for other problem settings, such as improving robust- ness (Hendrycks et al.) 2019), domain generalization (Carlucci et al. (2119) and few-short learning (Su et al. 2019). The most relevant paper to is Ghifary al. (2016), which uses self-supervision for unsupervised domain adaptation, but not through alignment. Their algorithm trains a denoising autoencoder /innent et al. (2008) only on the target data, together with the main classifier only on the labeled source data. They argue theoretically that this is better than training the autoencoder on both domains together. However, their theory is based on the critical assumption that the domains are already aligned, which is rarely true in practice. Consequently, their empirical results are much weaker than ours, as discussed in isection 5 Please see [Appendix A for more detailed comparisons with these works.
Note that it is possible to apply the standard pre- vre-triniig followed by fine-tuning regime to unsupervised domain adaptation, doing self-supervised pre-training on the target domain (or both the source and the target) and then fine-tuning on the souree. However, this gives almost no benefit over no-adaptation baseline, and is far from being competitive.
3

---

Under review as a conference paper at ICLR 2020
3 DESIGNING SELF-SUPERVISED TASKS FOR ADAPTATION
The design of auxiliary self-supervised tasks is an exciting area of research in itself. with many suc- cessful examples listed in /section 2] However, not
Task
Images and self-supervised labels
all of them are suitable for unsupervised domain adaptation. In order to induce alignment between the source and target, the labels created by self- supervision should not require capturing information on the very factors where the domains are meaning- lessly different, i.e. the factors of variation that we are trying to eliminate through adaptation.
Rotation
0
90
180
270
Location
(0.0) 0) (1,0)
(0, 1)
(1,1)
A particularly unsuitable tasks are these that try to predict pixels of the original image, as image in- painting (Pathak et al.] (016), colorization (Zhang) 2017 et al. 2017)
Table 1: Transformations and labels on sample input image created by two of the self-supervised tasks used in our work.
al. 2
or denoising autoencoder( Vincent et al. 2008 The
success of pixel-wise reconstruction depends strongly on brightness information, or other factors of variation in overall appearance (e.g. sunny vs. coudy) that are typically irrelevant to high-level visual concepts. Thus, instead of inducing aligmment, learning a pixel reconstruction task would stead serve to further separate the domains. We have experimented with using the colorization task and the denoising autoencoder for our training algorithm, and found their performance little better than the source only baseline, sometimes even worse!
In general, classification tasks that predict structural labels seems better suited for our purpose than reconstruction tasks that predict pixels. Therefore, we have settled on three classification-based supervuper tasks that combine simplicity with high performance:
Rotation Prediction: (Gidaris et al. 2018) An input image is rotated in 90-degree increments i.e. 0%, 90%, 180", and 270": the task is to predict the angle of rotation as a four-way classification problem.
Flip Prediction: An input image is randomly flipped vertically; the task is to predict whether the image is flipped or not
Patch Location Prediction: Patches are randomly cropped out of an input image; the task is to predict where the patches come from
Consider trivial illustrative example where the source and target are exactly the same except that the target pixels are all scaled down by a constant factor (e.g. daylight to dusk transition). All three of the aforementioned forms of self-supervision are suitable for this example, because pixel scaling
i.e. brightness is "orthogonal" to the prediction of rotation, flip and location.
4 METHOD
Our training algorithm is simple once a of K self-supervised tasks selected. We already have the loss function on the main prediction task, denoted Lo. that we do not have target labels for. Each self-supervised task corresponds to loss function Lg for I..... So altogether, our optimization
II is interesting note that works borrowing from semi-supervised learning, e.g. Ghifary al. 2016 use denoising autoencoder their training algorithm and obtain performance better than source only (but still not competitive with ours). As explained in Appendix A, this difference is due to the difference in training algorithms we use self supervision on both domains while they only use on the target. This further reflects target data for alignment whereas they use simply extra data in
the difference in philosophy,
semi-supervised learning.
We do not use horizontal flips. which is common data augmentation technique, since it is typically desirable for natural scene features be invariant to horizontal flips.
*For large images (e.g. segmentation), the crop comes from a continuous set of coordinates, and the task a regression problem in two dimensions, trained with the square loss. For small images (e.g. object recognition). the crop comes from one of the four quadrants, and the task is a four-way classification problem; this distinction exists only for case of implementation.
4

---

Under review as a conference paper at ICLR 2020
3 3
2
head
9
Supervised task
Source
Shared feuture encoder
Ronation head
Rotation prediction
head
Target
Location prediction
Self-Supervised Tasks
Figure 2:
Our method jointly trains a supervised head on labeled source data and self-supervised heads on unbaled data from both domains. The heads use high-level features from shared encoder, which learns to align the feature distributions.
problem is set up as a combination of these K + loss functions, as is done for standard multi-task learning (see Figure 2p. Implementation details are included in Appendix
Each loss function corresponds to a different "head" hg for = O...K, which produces predictions in the respective label space. All the task-specific heads (including ho for the actual prediction task) share a common feature extractor Φb Altogether, the parameters of Φ and hu, k = 0, ..k are the learned variables i.e. the free parameters of our optimization problem.
In our paper, & is deep convolutional neural network, and every h& is simply a linear layer i.e. multiplication by a matrix of size output space dimension X feature space dimension. If kth task is classification in nature, then h4 also has a softmax or sigmoid (for multi-class or binary) following the linear layer. The output space dimension is only four for rotation and location classification, and two for flip and location regression. Depending on the network architecture used, the feature space dimension ranges between 64 and 512. The point forced to share high-level features, is desirable for inducing alignment. A linear map from the highest-level features to the output is the smallest possible head and performs well empirically.
make every h2 low capacity, so the heads are
Let S = ((r,, vR), i 1...m) contain the labeled source data, and 7 = ((x;), i = 1...n) contain the unlabeled target data. Lo for the main prediction task takes in the labeled source data, and produces the following term in our objective:
(o h,)
Σ Lo(ho(((z)).
(a:))S
Each self-supervised task F6 for ls = 1...K modifies the input samples with some transformation fx and creates labels ÿ. Denote (S) = ((ff((a))) ūi), 1...m} as the self-supervised samples gen- erated from the source samples (with the original labels discarded). and FT(T) {(fa(m)) ..). 1...n) from the target. Then the loss La of each task 1...K produces the following term:
Ls(S,T; á hw)
Σ
Le(ha(o((a(()))
Σ
Lu(hu (o(aa)))((())) (1)
(EF(S)
(L(R))))F(T)
Note that (s, o, h2) for = 1. unlike Lo(S; o. ho), take in both the source and target data; emphasize for many times throughout the paper, this is critical for inducing alignment. Altogether, our optimization problem can be formalized as in multi-task learning 5
K
( ho) + Σ (A,,TTÚO,
min.
A.
'We have experimented with trade-off and found them unnecessary. Since labeled target validation set might not be available, it can he beneficial to reduce the number of hyper-parameters.
Ar = 1...K for the loss terms inside the sum
5

---

Under review as a conference paper at ICLR 2020
30
hurmatizedd meen divlanee Target emor (k) Source error
Normelized meen datence
Target error 1961
Source 196
Rotabion prodiction
Rotation prediction
20
40-
10-
20.
www
Epoch
Figure 3: Results on MNIST -NNIS-M (left) and CIFAR-IO-STLL- (right). Test error con- verges smoothly the source and target domains for the main task well the self-supervised task. This kind of smooth convergence is often seen in supervised learning, but rarely in adversarial learning. The centroid distance (linear MMD) between the feature distributions of the two domains converges alongside, even though is never explicitly optimized for.
At test-time, we discard the self-supervised heads and use ho(o(z)).
4.1 HEURISTIC FOR HYPER-PAR INTER-PRRAMETE TUNING AND EARLY STOPPING
As previously mentioned, in the unsupervised domain adaptation setting, there is no target label available and therefore target validation set, typical strategies for hyper-paramete tuning and early stopping that require a validation set cannot be applied. This ated; in fact, it is often unclear how previous works select their problem remains underappreci- detemine when training is finished, both of which can be important factors that impact performance, especially for complex algorithms using adversarial learning. In this subsection describe simple heuristic g merely as rule-of-thumb to make things work instead a technical innovation. Like almost all of deep learning, there is statistical guarantee this heuristic, but it is shown be practically effective in our experiments (see dieure 3). We only hope that it serves as the first guess of solution towards this underappreciated problem.
The main idea is that, because our method never explicitly optimizes for any measurement of distri- butional discrepancy, these measurements can instead be used for rrper-parameter tuning and early stopping. Since it would be counterproductive to introduce additional parameters in order to per- form hyper-parameter tuning, we simply use the distance between the mean of the source and target samples in the learned representation space, as produced by Φ. Formally, this can be expressed as
DIS' ('S",'::)==
1
Σ FES'
(xx)
1 Σ
((z)
(3)
FET
where S and T' unlabeled source and target validation sets
Our heuristic combines D(S', "'5","" TT; and the main task error on the (labeled) source validation set. Denote (f1,..., VT) and w (Wi,... Wr) the measurement vectors of those quantities respectively over T epochs. The final measurement vector is v/ min(v) + w/ min(w) i.c. normalized sum of the vectors; the epoch at which we perform early stopping is then simply arg migmi (1..7) u, Intuitively, this heuristic roughly corresponds to our goal of inducing alignment while preserving discriminability.
Appendix C contains some additional explanation of why this heuristic cannot be used for regular training. The short answer is given by the famous Goodhardt's 's law in economics and dynamical systems: "When a measurement becomes objective, it ceases be good measurement."
7 D above also known as the discrepancy under the linear kernel from the perspective of kernel MMD. Appendix C Contaiin some additional explanation of why the mean distance is suitable for our heuristic, even though it is specific form of MMD, which we claim to be difficult to optimize. The short answer is that it does require minimax optimization.

---

Under review as a conference paper at ICLR 2020
Source Target
MNIST MNIST-M
MNIST SVIIN
SVHN
MNIST
USPS MNIST
CIFAR-TO STL-10
MNIST
USPS
STL-10 CIFAR-I0
DANN (Ganin et al.
2016
81.5
35.7
73.6
DRCN Ghitary al.
2016
40.1
82.0
66.4
58.7
DSN (Bousmalis al.
2016
83.2
82.7
kNN-Ad (Sener al
2016
40.3
78.8
PixeIDA Bousmalns et al. 2017
98.2
ATT Saito et al
94.2
52.8
86.2
11-Model (French
2017
71.4
92.0
76.3
64.2
ADDA Tzeng et al.
76.0
89,4
90.1
CyCADA Hotlman of al.
22017
90.4
95.6
96.5
VADA (huu et al.
201
97.7
47.5
97.9
80.0
73.5
DIRT-T (Shu et al.
2018)
98.9
54.5
99.4
75.3
VADA (IN) Shu
2018)
95.7
73.3
94.5
78.3
71.4
DIRT-T (IN) (Shu et al.
76.5
99.4
73.3
Source only VADA & DIRFFT
58.5
27.9
77.0
76.3
63.6
Source only VADA & DIRT-T (IN)
59.9
40.9
82.4
77.0
62.6
Source only method
44.9
30.5
92.2
94.7
81.4
75.6
58.8
R
98.9
61.3
85.8
96.5
90.2
81.2
66.9
R+L+F
82.1
75.5
Table Test accuracy (%) on standard domain adaptation benchmarks. Our results are organized according to the self-supervised task(s) used: R for rotation, L. for location, and F for flip. We achieve state-of-the-at accuracy on four out the seven benchmarks.
5
EXPERIMENTS
5.1 SEVEN BENCHMARKS FOR OBJECT RECOGNITION
The seven benchmarks are based on the six datasets deseribed in Ampppndix D each with predefined training set test set split, and labels available both splits. Previous works (cited in Table 2 have created those seven benchmarks by picking pairs of datasets with the same label space, treating the source and the other the target, and training on the training set of the source with labels revealed and of the target with labels hidden. Following the standard setup of the field, labels on the target test set should only be used for evaluation, not hyper-parameet tuning or early stopping; therefore we apply the heuristic described in (suusscction 4.1
For the two natural scene benchmarks, we all three tasks described inssection 3 and flip prediction. For the five benchmarks on digits we do not use location because it yields trivial solutions that do not encourage the learning of semantic concepts. Given the image into the four quadrants, location prediction can be trivially solved by looking at the four corners where a white stroke determines category. Adding flip to the digits does not hurt performance, but does not improve significantly either,
rotation, location
digit cropped
do not report those results separately.
As shown in Table 2, despite the simplicity of our method, we achieve state-of-the-ar accuracy on four out of the seven benchmarks. In addition, show the source only results from closest competitor (VADA and DIRT-T), and note that our source only results are in fact lower than theirs those very benchmarks that perform the best on; this indicates that our success is indeed due to effectiveness in adaptation instead of the base architecture.
Our method fails the pair of benchmarks with SVHN, which rotation yields trivial solutions. Because SVHN digits are cropped from house numbers with multiple digits, majority of the images have parts of the adjacent digits the side. The main task head needs look the center, but the rotation head learns to look at the periphery and cheat. This failure case shows that the success of our method is tied how well the self-supervised task fits the application. Practioners should their domain knowledge evaluate how well the task fits, instead of blithely apply it to everything.
Also seen in Table is that our method excels object recognition natural scenes, especially with all three tasks together. For STL-IO-CCIFAR-ID our base model is considerably worse than that of VADA and DIRT-T, but still beats all the baselines. Adding location and flip gives an improvement of 8%, on top of the 8% already over source only. This is not surprising since those tasks were
We will make code public the paper accepted.

---

Under review as a conference paper at ICLR 2020
GTAS Cityseapes
sidewalkk
building
traffi light
uaffic sign
vegetation
motorbile
person
bicycle
raal
lence
pole
sky
rider
mick
hus
mloU
Source only
28.8
12.7
39.6
9.4
3.5
18.1
22.7
9.4
80.9
12.4
45.8
53.9
9.6
74.7
20,9
15.0
0.0
19.4
399
25.3
Ours
22.7
18.1
13.5
18.7
80.3
19.4
58.4
53.8
2.6
75.1
5.2
8.1
28.9
CyCADA
33.1
77.9
17.3
33.3
81.5
69.0
20.9
25.6
18.8
39.5
Ours + CYCADA
86,6
37,8
80.8
29.7
164
28.9
30,9
22.2
83.8
37.1
76,9
60.1
7.8
84.1
30.8
32.1
1.2
23.2
13.3
41.2
Oracle
97.3 79.8
88.6
32.5
48.2
56.3
63.6
73.3
89.0
58.9
93.0
78.2
55.2
92.2
45.0
67.3
39.6
49.9
73.6
67.4
Table 3: Test accuracy (%) on GTA5 - Cityscapes. Our method significantly improves source only, and also over CyCADA when combined. This indicates that additional self-supervision using our training algorithm further aligns the domains.
originally developed for ImageNet pre-training i.e. object recognition in natural scenes our method very successful when the task fits the application well.
5.2 BENCHMARK FOR SEMANTIC SEGMENTATION
To experiment with our method in more diverse applications, we also evaluate on challenging simulation-to-reat benchmark for semantic segmentation - GTA5 - Cityscapes. GTA5 (Richter) al. at.l. 2016) contains 24,966 video frames taken from the computer game, where dense segmentation labels are automatically given by the game engine. Cityscapes (Cordts et al. 2016) contains 5,000 video frames taken from real-world dash-cams. The main task to classify every pixel an image of the 19 classes shared across both datasets, and accuracy is measured by intersection over union (IoU). The best possible results are given as the oracle, when the labels on Cityscapes are available for training, typical supervised learning methods applicable.
Our results are shown in Table 3, and implementation details in Appendix F] Our self-supervised tasks were designed for classification, where the label image depends its global content, while in segmentation the labels tend to be highly local. Nevertheless, with very little modification, the source only baseline. In Appendix G we provide visual-
see significant improvements izations of the segmentation results, and make qualitative comparisons between those produced by the baseline and our method.
We also experiment with combining our method with another popular unsupervised domain adap- tation method - CyCADA (Hoffman et al. 2017), designed specifically for segmentation. Surpris- ingly, when operating on top of images produced by this already very strong baseline, our method further improves performance. This demonstrates that pixel-level adaptation methods might still benefit from an additional round of adaptation by inducing alignment through self-supervision. We emphasize that these results are obtained with a very simple instantiation of our method, as a start towards the development of self-supervised tasks more suitable for semantic segmentation.
DISCUSSION
We hope that this work encourages future researchers in unsupervised domain adaptation to con- sider the study of self-supervision as an alternative to adversarial learning, and researchers self- supervision to consider designing tasks and evaluating them in our problem setting. Most self- supervised tasks today were originally designed for pre-training and evaluated terms of accuracy gains downstream recognition, localization or detection tasks. It will be interesting to see if new self-supervised tasks can arise from the motivation of adaptation, for which alignment is the key objective. Moreover, domain experts could perhaps incorporate their dataset specific knowledge into the design self-supervised task specifically for their application.
One additional advantage of method, not considered in this paper, is that it might be particularly amenable to very small target sample size, when those other methods based on adversarial learning cannot accurately estimate the target distribution. We leave this topic for the future work.
8

---

Under review as a conference paper at ICLR 2020
REFERENCES
Pablo Arbelaez, Michael Maire, Charless Fowlkes, and Jitendra Malik. Contour detection and hi- erarchical image segmentation. IEEE transactions on pattern analysis and machine intelligence, 33(5):8989916, 2011.
Martin Arjovsky, Soumith Chintala, and Léon Bottou. arXiv: 077777 2017.
Wasserstein gan.
arXiv preprint
Sanjeev Arora, Rong Ge, Behnam Neyshabur, and Yi Zhang. Stronger generalization bounds for deep nets via a compression approach. arXiv preprint arXiv:1802.05229, 2018.
Konstantinos Bousmalis, George Trigeorgis, Nathan Silberman, Dilip Krishnan, and Dumitru Erhan. Domain separation networks. In Advances in Neural Information Processing Systems, pp. 343- 351, 2016.
Konstantinos Bousmalis, Nathan Silberman, David Dohan, Dumitru Erhan, and Dilip Krishnan. Unsupervised pixel-level domain adaptation with generative adversarial networks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 3722-3731, 2017.
Fabio M Carlucci, Antonio D' I'nnnceeeee, Silvia Bucci, Barbara Caputo, and Tatiana Tommasi. Do- main generalization by solving jigsaw puzzles. In Proceedings of the IEEE Conference Com- puter Vision and Pattern Recognition, pp. 2229-2238, 2019.
Hlang-Chiet Chen, George Papandreou, Florian Schroff, and Hartwig Adam. Rethinking atrous convolution for semantic image segmentation. 2017. URL http://arxiv.og/abb////6/60.
Minmin Chen, Kilian Q Weinberger, and John Blitzer. Co-training for domain adaptation. In Ad- vances neural information processing systems, pp. 1466,2464, 2011.
Xilun Chen, Yu Sun, Ben Athiwaratkun, Claire Cardie, and Kilian Weinberger. Adversarial deep averaging networks for cross- linss-l sentiment classification. Transactions of the Association for Computational Linguistics, 6:557-570, 2018.
Adam Coates, Andrew Ng. and Honglak Lee. An analysis single-layer networks in unsupervised feature learning. In Proceedings of the fourteenth international conference on artificial intelli- gence and statisties, pp. 215-223.1 2011.
Marius Cordts, Mohamed Omran, Sebastian Ramos, Timo Rehfeld, Markus Enzweiler, Rodrigo Benenson, Uwe Franke, Stefan Roth, and Bernt Schiele. The cityscapes dataset for semantic urban scene understanding. In Proc. of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2016.
Nicolas Courty, Rémi Flamary, Devis Tuia, and Alain Rakotomamonjy. Optimal transport for do- main adaptation. IEEE transactions on pattern analysis and machine intelligence, 39(9):1853- 1865, 2017.
Gabriela Csurka. Domain adaptation for visual applications: A comprehensive survey. arXiv preprint 1721::100..05374 2017.
D. Das and C. S. George Lee. Unsupervised domain adaptation using regularized hyper-graph matching. In 2018 25th IEEE International Conference on Image Processing (ICIP), pp. 3758- 3762, Oct 2018. doi: 10.1109//C1.22088.4455
Debasmit Das and C. S. George Lee. Graph matching and pseudo-label guided deep unsupervised domain adaptation. In Věra Kúrková, Yannis Manolopoulos, Barbara Hammer, Lazaros Iliadis, and Ilias Maglogiannis (eds.), Artificial Neural Networks and Machine Learning ICANN 2018, pp. 342-352, Cham, 2018. Springer International Publishing. ISBN 978-3-030-01422-7..
Carl Doersch and Andrew Zisserman. Multi-task self-supervised visual learning. In Proceedings of the IEEE International Conference on Computer Vision, pp. 2051-2060) 2017.
Carl Doersch, Abhinav Gupta, and Alexei A Efros. Unsupervised visual representation learning by context prediction. In Proceedings of the IEEE International Conference on Computer Vision, pp- 1422-1430, 2015.

---

Under review as a conference paper at ICLR 2020
John C Duchi, John Lafferty, Yuancheng Zhu, et al. Local minimax complexity of stochastic convex optimization. In Advances in Neural Information Processing Systems, pp. 3423-3431, 2016.
William Fedus, Mihaela Rosca, Balaji Lakshminarayanan, Andrew M Dai, Shakir Mohamed, and Ian Goodfellow. Many paths to equilibrium: Gans do not need to decrease a divergence at every step. arXiv preprini arXiv: 11t. 00.066,, 2017.
Geoffrey French, Michal Mackiewicz, and Mark Fisher. Self-ensembling for domain adaptation.
arXiv preprint arXiv: 2017.
Yaroslav Ganin and Victor Lempitsky. Unsupervised domain adaptation by backpropagation. arXiv preprint arXiv:1409. 7495, 2014.
Yaroslav Ganin, Evgeniya Ustinova, Hana Ajakan, Pascal Germain, Hugo Larochelle, François Laviolette, Mario Marchand, and Victor Lempitsky. Domain-adversarial training of neural net- works. The Journal of Machine Learning Research, 17(1):2096 2016.
Muhammad Ghifary, W Bastiaan Kleijn, Mengjie Zhang, David Balduzzi, and Wen Li. Deep ceanustrcttionccassfffaattt networks for unsupervised domain adaptation. In European Con- ference on Computer Vision, pp. 597-613. Springer, 2016.
Spyros Gidaris, Praveer Singh, and Nikos Komodakis. Unsupervised representation learning by predicting image rotations, arXiv preprint arXiv:1803.011 1803.07728, 2018.
Ian Goodfellow. Nips 2016 tutorial:
Generative adversarial networks.
arXiv preprint
arXiv:1701.00160. 2016.
Raghuraman Gopalan, Ruonan Li, and Rama Chellappa. Domain adaptation for object recognition: An unsupervised approach. In 2011 international conference on computer vision, pp. 999-1006. IEEE, 2011.
Ishaan Guirajani, Faruk Ahmed, Martin Arjovsky, Vincent Dumoulin, and Aaron C Courville. Im- proved training of wasserstein gans. In Advances in Neural Information Processing Systems, pp- 5767-5777, 2017.
Chuan Guo, Geoff Pleiss, Yu Sun, and Kilian Q Weinberger. On calibration of modern neural networks. In Proceedings of the 34th International Conference Machine Learning- Volume 70, pp. 1321-1330. JMLR. org, 2017.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Identity mappings in deep residual networks. European conference on computer vision, pp. 630-645. Springer, 2016.
Dan Hendrycks, Mantas Mazeika, Saurav Kadavath, and Dawn Song. Using self self-supervised learning can improve model robustness and uncertainty. arXiv preprint arXiv: /2340, 2019.
Judy Hoffman, Eric Tzeng, Taesung Park, Jun-Yan Zhu, Phillip Isola, Kate Saenko, Alexei Efros, and Trevor Darrell. Cycada: Cycle-consistent adversarial domain adaptation. arXiv preprint arXiv:1711.0322 2017.
Gao Huang, Yu Sun, Zhuang Liu, Daniel Sedra, and Kilian Q Weinberger. Deep networks with stochastic depth. In European conference on computer vision, pp. 646-661. Springer, 2016.
Chi Jin, Pranecth Netrapalli, and Michael I Jordan. Minmax optimization: Stable limit points of gradient descent ascent are locally optimal. arXiv preprint arXiv:1902.00611 2019.
Tero Karras, Timo Aila, Samuli Laine, and Jaakko Lehtinen. Progressive growing of gans for im- proved quality, stability, and variation. arXiv preprint arXiv: 10/96, 2017.
Alex Krizhevsky and Geoffrey Hinton. Learning multiple layers of features from tiny images. Tech- nical report, Citeseer, 2009.
Gustav Larsson, Michael Maire, and Gregory Shakhnarovich. Colorization as a proxy task for visual understanding. In CVPR, 2017.
10

---

Under review as a conference paper at ICLR 2020
Yann LeCun, Léon Bottou, Yoshua Bengio, Patrick Haffner, et al. Gradient-based learning applied to document recognition. Proceedings of the IEEE, (6(11))22782233 1998.
Jerry Li, Aleksander Madry, John Peebles, and Ludwig Schmidt. On the limitations of first-order approximation in gan dynamics. arXiv preprint arXiv: 170.09884, 2017.
Tengyuan Liang- How well can generative adversarial networks learn densities: A nonparametric view. arXiv preprint arXiv: 17lp:08244.04 2017.
Mingsheng Long, Yue Cao. Jianmin Wang, and Michael Jordan. Learning transferable features with deep adaptation networks. arXiv preprint arXiv: 5502.0279,, 2015.
Mingsheng Long, Han Zhu. Jianmin Wang, and Michael Jordan. Deep transfer learning with joint adaptation networks. In Proceedings of the 34th International Conference on Machine Learning- Volume 70, pp. 2208-22002 2017-221. JMLR. org. 2017.
Vaishnavh Nagarajan and J Zico Kolter. Gradient descent gan optimization is locally stable. In Advances in Neural Information Processing Systems, pp. 5585 05955559,2222 2017.
Yuval Netzer, Tao Wang, Adam Coates, Alessandro Bissacco, Bo Wu, and Andrew Ng. Reading digits in natural images with unsupervised feature learning. 2011.
Behnam Neyshabur, Srinadh Bhojanapalli, and Ayan Chakrabarti. Stabilizing gan training with multiple random projections. arXiv preprint
2017a.
Behnam Neyshabur, Ryota Tomioka, Ruslan Salakhutdinov, and Nathan Srebro. Geometry of opti- mization and implicit regularization in deep learning. arXiv preprint arXiv:1705.03071 20176.
Mehdi Noroozi and Paolo Favaro. Unsupervised learning of visual representations by solving jigsaw puzzles. In European Conference on Computer Vision, pp. 69-84. Springer, 2016.
Aaron van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predic- tive coding. arXiv preprint
2018.
Andrew Owens, Jiajun Wu, Josh H McDermott, William T Freeman, and Antonio Torralba. Ambient sound provides supervision for visual learning. In European Conference on Computer Vision (ECCV). 2016.
Deepak Pathak, Philipp Krahenbuhl, Jeff Donahue, Trevor Darrell, and Alexei A Efros. Context encoders: Feature learning by inpainting. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 2536-2566552 2544, 2016.
Stephan R Richter, Vibhav Vineet, Stefan Roth, and Vladlen Koltun. Playing for data: Ground truth from computer games. In European Conference on Computer Vision, pp. 102-118. Springer, 2016.
Kuniaki Saito, Yoshitaka Ushiku, and Tatsuya Harada. Asymmetric tri-training for unsupervised domain adaptation. In Proceedings of the 34th International Conference on Machine Learning- Volume 70, pp. 9988-2991 JMLR. org, 2017.
Tim Salimans, Ian Goodfellow, Wojciech Zaremba, Vicki Cheung, Alec Radford, and Xi Chen. Improved techniques for training gans. In Advances in neural information processing systems, pp. 2234-2242, 2016.
Ozan Sener, Hyun Oh Song, Ashutosh Saxena, and Silvio Savarese. Learning transferrable repre- sentations for unsupervised domain adaptation. In Advances in Neural Information Processing Systems, pp. 2110-2118, 2016.
Rui Shu, Hung H Bui, Hirokazu Narui, and Stefano Ermon. A dirt-t approach to unsupervised domain adaptation. arXiv preprint arXiv: N802::800000735 2018.
Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov. Dropout: a simple way to prevent neural networks from overfitting. The journal of machine learning research, 15(1):1929-199) 1958, 2014.
11

---

Under review as a conference paper at ICLR 2020
Long-Chyi Su, Subhransu Maji, and Bharath Hariharan. Boosting supervision with self-supervision for few-shot learning. arXiv preprint OrXiv:1906.0707, 2019.
Baochen Sun and Kate Saenko. Deep coral: Correlation alignment for deep domain adaptation. In European Conference on Computer Vision, pp. 443-450. Springer, 2016.
Yaniv Taigman, Adam Polyak, and Lior Wolf. Unsupervised cross-domain image generation. arXiv preprint arXiv: 1.02200, 2016.
Eric Tzeng, Judy Hoffmann Kate Saenko, and Trevor Darrell. Adversarial discriminative domain adaptation. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 7167-7176, 2017.
Pascal Vincent, Hugo Larochelle, Yoshua Bengio, and Pierre-Antoine Manzagol. Extracting and composing robust features with denoising autoencoders. In Proceedings of the 25th international conference on Machine learning, pp. 1096-1103, ACM, 2008.
Mei Wang and Weihong Deng. Deep visual domain adaptation: survey. Neurocomputing. 312: 135-153, 2018.
Xiaolong Wang and Abhinav Gupta. Unsupervised learning of visual representations using videos. In ICCV, 2015.
Xiaolong Wang, Allan Jabri, and Alexei A. Efros. Learning correspondence from the cycle- consistency of time. In CVPR, 2019.
Chiyuan Zhang, Samy Bengio, Moritz Hardt, Benjamin Recht, and Oriol Vinyals. Understanding deep learning requires rethinking generalization. arXiv preprint arXiv: /6/ 1,053000030,, 20.
Richard Zhang, Phillip Isola, and Alexei A Efros. Colorful image colorization. In European confer- ence on computer vision, pp. 649-666. Springer, 2016b.
Richard Zhang, Phillip Isola, and Alexei A Efros. Split-brain autoencoders: Unsupervised learning by cross-channel prediction. In Proceedings of the IEEE Conference Computer Vision and Pattern Recognition, pp. 1058-1067, 2017.
Yang Zou, Zhiding Yu, BVK Vijaya Kumar, and Jinsong Wang. Unsupervised domain adaptation for semantic segmentation via class-balanced self-training. Proceedings of the European Con- ference Computer Vision (ECCV), pp. 289-305, 2018.
12

---

Under review as a conference paper at ICLR 2020
APPENDIX
A ADDITIONAL DISCUSSION ON SPECIFIC PAPERS RELATED TO OURS
In this section we discuss the four papers mentioned in section 2 that are related ours, but different in training algorithm, philosophy or problem setting.
Networks (Ghifary et al. 2016). This method works in unsuper- vised domain adaptation, the same problem setting as ours. It learns a denoising autoencoder on the target, together with the main classifier on the source. However, they use only the target domain for reconstruction, claiming both empirically and theoretically that it is better than using both domains together. In contrast, we use both the source and target for self-supervision. These two algorithmic differences really reflect our fundamental difference in philosophy. They take the target data as anal- ogous to the unlabeled (source) data in supervisdd learning, so any form of self-supervision suitable for semi-supervised learning is good enough; their theoretical analysis is also directly bor- rowed from that of semi-supervised learning, which concludes that it is necessary and sufficient to only use the target data for self-supervision. We take data from both domains for self-supervision in order to align their feature distributions; also, both conceptually and empirically, we cannot use reconstruction tasks because they are unsuitable for inducing alignment (see [section 3).
Deep aecconttrution-Clasifficccc
Empirical experiments further support our arguments in three ways: 1) If we were to only use the target for -NNISTM, the easiest benchmark where we would otherwise observe a huge im- would
even on supf-suuerv tasks, we would perform barely better than the source only baseline,
provement. 2) If we were use the reconstruction task i.e. denoising autoencoder, again perform barely better than the source only baseline, and sometimes even worse, as described in|section 3] 3) As shown in Table our results are much better than theirs on all of the benchmarks by large margin. This shows that implementing the philosophy of alignment, developed for un- supervised domain adaptation, is much superior in our problem setting to blithely borrowing from semi-supervised learning.
In addition to the three classes of methods covered in infsection 2, unsupervised domain adaptation has also been studied using sample-to to sample correspondence through graph matching (Das & George Lee 2018 Das & Lee| 2018); these methods are less relevant to ours and not discussed in detail.
Using SuperSuue Learning Can Improve Model Robusiness and Uncertainty (Hendryeks et al.) 2019). This paper studies the setting of robustness, where there no sample provided, even unla- beled, from the target domain. Their method jointly trains for the main task and the self-supervised task only on the source domain, really because there is no target data of any kind. Because their problem setting is very challenging (with so little information provided evaluation is on a dataset of CIFAR- 10 images with the addition of 15 types of common corruptions (e.g. impulse noise, JPEG compression, snow weather, motion blur and pixelation), simulated by an algorithm. No idea on unsupervised domain adaptation is mentioned.
the target), their primary
Domain Generalization by Solving Jigsaw Puzzles (Carlucci al. 2019). This paper studies two setting. The first is the robustness setting, exactly the same as in Hendrycks et al. (201)). ex- cept that evaluation is done on MNIST-MNNSTT method from the robustness community that trains on adversarial examples and uses no target data, Again, because their problem setting is very challenging, the accuracy is low for both their proposed method and the baseline. The second setting is called domain generalization, which is very similar to meta-learning. The goal is to perform well simultaenously on multiple distributions, all labeled. Evaluation is done using the mean accuracy on all the domains. Beside the name, there is little similarity between the setting of unsupervised domain adaptation and domain generalization, which has no unsupervised component.
and MNIST-SVHN Their baseline is also
Boosting Supervision with Self-Supervision for Few-shot Learning (Su et al. 2019). As evident from the title, this paper studies the setting of few-shot learning, where the goal to perform well very small dataset. Again, there is no unsupervised component and little connection to our setting.
Most of our results have already been produced when Carlucci et al. (0119) and Su al. (2019)
were presented at a conference. Hendrycks et al. (0099) was submitted to IP 2019 and should be considered concurrent work.
13

---

Under review as a conference paper at ICLR 2020
B ADDITIONAL ALGORITHMIC DETAILS
In practice and for our experiments, the source and target datasets are often imbalanced in size. If were to blithely solve for the objective in Equation 2] the domain with larger dataset would carry more weight for every (x T; o. h2,,, k I...K because we are summing over all samples in both datasets. We would like to keep the two sums inside Cs roughly balanced such that in terms of features produced by Ø, neither of the two domains would dominate the other. This is easy to achieve in our implementation through balanced batches. When we need sample a batch for task k, we simply sample half the batch from the source, and another half from the target, then put them together. In the end, our implementation requires little change on top of an existing supervised learning codebase. Each self-supervised task is defined as module, and adding a new one only amounts to defining the structural modifications and the loss function.
We optimize the objective in Equation 2 with stochastic gradient descent (SGD). One can simply take joint step on Equation 2. which covers all the losses for & = 0 and k = 1...K However, the implementation would then have to store the gradients with respect to all K + 1 losses together. For memory efficiency, our implementation loops over I = 1...K; for each self-supervised task k it samples batch of combined source and target data (without their original labels), structurally modifies the images by SR. creates new labels according to the modification, and obtains a loss for ha and o on this batch. So a gradient step is taken for each k = 1...,, before finally a batch of original source images and labels are sampled for gradient step on ho and ø. In terms of test accuracy, these two implementations make little difference (usually less than 1 1%). and the choice simply comes down to a time-memory trade-off
Because our source only baseline on STL-10 - CIFAR-10 10 is considerably worse than the closest competitor DIRT-T, we borrow from their implementation and use dropout regularization (Srivastava) al., 2014) with p 0.5, only on this benchmark. This makes our source only baseline closer to theirs, so to allow for a fair comparison of the adaptation methods. Without this modification, our accuracy for source only, R, and R+L+F are respectively 56.1, 65.6 and 74.0%.
C ADDITIONAL DISCUSSION ON THE MEAN DISTANCE
In this section we answer some potential questions about our selection rule, which uses the mean distance, from the perspective of a possibly confused reader.
Q: The motivation of the paper is that methods based on minimax optimization, such those using kerneI-MMD, are difficult to optimize. Since you are using the mean distance (MMD under the linear kernel) for hyper-parameter tuning and early stopping, which is form of model selection, how are you different from those other methods and why is optimization easy for you?
A: Our method is fundamentally different from those other methods based MMD, and there- fore more amenable to optimization in the following two ways: 1) Even though we use the mean distance, which is a form of MMD, never pose minimax optimization problem. The model parameters minimize the loss functions of the tasks, which we hope makes the mean distance small (see Figure 3). Model selection using (subsection 4.1 also minimizes the mean distance. In our method, all the loss functions, for model parameters and peper-parameters, work towards the same goal of inducing alignment, so optimization is easier. 2) Even though hyper-paramen early stopping are forms of model selection just like training, there is a qualitative difference in the degrees of freedom involved. For (subsection 4. when performing early stopping for example, optimization amounts to a grid search over the epochs after training is finished, and there is only one degree of freedom.
tuning and
Q: Why is the mean distance suitable for model selection when it comes to hyper-parameter parameter tuning and early stopping. but not regular training?
A: Again, we agree that hyper-parameter tuning and early stopping are forms of model selection. However, unlike model parameters ranging in the hundred of thousands, there are at most a few hyper-parameters and only one parameter for early stopping. Model parameters can easily overfit to the mean distance while those few degrees of freedom we use cannot. This is precisely also the reason why previous works using MMD resort minimax optimization.
14

---

Under review as a conference paper at ICLR 2020
D DETAILS OF THE SIX DATASETS USED FOR OBJECT RECOGNITION
1) MNIST (LeCun et al. the training set and 10,000 in the test set.
1998): greyseale images of handwritten digits 0 to 9; 60,000 samples in
2) MNIST-M (Ganin et al. from the BSDS500 dataset Arbelaez et al. (2011); ; same training / test set size as MNIST.
2016); constructed by blending MNIST digits with random color patches
3) SVHN (Netzer et al. 2011]: colored images of cropped out house numbers from Google Street View; the task is to classify the digit the center; 73,257 samples in the training set, 26,032 in the test set and 531,131 easier samples for additional training.
4) USPS: greyscale images of handwritten digits only slightly different from MNIST; 7291 samples in the training set and 2007 in the test set.
5) CIFAR-IO (Krizhevsky & Hinton 2009): colored images of 10 classes of centered natural scene objects; 50,000 samples in the training set and 10,000 in the test set.
6) STL-10 (Coates et al., T) colored images of objects only slightly different from CIFAR-10; 5000 samples in the training set and 8000 in the test set.
Because CIFAR-10 10 and STL-10 differ in one class category, we follow common practice (Shu al. 2018 French et al. 2017 7 Ghifary et al. 2016) and delete the offending categories, so each of these Iwo datasets actually only has classes.
E IMPLEMENTATION DETAILS ON THE OBJECT RECOGNITION BENCHMARKS
We use a 26-layer pre-activation ResNet (He et al. 2016) as our test-time model lo (o(()), where ho the last linear layer that makes the predictions, and ø is everything before that. For unsupervised domain adaptation, there is no consensus on what base architecture to use among the previous works. Our choice is simply base on the widespread adoption of the ResNet architecture and the ease of implementation. In Table 2 we provide the source only results using our base architecture, and the ones from Shu et al. (2018), our closest competitor. Our source only results are in fact worse than theirs, indicating that our improvements are indeed made through adaptation.
At training time, the self-supervised heads hgg k K... are simply linear layers connected to the end of 6 as discussed in |section 4 There is no other modification on the standard ResNet. For all experiments on the object recognition benchmarks, we optimize our model with SGD using weight decay 5e-4 and momentum 0.9, with batch size of 128. We use initial learning rate of 0.1 and a two milestone schedule, where the learning rate drops by a factor of 10 at each milestone. All these optimization hyper-parameters are taken directly from the standard literature (He et al. [016]] Huang et al. 2016 Guo et al. 2017) without any modification for our problem setting. We select the total number of epochs and the two milestones based on convergence of the source classifier hog and unsupervised classifiers /t,,..,,iu, lxg. Early stopping is done using the selection heuristic discussed in subsection 4.1 For fair comparison with our baselines, we do not perform data augmentation, following previous works (Hoffman et al. 2017 Sener et al., 2016 Ghifary et al. 2016).
F IMPLEMENTATION DETAILS ON GTA5 > CITYSCAPES
For our experiments, we initialize our model from the DeepLab-v3 architecture (Chen et al. 2017 pre-trained on ImageNet, as commonly done in the field. Each self-supervised head consists of a global average pooling layer on the pre-logit layer, followed by single linear layer. To take advan- tage of the large size of the natural scene images, we use the continuous i.e. regression version of location prediction. The self-supervised head is trained on the square loss, to regress the coordinates (in two dimensions) that the patch is cropped from. Natural for the regression version, instead of cropping from the quadrants like for the classification version on the small datasets, we instead crop out x00000000 patches taken at random from the segmentation scenes. We optimize model with SGD using a learning rate of 0.007 for 15,000 iterations, with a batch size of 48, Once again, we use the selection heuristic in subsection 4.1 for early-stopping.
15

---

Under review as a conference paper at ICLR 2020
Image
Ground truth label
Baseline prediction
Our prediction
Table 4: Qualitative comparison of segmentation results using our method, the baseline, and the ground truth label.
G QUALITATIVE COMPARISON ON GTAS - CITYSCAPES
We thank the anonymous Reviewer for suggeting shows segmentation results produced before and after adaptation(as shown in Table 4 alongside the original input image and the ground truth label.
add these visualizations. This section
Qualitatively, from visual inspection of these images, we can see that the quality of produced seg- mentations improves drastically after adaptation with our method. In particular, we note that location prediction as self-supervised task helps to prevent nonsensical label configurations, which are of- ten produced by the baseline. For example, the baseline model sometimes predicts "building" or "terrain" for pixels directly in front of car. These emror are largely corrected for by our proposed adaptation method, leading more accurate "road" predictions.
16


Reviews:
[{"review_id": "rkxxcB1Rtr", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper presents a novel unsupervised domain adaptation framework for neural networks. Similarly to existing approaches, it performs adaptation by aligning representations of the source and the target domains. The main difference is that this alignment is achieved not through explicitly minimizing some distribution discrepancy (this usually leads to challenging minimax optimization problems). Instead, the authors propose to use a battery of auxiliary self-supervised learning (SSL) tasks for both domains simultaneously. Each task is meant to align the source and the target representations along a direction of variation relevant to that task. Assuming that the battery is diverse enough, optimizing the representation for all the tasks leads to matching of the distributions. \n\nPros:\n+ The paper is well-written and easy to read.\n+ I like the simplicity of the idea and the fact that it achieves competitive performance without any adversarial learning (which may be very tricky to deal with).\n+ The paper presents a reasonable procedure for hyper-parameter tuning and early stopping which seems to work well in practice.\n\nCons:\n- The paper is purely practical with no theory backing the approach. As a result, the discussion of guarantees and limitations is quite brief.\n- It’s unclear how easy it is to come up with a reasonable set of SSL tasks for a particular pair of domains. It seems that it may become a serious problem when the method is applied to something other than benchmarks. Table 2 reveals that there is no consistent improvement over the existing approaches which suggests that the chosen battery of SSL tasks is not universal (as the authors themselves admit). On a related note, it’s a bit disappointing that the authors mention SVHN results as a failure case but never provide a way to address the issue.\n- It would be nice to some results for the Office dataset for completeness. The authors could use a pre-trained network as a starting points just like it’s done in other papers. According to the last paragraph of Section 6 this experiment should be feasible.\n\nNotes/questions:\n* Table 2, last column: The performance of DIRT-T seems to be better than that of the proposed method and yet the latter is highlighted and not the former.\n\nOverall, I think it’s a good paper presenting a thought-provoking idea. In my opinion, the weakest point of the work is the lack of any (neither principled nor practical) guidance as to how to choose the set of self-supervised tasks. Despite this I feel that this submission should be accepted but at the same time I’m curious to see what the authors have to say regarding the concerns I raised in my review.", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "rklFudcVqB", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper describes an approach to domain adaptation that uses\nself-supervised losses to encourage source/target domain alignment for\nunsupervised domain adaptation. The authors propose to use four\nself-supervised tasks (variants of tasks used in the self-supervised\nrepresentation learning for object recognition literature) that are\nused with a combined loss including unlabeled source and target\ntraining samples. The authors also propose an alignment heuristic for\nguiding early stopping. Experimental results on a standard battery of\ndomain adaptation problems are given, plus some intriguing baseline\nresults for semantic segmentation.\n\nThe paper is written very well and the technical development and\nmotivations for each decision are well discussed and argued.\n\n1. The experimental evaluation is a bit limited as the object\n   recognition datasets are a bit limited. Results on Office or\n   Office-Home would be nice.\n\n2. Using location classification for semantic segmentation seems\n   intuitively to be encouraging the network to learn coarse spatial\n   priors (which should be invariant across the two domains). Have you\n   looked at how alignment is actually happening? More qualitative\n   analysis in this direction would be useful to appreciate the\n   proposed approach.\n\n3. Related to the previous point, it would be interesting to see how\n   semgmentations in the unsupervised domain gradually change and\n   improve with increasing alignment.\n\nIn summary: the ideas are simple, intuitive, and well-explained -- I\nthink the results reported would be easy to reproduce with minimal\nhead scratching. The experiments are interesting and not overstated.\n", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "HkxybvFNqH", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper introduces an unsupervised domain adaptation method that uses self-supervised tasks to bring the two different domains closer together. It runs experiments on some classic benchmarks.\n\nMy score for this paper is weakly rejected because \n\n(1) the concept of self-supervision is not first proposed by this paper. The proposed method is not novel. It introduces three simple self-supervision tasks: flip, rotation and location, and the performance is not better than previous results such as DIRT-T; \n\n(2) there are 7 benchmarks in Table2, but only 2 of 7 has result on R+L+F. In the paper, it mentioned because the result is not better, but the author should still provide them. \n\n(3) it emphasizes the contribution of encouraging more study of self-supervision for unsupervised domain adaptation. It doesn’t provide any way for how to design self-supervision task or whether more tasks is better. I think it is an interesting paper, but not enough as a conference paper, maybe a workshop paper. \n\n(4) there are some classic unsupervised domain adaption benchmarks like Office Dataset, and Bing-Caltech dataset, why not run the method on them?\n\n(5) In ICCV 2019, there is a paper \"S4L: Self-Supervised Semi-Supervised Learning\". The proposed method is almost same. I think the difference is this paper changes the setting and considers the unsupervised data as target domain and supervised data as source domain. ", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}]

Author rebuttal:

Meta-review: Thanks for your detailed replies to the reviewers, which helped us a lot to clarify several issues.
Although the paper discusses an interesting topic and contains potentially interesting idea, its novelty is limited.
Given the high competition of ICLR2020, this paper is still below the bar unfortunately.
Decision: Reject