Unsupervised Domain Adaptation through Self-Supervision

1 INTRODUCTION . Visual distribution shifts are fundamental to our constantly evolving world . We humans face them all the time , e.g . when we navigate a foreign city , read text in a new font , or recognize objects in an environment we have never encountered before . These real-world challenges to the human visual perception have direct parallels in computer vision . Formally , a distribution shift happens when a model is trained on data from one distribution ( source ) , but the goal is to make good predictions on some other distribution ( target ) that shares the label space with the source . Often computational models struggle even for pairs of distributions that humans find intuitively similar . Our paper studies the setting of unsupervised domain adaptation , with labeled data in the source domain , but only unlabeled data in the target domain . The general philosophy of the field is to induce alignment of the source and target domains through some transformation . In the context of deep learning , a convolutional neural network maps images to learned representations in some feature space , so inducing alignment is done by making the distribution shifts small between the source and target in this shared feature space ( Csurka , 2017 ; Wang & Deng , 2018 ; Gopalan et al. , 2011 ) . If , in addition , such representations preserve discriminability on the source domain , then we can learn a good classifier on the source , which now generalizes to the target under the reasonable assumption that the representations of the two domains have the same ground truth . Most existing approaches implement this philosophy of alignment by minimizing a measurement of distributional discrepancy in the feature space ( details in section 2 ) , often some form of maximum mean discrepancy ( MMD ) e.g . Long et al . ( 2017 ) , or a learned discriminator of the source and target as an approximation to the total variation distance e.g . Ganin et al . ( 2016 ) . Both measurements lead to the formulation of the training objective as a minimax optimization problem a.k.a . adversarial learning , which is known to be very difficult to solve . Unless carefully balanced , the push and pull in opposite directions can often cause wild fluctuations in the discrepancy loss and lead to sudden divergence ( details in section 2 ) . Therefore , we propose to avoid minimax optimization altogether through a very different approach . Our main idea is to achieve alignment between the source and target domains by training a model on the same task in both domains simultaneously . Indeed , if we had labels in both domains , we could simply use our original classification task for this . However , since we lack labels in the target domain , we propose to use a self-supervised auxiliary task , which creates its own labels directly from the data ( see section 3 ) . In fact , we can use multiple self-supervised tasks , each one aligning the two domains along a direction of variation relevant to that task . Jointly training all the self-supervised tasks on both domains together with the original task on the source domain produces well-aligned representations as shown in Figure 1 . Like all of deep learning , we can only empirically verify that at least in our experiments , the model does not overfit by internally creating a different decision boundary for each domain along different dimensions , which would then yield bad results . Recent research suggests that stochastic gradient descent is indeed unlikely to find such overfitting solutions with a costly decision boundary of high complexity ( implicit regularization ) , even though the models have enough capacity ( Zhang et al. , 2016a ; Neyshabur et al. , 2017b ; Arora et al. , 2018 ) . The key contribution of our work is to draw a connection between unsupervised domain adaptation and self-supervised learning . While we do not propose any fundamentally new self-supervised tasks , we offer insights in section 3 on how to select the right ones for adaptation , and propose in section 4 a novel training algorithm on those tasks , using batches of samples from both domains . Additionally , we demonstrate that domain alignment could be achieved with a simple and stable algorithm , without the need for adversarial learning . In section 5 , we report state-of-the-art results on several standard benchmarks . 2 RELATED WORK . In this section we provide a brief overview of the two fields that our work would like to bridge . 2.1 UNSUPERVISED DOMAIN ADAPTATION . Methods for unsupervised domain adaptation in computer vision can be divided into three broad classes . The dominant class , which our work belongs to , aims to induce alignment between the source and the target domains in some feature space . This has been done by optimizing for some measurement of distributional discrepancy . One popular measurement is the maximum mean discrepancy ( MMD ) – the distance between the mean of the two domains in some reproducing kernel Hilbert space , where the kernel is chosen to maximize the distance ( Bousmalis et al. , 2016 ; Long et al. , 2015 ; 2017 ) . Another way to obtain a measurement of discrepancy is to train an adversarial discriminator that distinguishes between the two domains ( Ganin & Lempitsky , 2014 ; Ganin et al. , 2016 ; Tzeng et al. , 2017 ) . However , both MMD and adversarial training are formulated as minimax optimization problems , which are widely known , both in theory and practice , to be very difficult ( Fedus et al. , 2017 ; Duchi et al. , 2016 ; Liang , 2017 ; Jin et al. , 2019 ) . Since the optimization landscape is much more complex than in standard supervised learning , training often does not converge or converges to a bad local minimum ( Goodfellow , 2016 ; Nagarajan & Kolter , 2017 ; Li et al. , 2017 ) , and requires carefully balancing the two sets of parameters ( for minimization and maximization ) so one does not dominate the other ( Salimans et al. , 2016 ; Neyshabur et al. , 2017a ) . To make minimax optimization easier , researchers have proposed numerous modifications to the loss function , network design , and training procedure ( Arjovsky et al. , 2017 ; Gulrajani et al. , 2017 ; Karras et al. , 2017 ; Courty et al. , 2017 ; Sun & Saenko , 2016 ; Shu et al. , 2018 ; Sener et al. , 2016 ) . Over the years , these modifications have yielded practical improvements on many standard benchmarks , but have also made the state-of-the-art algorithms very complicated . Often practitioners are not sure which tricks are necessary for which applications , and implementing these tricks can be bug-prone and frustrating . To make matters worse , since there is no labeled target data available for a validation set , practitioners have no way to perform hyper-parameter tuning or early stopping . The second class of methods directly transforms the source images to resemble the target images with generative models ( Taigman et al. , 2016 ; Hoffman et al. , 2017 ; Bousmalis et al. , 2017 ) . While similar to the first class in the philosophy of alignment , these methods operate on image pixels directly instead of an intermediate representation space , and therefore can benefit from an additional round of adaptation in some representation space . In subsection 5.2 we demonstrate that composing our method with a popular pixel-level method yields stronger performance than either alone . The third class of methods uses a model trained on the labeled source data to estimate labels on the target data , then trains on some of those estimated pseudo-labels ( e.g . the most confident ones ) , therefore bootstrapping through the unlabeled target data . Sometimes called self-ensembling ( French et al. , 2017 ) , this technique is borrowed from semi-supervised learning , where it is called co-training ( Saito et al. , 2017 ; Zou et al. , 2018 ; Chen et al. , 2018 ; 2011 ) . In contrast , our method uses joint training ( of the main and self-supervised tasks ) , different from co-training in every aspect except the name . 2.2 SELF-SUPERVISED FEATURE LEARNING . The emerging field of self-supervised learning uses the machinery of supervised learning on problems where external supervision is not available . The idea is to use data itself as supervision for auxiliary ( also called “ pretext ” ) tasks that learn deep feature representations which will hopefully be informative for downstream “ real ” tasks . Many such auxiliary tasks have been proposed in the literature , including colorization ( predicting the chrominance channels of an image given its luminance ) ( Zhang et al. , 2016b ; Larsson et al. , 2017 ; Zhang et al. , 2017 ) , image inpainting Pathak et al . ( 2016 ) , spatial context prediction ( Doersch et al. , 2015 ) , solving jigsaw puzzles ( Noroozi & Favaro , 2016 ) , image rotation prediction ( Gidaris et al. , 2018 ) , predicting audio from video ( Owens et al. , 2016 ) , contrastive predictive coding ( Oord et al. , 2018 ) , etc . Researchers have also experimented with self-supervision on videos ( Wang & Gupta , 2015 ; Wang et al. , 2019 ) , and combining multiple self-supervised tasks together ( Doersch & Zisserman , 2017 ) . Typically , self-supervision is used as a pre-training step on unlabeled data ( e.g . the ImageNet training set without labels ) to initialize a deep learning model , followed by fine-tuning on a labeled training set ( e.g . PASCAL VOC ) and evaluating on the corresponding test set . Instead , in this paper , we train the self-supervised tasks together with the main supervised task , encouraging a consistent representation that both aligns the two domains and does well on the main task1 . Recently , self-supervision has also been used for other problem settings , such as improving robustness ( Hendrycks et al. , 2019 ) , domain generalization ( Carlucci et al. , 2019 ) and few-short learning ( Su et al. , 2019 ) . The most relevant paper to us is Ghifary et al . ( 2016 ) , which uses self-supervision for unsupervised domain adaptation , but not through alignment . Their algorithm trains a denoising autoencoder Vincent et al . ( 2008 ) only on the target data , together with the main classifier only on the labeled source data . They argue theoretically that this is better than training the autoencoder on both domains together . However , their theory is based on the critical assumption that the domains are already aligned , which is rarely true in practice . Consequently , their empirical results are much weaker than ours , as discussed in section 5 . Please see Appendix A for more detailed comparisons with these works . 1Note that it is possible to apply the standard pre-training followed by fine-tuning regime to unsupervised domain adaptation , doing self-supervised pre-training on the target domain ( or both the source and the target ) and then fine-tuning on the source . However , this gives almost no benefit over no-adaptation baseline , and is far from being competitive . 3 DESIGNING SELF-SUPERVISED TASKS FOR ADAPTATION . The design of auxiliary self-supervised tasks is an exciting area of research in itself , with many successful examples listed in section 2 . However , not all of them are suitable for unsupervised domain adaptation . In order to induce alignment between the source and target , the labels created by selfsupervision should not require capturing information on the very factors where the domains are meaninglessly different , i.e . the factors of variation that we are trying to eliminate through adaptation . A particularly unsuitable tasks are these that try to predict pixels of the original image , as image inpainting ( Pathak et al. , 2016 ) , colorization ( Zhang et al. , 2016b ; Larsson et al. , 2017 ; Zhang et al. , 2017 ) or denoising autoencoder ( Vincent et al. , 2008 ) . The success of pixel-wise reconstruction depends strongly on brightness information , or other factors of variation in overall appearance ( e.g . sunny vs. coudy ) that are typically irrelevant to high-level visual concepts . Thus , instead of inducing alignment , learning a pixel reconstruction task would instead serve to further separate the domains . We have experimented with using the colorization task and the denoising autoencoder for our training algorithm , and found their performance little better than the source only baseline , sometimes even worse ! 2 In general , classification tasks that predict structural labels seems better suited for our purpose than reconstruction tasks that predict pixels . Therefore , we have settled on three classification-based self-supervised tasks that combine simplicity with high performance : Rotation Prediction : ( Gidaris et al. , 2018 ) An input image is rotated in 90-degree increments i.e . 0° , 90° , 180° , and 270° ; the task is to predict the angle of rotation as a four-way classification problem . Flip Prediction : An input image is randomly flipped vertically ; the task is to predict whether the image is flipped or not 3 . Patch Location Prediction : Patches are randomly cropped out of an input image ; the task is to predict where the patches come from 4 . Consider a trivial illustrative example where the source and target are exactly the same except that the target pixels are all scaled down by a constant factor ( e.g . daylight to dusk transition ) . All three of the aforementioned forms of self-supervision are suitable for this example , because pixel scaling i.e . brightness is “ orthogonal ” to the prediction of rotation , flip and location .

Summary: This paper presents a novel unsupervised domain adaptation framework for neural networks. Similarly to existing approaches, it performs adaptation by aligning representations of the source and the target domains. The main difference is that this alignment is achieved not through explicitly minimizing some distribution discrepancy (this usually leads to challenging minimax optimization problems). Instead, the authors propose to use a battery of auxiliary self-supervised learning (SSL) tasks for both domains simultaneously. Each task is meant to align the source and the target representations along a direction of variation relevant to that task. Assuming that the battery is diverse enough, optimizing the representation for all the tasks leads to matching of the distributions.