AdaMatch: A Unified Approach to Semi-Supervised Learning and Domain Adaptation

1 INTRODUCTION . Since the inception of domain adaptation and knowledge transfer , researchers have been well aware of various configurations of labeled or unlabeled data and assumptions on domain shift ( Csurka , 2017 ) . Unsupervised domain adaptation ( UDA ) , semi-supervised learning ( SSL ) , and semi-supervised domain adaptation ( SSDA ) all use different configurations of labeled and unlabeled data , with the major distinction being that , unlike SSL , UDA and SSDA assume a domain shift between the labeled and unlabeled data ( see Table 1 ) . However , currently the fields of SSL and UDA/SSDA are fragmented : different techniques are developed in isolation for each setting , and there are only a handful of algorithms that are evaluated on both ( French et al. , 2018 ) . Techniques that leverage unlabeled data are of utmost importance in practical applications of machine learning because labeling data is expensive . It is also the case that in practice the available unlabeled data will have a distribution shift . Addressing this distribution shift is necessary because neural networks are not robust ( Recht et al. , 2019a ; Biggio & Roli , 2018 ; Szegedy et al. , 2013 ; Hendrycks & Dietterich , 2019 ; Azulay & Weiss , 2018 ; Shankar et al. , 2019 ; Gu et al. , 2019 ; Taori et al. , 2020 ) to even slight differences between the training distribution and test distribution . Although there are techniques to improve out-of-distribution robustness assuming no access to unlabeled data from the target domain ( Hendrycks et al. , 2020 ; Zhang , 2019 ; Engstrom et al. , 2019 ; Geirhos et al. , 2018 ; Yang et al. , 2019 ; Zhang et al. , 2019 ) , it is common in practice to have access to unlabeled data in a shifted domain ( i.e . the UDA or SSDA setting ) , and leveraging this unlabeled data allows for much higher accuracy . Moreover , while SSDA ( Donahue et al. , 2013 ; Yao et al. , 2015 ; Ao et al. , 2017 ; Saito et al. , 2019 ) has received less attention than both SSL and UDA , we believe it describes a realistic scenario in practice and should be equally considered . In this work , we introduce AdaMatch , a unified solution designed to solve the tasks of UDA , SSL , and SSDA using the same set of hyperparameters regardless of the dataset or task . AdaMatch extends FixMatch ( Sohn et al. , 2020 ) by ( 1 ) addressing the distribution shift between source and target domains present in the batch norm statistics , ( 2 ) adjusting the pseudo-label confidence threshold on-the-fly , and ( 3 ) using a modified version of distribution alignment from Berthelot et al . ( 2020 ) . AdaMatch sets a new state-of-the-art accuracy of 28.7 % for UDA without pre-training and 33.4 % with pre-training on DomainNet , an increase of 11.1 % when compared on the same code base . With just one label per class on the target dataset , AdaMatch is more data efficient than other method , achieving a gain of 6.1 % over UDA and 13.6 % with 5 labels . We additionally promote democratic research by reporting results on a smaller 64 × 64 DomainNet . This results in a minimal drop in accuracy compared to the full resolution , and compared to the practice of sub-selecting dataset pairs , does not bias the results towards easier or harder datasets . Finally , we perform an extensive ablation analysis to understand the importance of each improvement and modification that distinguishes AdaMatch from prior semi-supervised learning methods . 2 RELATED WORK . Unsupervised Domain Adaptation ( UDA ) . The UDA research area focuses on studying the performance of models trained on a labeled source domain and an unlabeled target domain from a differing distribution with the goal of obtaining high accuracy on the target domain . Inspired by the theoretical analysis of domain adaptation ( Ben-David et al. , 2010 ) , a major focus of research in UDA has been reducing the discrepancy of representations between domains , so that a classifier that is learned on the source features works well on the target features . UDA methods can be categorized by the technique they use to measure this discrepancy . For example , ( Long et al. , 2013 ; Tzeng et al. , 2014 ; 2015 ) use the maximum mean discrepancy ( Gretton et al. , 2012 ) , ( Sun & Saenko , 2016 ) use correlation alignment across domains , and domain adversarial neural networks ( Ajakan et al. , 2014 ; Ganin et al. , 2016 ; Bousmalis et al. , 2017 ; Saito et al. , 2018 ) measure the domain discrepancy using a discriminator network . Maximum classifier discrepancy ( Saito et al. , 2018 ) ( MCD ) measures the domain discrepancy via multiple task classifiers and is shown to achieve state-of-the-art performance . Semi-Supervised Learning ( SSL ) . In the SSL setting , a portion of the training dataset is labeled and the remaining portion is unlabeled . SSL has seen great progress in recent years , including temporal ensemble ( Laine & Aila , 2017 ) , mean teacher ( Tarvainen & Valpola , 2017 ) , MixMatch ( Berthelot et al. , 2019 ) , ReMixMatch ( Berthelot et al. , 2020 ) , FixMatch ( Sohn et al. , 2020 ) , unsupervised data augmentation ( Xie et al. , 2019 ) , to name a few . While NoisyStudent ( Xie et al. , 2020 ) , an SSL algorithm , uses labeled data from ImageNet and unlabeled data from JFT in training , they do not leverage this distribution shift during training , and they only evaluate on the source domain ( i.e. , ImageNet ) , not the target domain . While there is no technical barrier to applying SSL methods to UDA , only a few SSL methods have been applied to solve UDA problems ; for example , on top of the discrepancy reduction techniques of UDA , several works ( French et al. , 2018 ; Long et al. , 2018 ; Saito et al. , 2019 ; Tran et al. , 2019 ) propose to combine SSL techniques such as entropy minimization ( Grandvalet & Bengio , 2005 ) or pseudo-labeling ( Lee , 2013 ) . Semi-Supervised Domain Adaptation ( SSDA ) has been studied in several settings , including vision ( Donahue et al. , 2013 ; Yao et al. , 2015 ; Ao et al. , 2017 ; Saito et al. , 2019 ) and natural language processing ( Jiang & Zhai , 2007 ; Daumé III et al. , 2010 ; Guo & Xiao , 2012 ) . Since SSDA assumes access to labeled data from multiple domains , early works have used separate models for each domain and regularized them with constraints ( Donahue et al. , 2013 ; Yao et al. , 2015 ; Ao et al. , 2017 ; Daumé III et al. , 2010 ; Guo & Xiao , 2012 ) . However , such methods are difficult to adapt to the UDA setting where labeled data is only available in a single domain . One recent exception is minimax entropy ( MME ) regularization ( Saito et al. , 2019 ) , which can work in both the UDA and SSDA setting . However , unlike AdaMatch , MME requires a pre-trained network to work well . Transfer learning is used to boost accuracy on small datasets by initializing model parameters with pre-trained weights first learned on a separate , larger dataset , which can compensate for the limited amount of labeled source data and boost the overall performance ( Recht et al. , 2019b ; Kolesnikov et al. , 2019 ) . For example , standard experimental protocols on several UDA benchmarks , including Office-31 ( Saenko et al. , 2010 ) , PACS ( Li et al. , 2017 ) , and DomainNet ( Peng et al. , 2019 ) , use ImageNet-pretrained models to initialize model parameters . Though useful for some cases in practice , it may not be the most general protocol to evaluate the advancement of UDA algorithms , especially in situations where no datasets exist for pre-training ( for example , images with arbitrary number of channels ) , or for domains other than vision , where no pre-training datasets exist . Thus , in this work , we mainly focus on evaluating methods under a non-transfer learning setting , which we consider to be more general . Although we also achieve state of the art results with transfer learning , we only present them for historical reasons and to illustrate that our method can use that setting too . 3 ADAMATCH . We now introduce AdaMatch , a new algorithm inspired by modern semi-supervised learning techniques , aimed at solving UDA , SSL , and SSDA . As typical in SSL , AdaMatch takes both an unlabeled and labeled dataset as input . We assume that the labeled data is drawn from a source domain while the unlabeled data is drawn from a target domain ( for the SSL task , these domains are the same ) . Notation . We use capital letters X , Y , Z to denote minibatches of examples , labels and logits . Specifically , XSL ⊂ RnSL×d and YSL ⊂ { 0 , 1 } nSL×k denote the minibatch of source images and labels , respectively . Similarly , the minibatch of unlabeled target images is XTU ⊂ RnTU×d . Here , k is the number of classes and d is the input dimension ( for images d = h · w · c , where h is height , w is width , and c is the number of channels ) . The minibatch size for the labeled data is nSL and the minibatch size of the unlabeled images is nTU . Additionally , we use Y ( i ) to refer to its i-th row , and Y ( i , j ) to refer to the i , j−th element of Y . The model f : Rd → Rk takes images as input and outputs logits for each of k classes . Importantly , the source and target domain are the same classification task , so the number of classes k and the image dimension d is the same for both domains . 3.1 METHOD DESCRIPTION . AdaMatch introduces three new techniques to account for differences between the source and target distributions – random logit interpolation , a relative confidence threshold , and a modified distribution alignment from ReMixMatch ( Berthelot et al. , 2020 ) – but builds upon the algorithmic backbone from FixMatch ( Sohn et al. , 2020 ) . We first provide a high-level overview of the algorithm and then discuss the implementation details of the various components . Overview . A high-level depiction of AdaMatch is in Figure 1 . Two augmentations are made for each image : a weak and a strong one with the intent to make the class prediction harder on the strongly augmented image . Next , we obtain logits by running two batches through the model : a batch of the source images and batch composed of both the source and target images . Each of the resulting batches of logits are influenced by their respective batch norm statistics , i.e . the source batch is only influenced by the source data batch norm statistics while the batch that combines source and target is influenced by both domains batch norm statistics . Two loss terms are then computed : • The source loss term is responsible for predicting correct source labels and for aligning source and target logit domains . We first combine logits for the source images using random logit interpolation , which encourages the model to produce the same label for the hyperspace connecting source logits obtained from 1 ) only source examples and 2 ) a combination of source and target examples . In practice , this creates an implicit constraint to align the source and target domains in logit space . The newly obtained source logits are then used to compute the cross-entropy loss for the source data . • The target loss term is responsible for predicting the correct target labels and for aligning the target predictions to a desired class distribution . Since we don ’ t assume access to labels for the target images , we create a pseudo-label for these images as follows . First , we rectify the class distribution obtained from weakly augmented target images to a desired class distribution using distribution alignment . If the target class distribution is known , it can be used directly . In the general case where it is not known , we use the source class distribution instead . We then select entries of the batch for which the rectified probabilities of the weakly augmented target image predictions are above a user-defined confidence threshold . A pseudo label is then made for these outputs by selecting the most confident class , and these pseudo-labels are used with a standard cross-entropy loss applied to the logits of the strongly augmented images . Augmentation . For a dataset D ∈ { SL , TU } , we augment each image batch XD fed into AdaMatch twice , once with a weak augmentation and once with a strong augmentation , using the same types of weak and strong augmentations as ( Berthelot et al. , 2020 ) . This forms a pair of batches XD , w and XD , s respectively , which we denote together as X aug D = { XD , w , XD , s } . From these pairs of batches , we then compute logits Z ′SL , Z ′′ SL and ZTU as follows : { Z ′SL , ZTU } = f ( { X aug SL , X aug TU } ; θ ) ( 1 ) Z ′′SL = f ( X aug SL ; θ ) ( 2 ) That is , we compute logits by calling the model twice for both the strong and weakly augmented images . The first time we pass both source and target inputs together in the same batch so the batch normalization statistics are shared , and the second time we only pass the source label data . We only update batch normalization statistics when computing { Z ′SL , ZTU } = f ( { X aug SL , X aug TU } ; θ ) to avoid double counting source labeled data . Note that without batch normalization , we would have Z ′SL ≡ Z ′′SL . But batch normalization may make them slightly different . Random logit interpolation . The role of random logit interpolation is to randomly combine the joint batch statistics from the source and target domains with the batch statistics from the source domain , which has the effect of producing batch statistics that are more representative of both domains . More precisely , during training , we obtain logits ZSL by randomly interpolating the logits Z ′SL and Z ′′ SL computed with different batch statistics : ZSL = λ · Z ′SL + ( 1− λ ) · Z ′′SL ( 3 ) where we sample λ ∼ UnSL·k ( 0 , 1 ) . Note that each individual logit gets its own random factor . Our underlying goal here is to minimize the loss for every point between Z ′SL and Z ′′ SL , which can be accomplished by either 1 ) having Z ′SL and Z ′′ SL be equal to each other , or 2 ) having the whole line between Z ′SL and Z ′′ SL be a minima . Rather than picking one of the two ways , we formulate the problem as minimizing the loss for all connecting points , which gives the model the freedom to find the best possible solution . We then use random logit interpolation as a cost effective implementation of this loss formulation . By sampling a random point along the line between Z ′SL and Z ′′ SL at each batch , over time , we implicitly allow the model to minimize the loss for all points on the line . Distribution alignment . Distribution alignment ( Berthelot et al. , 2020 ) can be seen as an additional form of model regularization that helps constrain the distribution of the class predictions to be more aligned with the true distribution . Without it , the classifier could just predict the most prevalent class or exhibit other failure modes . Ideally , if the target label distribution is known , we would use it directly . However , when the target label distribution is unknown , we approximate it using the only available distribution – the source label distribution . A limitation of this approach is that the more the source label distribution differs from the target distribution , the more incorrect the approximation will be , which may cause the model performance to degrade . However , in practice , we find that aligning the target pseudo-labels to match the source label distribution helps significantly . Unlike ReMixMatch ( Berthelot et al. , 2020 ) , we estimate the source label distribution from the output of the model rather than using the true labels . We make this change since the model may not be capable of matching the ground truth source label distribution ( particularly when source accuracy is low ) but matching the source output distribution is a more attainable goal . To implement this , we first extract the logits for weakly and strongly augmented samples from the batch ( by indexing ) : ZSL = { ZSL , w , ZSL , s } ZTU = { ZTU , w , ZTU , s } ( 4 ) Then , we compute pseudo-labels for labeled sources and unlabeled targets : ŶSL , w = softmax ( ZSL , w ) ∈ RnSL·k ŶTU , w = softmax ( ZTU , w ) ∈ RnTU ·k ( 5 ) Using distribution alignment , we rectify the target unlabeled pseudo-labels by multiplying them by the ratio of the expected value of the weakly augmented source labels E [ ŶSL , w ] ∈ Rk to the expected value of the target labels E [ ŶTU , w ] ∈ Rk , obtaining the final pseudo-labels ỸTU , w ∈ RnTU ·k : ỸTU , w = normalize ( ŶTU , w E [ ŶSL , w ] E [ ŶTU , w ] ) ( 6 ) normalize ensures that the distribution still sums to 1 . As could be seen E [ ỸTU , w ] = E [ ŶSL , w ] , which confirms that distribution alignment makes the target pseudo-labels follow the source label distribution . If the target label distribution is known , one can simply replace the term E [ ŶSL , w ] with it in the formula above . Relative confidence threshold . A confidence threshold is typically used to select which predicted labels are confident enough to be used as pseudo-labels ( Lee , 2013 ) . However , since machine learning models are poorly calibrated ( Guo et al. , 2017 ) , especially on out-of-distribution data Ovadia et al . ( 2019 ) , the confidence varies from dataset to dataset depending on the ability of the model to learn its task . To address this issue , we introduce a relative confidence threshold which adjusts a user-provided confidence threshold relative to the confidence level of the classifier on the weakly augmented source data . Specifically , we define the relative confidence threshold cτ as the mean confidence of the top-1 prediction on the weakly augmented source data multiplied by a user provided threshold τ : cτ = τ nSL nSL∑ i=1 max j∈ [ 1 .. k ] ( Ŷ ( i , j ) SL , w ) ( 7 ) We then compute a binary mask ∈ { 0 , 1 } nTU by thresholding the weakly augmented target images with the relative confidence threshold cτ : mask ( i ) = max j∈ [ 1 .. k ] ( Ỹ ( i , j ) TU , w ) ≥ cτ ( 8 ) Loss function . The loss L ( θ ) sums Lsource ( θ ) for the source and Ltarget ( θ ) for the target . Lsource ( θ ) = 1 nSL nSL∑ i=1 H ( Y ( i ) SL , Z ( i ) SL , w ) + 1 nSL nSL∑ i=1 H ( Y ( i ) SL , Z ( i ) SL , s ) ( 9 ) Ltarget ( θ ) = 1 nTU nTU∑ i=1 H ( stop_gradient ( Ỹ ( i ) TU , w ) , Z ( i ) TU , s ) ·mask ( i ) ( 10 ) L ( θ ) = Lsource ( θ ) + µ ( t ) Ltarget ( θ ) ( 11 ) whereH ( p , q ) =− ∑ p ( x ) log q ( x ) is the cross-entropy loss and stop_gradient is a function that prevents gradient from back-propagating on its argument . Prevention of gradient back-propagation on guessed labels is a standard practice in SSL works that favors convergence . µ ( t ) is a warmup function that controls the unlabeled loss weight at every step of the training . In practice we use µ ( t ) = 1/2 − cos ( min ( π , 2πt/T ) ) /2 where T is the total training steps . This particular function smoothly raises from 0 to 1 for the first half of the training and remains at 1 for the second half . As can be noted , Lsource ( θ ) is the cross-entropy loss typically used in fully supervised learning which the nuance that we call it twice : once on weakly augmented samples and once on strongly augmented ones . Similarly , Ltarget ( θ ) is the masked cross-entropy loss , where entries for which the confidence is less than cτ are zeroed . This loss term is exactly the same as in FixMatch ( Sohn et al. , 2020 ) .

Summary: This paper presents a unified solution for unsupervised domain adaptation (UDA), semi-supervised learning (SSL), and semi-supervised domain adaptation (SSDA). The authors extend the state-of-the-art SSL method, that is FixMatch, to make it capable of handling DA setting where target data stem from a different distribution with source data. Experimental results show that the proposed method performs on par or significantly better than respective state-of-the-art methods in UDA, SSL, and SSDA.