AdaMatch: A Unified Approach to Semi-Supervised Learning and Domain Adaptation

1 INTRODUCTION . Since the inception of domain adaptation and knowledge transfer , researchers have been well aware of various configurations of labeled or unlabeled data and assumptions on domain shift ( Csurka , 2017 ) . Unsupervised domain adaptation ( UDA ) , semi-supervised learning ( SSL ) , and semi-supervised domain adaptation ( SSDA ) all use different configurations of labeled and unlabeled data , with the major distinction being that , unlike SSL , UDA and SSDA assume a domain shift between the labeled and unlabeled data ( see Table 1 ) . However , currently the fields of SSL and UDA/SSDA are fragmented : different techniques are developed in isolation for each setting , and there are only a handful of algorithms that are evaluated on both ( French et al. , 2018 ) . Techniques that leverage unlabeled data are of utmost importance in practical applications of machine learning because labeling data is expensive . It is also the case that in practice the available unlabeled data will have a distribution shift . Addressing this distribution shift is necessary because neural networks are not robust ( Recht et al. , 2019a ; Biggio & Roli , 2018 ; Szegedy et al. , 2013 ; Hendrycks & Dietterich , 2019 ; Azulay & Weiss , 2018 ; Shankar et al. , 2019 ; Gu et al. , 2019 ; Taori et al. , 2020 ) to even slight differences between the training distribution and test distribution . Although there are techniques to improve out-of-distribution robustness assuming no access to unlabeled data from the target domain ( Hendrycks et al. , 2020 ; Zhang , 2019 ; Engstrom et al. , 2019 ; Geirhos et al. , 2018 ; Yang et al. , 2019 ; Zhang et al. , 2019 ) , it is common in practice to have access to unlabeled data in a shifted domain ( i.e . the UDA or SSDA setting ) , and leveraging this unlabeled data allows for much higher accuracy . Moreover , while SSDA ( Donahue et al. , 2013 ; Yao et al. , 2015 ; Ao et al. , 2017 ; Saito et al. , 2019 ) has received less attention than both SSL and UDA , we believe it describes a realistic scenario in practice and should be equally considered . In this work , we introduce AdaMatch , a unified solution designed to solve the tasks of UDA , SSL , and SSDA using the same set of hyperparameters regardless of the dataset or task . AdaMatch extends FixMatch ( Sohn et al. , 2020 ) by ( 1 ) addressing the distribution shift between source and target domains present in the batch norm statistics , ( 2 ) adjusting the pseudo-label confidence threshold on-the-fly , and ( 3 ) using a modified version of distribution alignment from Berthelot et al . ( 2020 ) . AdaMatch sets a new state-of-the-art accuracy of 28.7 % for UDA without pre-training and 33.4 % with pre-training on DomainNet , an increase of 11.1 % when compared on the same code base . With just one label per class on the target dataset , AdaMatch is more data efficient than other method , achieving a gain of 6.1 % over UDA and 13.6 % with 5 labels . We additionally promote democratic research by reporting results on a smaller 64 × 64 DomainNet . This results in a minimal drop in accuracy compared to the full resolution , and compared to the practice of sub-selecting dataset pairs , does not bias the results towards easier or harder datasets . Finally , we perform an extensive ablation analysis to understand the importance of each improvement and modification that distinguishes AdaMatch from prior semi-supervised learning methods . 2 RELATED WORK . Unsupervised Domain Adaptation ( UDA ) . The UDA research area focuses on studying the performance of models trained on a labeled source domain and an unlabeled target domain from a differing distribution with the goal of obtaining high accuracy on the target domain . Inspired by the theoretical analysis of domain adaptation ( Ben-David et al. , 2010 ) , a major focus of research in UDA has been reducing the discrepancy of representations between domains , so that a classifier that is learned on the source features works well on the target features . UDA methods can be categorized by the technique they use to measure this discrepancy . For example , ( Long et al. , 2013 ; Tzeng et al. , 2014 ; 2015 ) use the maximum mean discrepancy ( Gretton et al. , 2012 ) , ( Sun & Saenko , 2016 ) use correlation alignment across domains , and domain adversarial neural networks ( Ajakan et al. , 2014 ; Ganin et al. , 2016 ; Bousmalis et al. , 2017 ; Saito et al. , 2018 ) measure the domain discrepancy using a discriminator network . Maximum classifier discrepancy ( Saito et al. , 2018 ) ( MCD ) measures the domain discrepancy via multiple task classifiers and is shown to achieve state-of-the-art performance . Semi-Supervised Learning ( SSL ) . In the SSL setting , a portion of the training dataset is labeled and the remaining portion is unlabeled . SSL has seen great progress in recent years , including temporal ensemble ( Laine & Aila , 2017 ) , mean teacher ( Tarvainen & Valpola , 2017 ) , MixMatch ( Berthelot et al. , 2019 ) , ReMixMatch ( Berthelot et al. , 2020 ) , FixMatch ( Sohn et al. , 2020 ) , unsupervised data augmentation ( Xie et al. , 2019 ) , to name a few . While NoisyStudent ( Xie et al. , 2020 ) , an SSL algorithm , uses labeled data from ImageNet and unlabeled data from JFT in training , they do not leverage this distribution shift during training , and they only evaluate on the source domain ( i.e. , ImageNet ) , not the target domain . While there is no technical barrier to applying SSL methods to UDA , only a few SSL methods have been applied to solve UDA problems ; for example , on top of the discrepancy reduction techniques of UDA , several works ( French et al. , 2018 ; Long et al. , 2018 ; Saito et al. , 2019 ; Tran et al. , 2019 ) propose to combine SSL techniques such as entropy minimization ( Grandvalet & Bengio , 2005 ) or pseudo-labeling ( Lee , 2013 ) . Semi-Supervised Domain Adaptation ( SSDA ) has been studied in several settings , including vision ( Donahue et al. , 2013 ; Yao et al. , 2015 ; Ao et al. , 2017 ; Saito et al. , 2019 ) and natural language processing ( Jiang & Zhai , 2007 ; Daumé III et al. , 2010 ; Guo & Xiao , 2012 ) . Since SSDA assumes access to labeled data from multiple domains , early works have used separate models for each domain and regularized them with constraints ( Donahue et al. , 2013 ; Yao et al. , 2015 ; Ao et al. , 2017 ; Daumé III et al. , 2010 ; Guo & Xiao , 2012 ) . However , such methods are difficult to adapt to the UDA setting where labeled data is only available in a single domain . One recent exception is minimax entropy ( MME ) regularization ( Saito et al. , 2019 ) , which can work in both the UDA and SSDA setting . However , unlike AdaMatch , MME requires a pre-trained network to work well . Transfer learning is used to boost accuracy on small datasets by initializing model parameters with pre-trained weights first learned on a separate , larger dataset , which can compensate for the limited amount of labeled source data and boost the overall performance ( Recht et al. , 2019b ; Kolesnikov et al. , 2019 ) . For example , standard experimental protocols on several UDA benchmarks , including Office-31 ( Saenko et al. , 2010 ) , PACS ( Li et al. , 2017 ) , and DomainNet ( Peng et al. , 2019 ) , use ImageNet-pretrained models to initialize model parameters . Though useful for some cases in practice , it may not be the most general protocol to evaluate the advancement of UDA algorithms , especially in situations where no datasets exist for pre-training ( for example , images with arbitrary number of channels ) , or for domains other than vision , where no pre-training datasets exist . Thus , in this work , we mainly focus on evaluating methods under a non-transfer learning setting , which we consider to be more general . Although we also achieve state of the art results with transfer learning , we only present them for historical reasons and to illustrate that our method can use that setting too . 3 ADAMATCH . We now introduce AdaMatch , a new algorithm inspired by modern semi-supervised learning techniques , aimed at solving UDA , SSL , and SSDA . As typical in SSL , AdaMatch takes both an unlabeled and labeled dataset as input . We assume that the labeled data is drawn from a source domain while the unlabeled data is drawn from a target domain ( for the SSL task , these domains are the same ) . Notation . We use capital letters X , Y , Z to denote minibatches of examples , labels and logits . Specifically , XSL ⊂ RnSL×d and YSL ⊂ { 0 , 1 } nSL×k denote the minibatch of source images and labels , respectively . Similarly , the minibatch of unlabeled target images is XTU ⊂ RnTU×d . Here , k is the number of classes and d is the input dimension ( for images d = h · w · c , where h is height , w is width , and c is the number of channels ) . The minibatch size for the labeled data is nSL and the minibatch size of the unlabeled images is nTU . Additionally , we use Y ( i ) to refer to its i-th row , and Y ( i , j ) to refer to the i , j−th element of Y . The model f : Rd → Rk takes images as input and outputs logits for each of k classes . Importantly , the source and target domain are the same classification task , so the number of classes k and the image dimension d is the same for both domains . 3.1 METHOD DESCRIPTION . AdaMatch introduces three new techniques to account for differences between the source and target distributions – random logit interpolation , a relative confidence threshold , and a modified distribution alignment from ReMixMatch ( Berthelot et al. , 2020 ) – but builds upon the algorithmic backbone from FixMatch ( Sohn et al. , 2020 ) . We first provide a high-level overview of the algorithm and then discuss the implementation details of the various components . Overview . A high-level depiction of AdaMatch is in Figure 1 . Two augmentations are made for each image : a weak and a strong one with the intent to make the class prediction harder on the strongly augmented image . Next , we obtain logits by running two batches through the model : a batch of the source images and batch composed of both the source and target images . Each of the resulting batches of logits are influenced by their respective batch norm statistics , i.e . the source batch is only influenced by the source data batch norm statistics while the batch that combines source and target is influenced by both domains batch norm statistics . Two loss terms are then computed : • The source loss term is responsible for predicting correct source labels and for aligning source and target logit domains . We first combine logits for the source images using random logit interpolation , which encourages the model to produce the same label for the hyperspace connecting source logits obtained from 1 ) only source examples and 2 ) a combination of source and target examples . In practice , this creates an implicit constraint to align the source and target domains in logit space . The newly obtained source logits are then used to compute the cross-entropy loss for the source data . • The target loss term is responsible for predicting the correct target labels and for aligning the target predictions to a desired class distribution . Since we don ’ t assume access to labels for the target images , we create a pseudo-label for these images as follows . First , we rectify the class distribution obtained from weakly augmented target images to a desired class distribution using distribution alignment . If the target class distribution is known , it can be used directly . In the general case where it is not known , we use the source class distribution instead . We then select entries of the batch for which the rectified probabilities of the weakly augmented target image predictions are above a user-defined confidence threshold . A pseudo label is then made for these outputs by selecting the most confident class , and these pseudo-labels are used with a standard cross-entropy loss applied to the logits of the strongly augmented images . Augmentation . For a dataset D ∈ { SL , TU } , we augment each image batch XD fed into AdaMatch twice , once with a weak augmentation and once with a strong augmentation , using the same types of weak and strong augmentations as ( Berthelot et al. , 2020 ) . This forms a pair of batches XD , w and XD , s respectively , which we denote together as X aug D = { XD , w , XD , s } . From these pairs of batches , we then compute logits Z ′SL , Z ′′ SL and ZTU as follows : { Z ′SL , ZTU } = f ( { X aug SL , X aug TU } ; θ ) ( 1 ) Z ′′SL = f ( X aug SL ; θ ) ( 2 ) That is , we compute logits by calling the model twice for both the strong and weakly augmented images . The first time we pass both source and target inputs together in the same batch so the batch normalization statistics are shared , and the second time we only pass the source label data . We only update batch normalization statistics when computing { Z ′SL , ZTU } = f ( { X aug SL , X aug TU } ; θ ) to avoid double counting source labeled data . Note that without batch normalization , we would have Z ′SL ≡ Z ′′SL . But batch normalization may make them slightly different . Random logit interpolation . The role of random logit interpolation is to randomly combine the joint batch statistics from the source and target domains with the batch statistics from the source domain , which has the effect of producing batch statistics that are more representative of both domains . More precisely , during training , we obtain logits ZSL by randomly interpolating the logits Z ′SL and Z ′′ SL computed with different batch statistics : ZSL = λ · Z ′SL + ( 1− λ ) · Z ′′SL ( 3 ) where we sample λ ∼ UnSL·k ( 0 , 1 ) . Note that each individual logit gets its own random factor . Our underlying goal here is to minimize the loss for every point between Z ′SL and Z ′′ SL , which can be accomplished by either 1 ) having Z ′SL and Z ′′ SL be equal to each other , or 2 ) having the whole line between Z ′SL and Z ′′ SL be a minima . Rather than picking one of the two ways , we formulate the problem as minimizing the loss for all connecting points , which gives the model the freedom to find the best possible solution . We then use random logit interpolation as a cost effective implementation of this loss formulation . By sampling a random point along the line between Z ′SL and Z ′′ SL at each batch , over time , we implicitly allow the model to minimize the loss for all points on the line . Distribution alignment . Distribution alignment ( Berthelot et al. , 2020 ) can be seen as an additional form of model regularization that helps constrain the distribution of the class predictions to be more aligned with the true distribution . Without it , the classifier could just predict the most prevalent class or exhibit other failure modes . Ideally , if the target label distribution is known , we would use it directly . However , when the target label distribution is unknown , we approximate it using the only available distribution – the source label distribution . A limitation of this approach is that the more the source label distribution differs from the target distribution , the more incorrect the approximation will be , which may cause the model performance to degrade . However , in practice , we find that aligning the target pseudo-labels to match the source label distribution helps significantly . Unlike ReMixMatch ( Berthelot et al. , 2020 ) , we estimate the source label distribution from the output of the model rather than using the true labels . We make this change since the model may not be capable of matching the ground truth source label distribution ( particularly when source accuracy is low ) but matching the source output distribution is a more attainable goal . To implement this , we first extract the logits for weakly and strongly augmented samples from the batch ( by indexing ) : ZSL = { ZSL , w , ZSL , s } ZTU = { ZTU , w , ZTU , s } ( 4 ) Then , we compute pseudo-labels for labeled sources and unlabeled targets : ŶSL , w = softmax ( ZSL , w ) ∈ RnSL·k ŶTU , w = softmax ( ZTU , w ) ∈ RnTU ·k ( 5 ) Using distribution alignment , we rectify the target unlabeled pseudo-labels by multiplying them by the ratio of the expected value of the weakly augmented source labels E [ ŶSL , w ] ∈ Rk to the expected value of the target labels E [ ŶTU , w ] ∈ Rk , obtaining the final pseudo-labels ỸTU , w ∈ RnTU ·k : ỸTU , w = normalize ( ŶTU , w E [ ŶSL , w ] E [ ŶTU , w ] ) ( 6 ) normalize ensures that the distribution still sums to 1 . As could be seen E [ ỸTU , w ] = E [ ŶSL , w ] , which confirms that distribution alignment makes the target pseudo-labels follow the source label distribution . If the target label distribution is known , one can simply replace the term E [ ŶSL , w ] with it in the formula above . Relative confidence threshold . A confidence threshold is typically used to select which predicted labels are confident enough to be used as pseudo-labels ( Lee , 2013 ) . However , since machine learning models are poorly calibrated ( Guo et al. , 2017 ) , especially on out-of-distribution data Ovadia et al . ( 2019 ) , the confidence varies from dataset to dataset depending on the ability of the model to learn its task . To address this issue , we introduce a relative confidence threshold which adjusts a user-provided confidence threshold relative to the confidence level of the classifier on the weakly augmented source data . Specifically , we define the relative confidence threshold cτ as the mean confidence of the top-1 prediction on the weakly augmented source data multiplied by a user provided threshold τ : cτ = τ nSL nSL∑ i=1 max j∈ [ 1 .. k ] ( Ŷ ( i , j ) SL , w ) ( 7 ) We then compute a binary mask ∈ { 0 , 1 } nTU by thresholding the weakly augmented target images with the relative confidence threshold cτ : mask ( i ) = max j∈ [ 1 .. k ] ( Ỹ ( i , j ) TU , w ) ≥ cτ ( 8 ) Loss function . The loss L ( θ ) sums Lsource ( θ ) for the source and Ltarget ( θ ) for the target . Lsource ( θ ) = 1 nSL nSL∑ i=1 H ( Y ( i ) SL , Z ( i ) SL , w ) + 1 nSL nSL∑ i=1 H ( Y ( i ) SL , Z ( i ) SL , s ) ( 9 ) Ltarget ( θ ) = 1 nTU nTU∑ i=1 H ( stop_gradient ( Ỹ ( i ) TU , w ) , Z ( i ) TU , s ) ·mask ( i ) ( 10 ) L ( θ ) = Lsource ( θ ) + µ ( t ) Ltarget ( θ ) ( 11 ) whereH ( p , q ) =− ∑ p ( x ) log q ( x ) is the cross-entropy loss and stop_gradient is a function that prevents gradient from back-propagating on its argument . Prevention of gradient back-propagation on guessed labels is a standard practice in SSL works that favors convergence . µ ( t ) is a warmup function that controls the unlabeled loss weight at every step of the training . In practice we use µ ( t ) = 1/2 − cos ( min ( π , 2πt/T ) ) /2 where T is the total training steps . This particular function smoothly raises from 0 to 1 for the first half of the training and remains at 1 for the second half . As can be noted , Lsource ( θ ) is the cross-entropy loss typically used in fully supervised learning which the nuance that we call it twice : once on weakly augmented samples and once on strongly augmented ones . Similarly , Ltarget ( θ ) is the masked cross-entropy loss , where entries for which the confidence is less than cτ are zeroed . This loss term is exactly the same as in FixMatch ( Sohn et al. , 2020 ) .

Summary: The paper extends FixMatch as a unified approach for semi-Supervised Learning, Unsupervised Domain Adaptation, and Semi-supervised Domain Adaptation.  The proposal learns representations with a cross-entropy loss between labels and logits for a strong and weak augmentation versions of the source data while the target data helps with regularization.  The proposal uses a random mixture the logits of the two augmentations, and normalizes the distribution of the pseudo-labels with the ratio of the expected logits of the augmentations.  The contribution is the mixture of aligning the label distributions by normalizing them through the ratio between the source and target predicted labels, using relative thresholds to compare source and target label distributions, and the random logit interpolation to regularize the representations.  This combinations let the method outperform the reported methods in the three tasks on two datasets.