Year: 2021

Abstract:
We extend semi-supervised learning to the problem of domain adaptation to learn significantly higher-accuracy models that train on one data distribution and test on a different one. With the goal of generality, we introduce AdaMatch, a unified solution for unsupervised domain adaptation (UDA), semi-supervised learning (SSL), and semi-supervised domain adaptation (SSDA). In an extensive experimental study, we compare its behavior with respective state-of-the-art techniques from SSL, SSDA, and UDA and find that AdaMatch either matches or significantly exceeds the state-of-the-art in each case using the same hyper-parameters regardless of the dataset or task. For example, AdaMatch nearly doubles the accuracy compared to that of the prior state-of-the-art on the UDA task for DomainNet and even exceeds the accuracy of the prior state-of-the-art obtained with pre-training by 6.4% when AdaMatch is trained completely from scratch. Furthermore, by providing AdaMatch with just one labeled example per class from the target domain (i.e., the SSDA setting), we increase the target accuracy by an additional 6.1%, and with 5 labeled examples, by 13.6%.1

heading: 1 INTRODUCTION
text: Since the inception of domain adaptation and knowledge transfer, researchers have been well aware of various configurations of labeled or unlabeled data and assumptions on domain shift (Csurka, 2017). Unsupervised domain adaptation (UDA), semi-supervised learning (SSL), and semi-supervised domain adaptation (SSDA) all use different configurations of labeled and unlabeled data, with the major distinction being that, unlike SSL, UDA and SSDA assume a domain shift between the labeled and unlabeled data (see Table 1). However, currently the fields of SSL and UDA/SSDA are fragmented: different techniques are developed in isolation for each setting, and there are only a handful of algorithms that are evaluated on both (French et al., 2018).
Techniques that leverage unlabeled data are of utmost importance in practical applications of machine learning because labeling data is expensive. It is also the case that in practice the available unlabeled data will have a distribution shift. Addressing this distribution shift is necessary because neural networks are not robust (Recht et al., 2019a; Biggio & Roli, 2018; Szegedy et al., 2013; Hendrycks & Dietterich, 2019; Azulay & Weiss, 2018; Shankar et al., 2019; Gu et al., 2019; Taori et al., 2020) to even slight differences between the training distribution and test distribution. Although there are techniques to improve out-of-distribution robustness assuming no access to unlabeled data from the target domain (Hendrycks et al., 2020; Zhang, 2019; Engstrom et al., 2019; Geirhos et al., 2018; Yang et al., 2019; Zhang et al., 2019), it is common in practice to have access to unlabeled data in a shifted domain (i.e. the UDA or SSDA setting), and leveraging this unlabeled data allows for much higher accuracy. Moreover, while SSDA (Donahue et al., 2013; Yao et al., 2015; Ao et al., 2017; Saito et al., 2019) has received less attention than both SSL and UDA, we believe it describes a realistic scenario in practice and should be equally considered.
In this work, we introduce AdaMatch, a unified solution designed to solve the tasks of UDA, SSL, and SSDA using the same set of hyperparameters regardless of the dataset or task. AdaMatch extends FixMatch (Sohn et al., 2020) by (1) addressing the distribution shift between source and target domains present in the batch norm statistics, (2) adjusting the pseudo-label confidence threshold on-the-fly, and (3) using a modified version of distribution alignment from Berthelot et al. (2020).
AdaMatch sets a new state-of-the-art accuracy of 28.7% for UDA without pre-training and 33.4% with pre-training on DomainNet, an increase of 11.1% when compared on the same code base. With just one label per class on the target dataset, AdaMatch is more data efficient than other method, achieving a gain of 6.1% over UDA and 13.6% with 5 labels. We additionally promote democratic research by reporting results on a smaller 64 × 64 DomainNet. This results in a minimal drop in accuracy compared to the full resolution, and compared to the practice of sub-selecting dataset pairs, does not bias the results towards easier or harder datasets. Finally, we perform an extensive ablation analysis to understand the importance of each improvement and modification that distinguishes AdaMatch from prior semi-supervised learning methods.

heading: 2 RELATED WORK
text: Unsupervised Domain Adaptation (UDA). The UDA research area focuses on studying the performance of models trained on a labeled source domain and an unlabeled target domain from a differing distribution with the goal of obtaining high accuracy on the target domain. Inspired by the theoretical analysis of domain adaptation (Ben-David et al., 2010), a major focus of research in UDA has been reducing the discrepancy of representations between domains, so that a classifier that is learned on the source features works well on the target features. UDA methods can be categorized by the technique they use to measure this discrepancy. For example, (Long et al., 2013; Tzeng et al., 2014; 2015) use the maximum mean discrepancy (Gretton et al., 2012), (Sun & Saenko, 2016) use correlation alignment across domains, and domain adversarial neural networks (Ajakan et al., 2014; Ganin et al., 2016; Bousmalis et al., 2017; Saito et al., 2018) measure the domain discrepancy using a discriminator network. Maximum classifier discrepancy (Saito et al., 2018) (MCD) measures the domain discrepancy via multiple task classifiers and is shown to achieve state-of-the-art performance.
Semi-Supervised Learning (SSL). In the SSL setting, a portion of the training dataset is labeled and the remaining portion is unlabeled. SSL has seen great progress in recent years, including temporal ensemble (Laine & Aila, 2017), mean teacher (Tarvainen & Valpola, 2017), MixMatch (Berthelot et al., 2019), ReMixMatch (Berthelot et al., 2020), FixMatch (Sohn et al., 2020), unsupervised data augmentation (Xie et al., 2019), to name a few. While NoisyStudent (Xie et al., 2020), an SSL algorithm, uses labeled data from ImageNet and unlabeled data from JFT in training, they do not leverage this distribution shift during training, and they only evaluate on the source domain (i.e., ImageNet), not the target domain. While there is no technical barrier to applying SSL methods to UDA, only a few SSL methods have been applied to solve UDA problems; for example, on top of the discrepancy reduction techniques of UDA, several works (French et al., 2018; Long et al., 2018; Saito et al., 2019; Tran et al., 2019) propose to combine SSL techniques such as entropy minimization (Grandvalet & Bengio, 2005) or pseudo-labeling (Lee, 2013).
Semi-Supervised Domain Adaptation (SSDA) has been studied in several settings, including vision (Donahue et al., 2013; Yao et al., 2015; Ao et al., 2017; Saito et al., 2019) and natural language processing (Jiang & Zhai, 2007; Daumé III et al., 2010; Guo & Xiao, 2012). Since SSDA assumes access to labeled data from multiple domains, early works have used separate models for each domain and regularized them with constraints (Donahue et al., 2013; Yao et al., 2015; Ao et al., 2017; Daumé III et al., 2010; Guo & Xiao, 2012). However, such methods are difficult to adapt to the UDA setting where labeled data is only available in a single domain. One recent exception is minimax entropy (MME) regularization (Saito et al., 2019), which can work in both the UDA and SSDA setting. However, unlike AdaMatch, MME requires a pre-trained network to work well.
Transfer learning is used to boost accuracy on small datasets by initializing model parameters with pre-trained weights first learned on a separate, larger dataset, which can compensate for the limited amount of labeled source data and boost the overall performance (Recht et al., 2019b; Kolesnikov et al., 2019). For example, standard experimental protocols on several UDA benchmarks, including Office-31 (Saenko et al., 2010), PACS (Li et al., 2017), and DomainNet (Peng et al., 2019), use ImageNet-pretrained models to initialize model parameters. Though useful for some cases in practice, it may not be the most general protocol to evaluate the advancement of UDA algorithms, especially
in situations where no datasets exist for pre-training (for example, images with arbitrary number of channels), or for domains other than vision, where no pre-training datasets exist. Thus, in this work, we mainly focus on evaluating methods under a non-transfer learning setting, which we consider to be more general. Although we also achieve state of the art results with transfer learning, we only present them for historical reasons and to illustrate that our method can use that setting too.

heading: 3 ADAMATCH
text: We now introduce AdaMatch, a new algorithm inspired by modern semi-supervised learning techniques, aimed at solving UDA, SSL, and SSDA. As typical in SSL, AdaMatch takes both an unlabeled and labeled dataset as input. We assume that the labeled data is drawn from a source domain while the unlabeled data is drawn from a target domain (for the SSL task, these domains are the same).
Notation. We use capital letters X , Y , Z to denote minibatches of examples, labels and logits. Specifically, XSL ⊂ RnSL×d and YSL ⊂ {0, 1}nSL×k denote the minibatch of source images and labels, respectively. Similarly, the minibatch of unlabeled target images is XTU ⊂ RnTU×d. Here, k is the number of classes and d is the input dimension (for images d = h · w · c, where h is height, w is width, and c is the number of channels). The minibatch size for the labeled data is nSL and the minibatch size of the unlabeled images is nTU . Additionally, we use Y (i) to refer to its i-th row, and Y (i,j) to refer to the i, j−th element of Y . The model f : Rd → Rk takes images as input and outputs logits for each of k classes. Importantly, the source and target domain are the same classification task, so the number of classes k and the image dimension d is the same for both domains.

heading: 3.1 METHOD DESCRIPTION
text: AdaMatch introduces three new techniques to account for differences between the source and target distributions – random logit interpolation, a relative confidence threshold, and a modified distribution alignment from ReMixMatch (Berthelot et al., 2020) – but builds upon the algorithmic backbone from FixMatch (Sohn et al., 2020). We first provide a high-level overview of the algorithm and then discuss the implementation details of the various components.
Overview. A high-level depiction of AdaMatch is in Figure 1. Two augmentations are made for each image: a weak and a strong one with the intent to make the class prediction harder on the strongly augmented image. Next, we obtain logits by running two batches through the model: a batch of the source images and batch composed of both the source and target images. Each of the resulting batches of logits are influenced by their respective batch norm statistics, i.e. the source batch is only influenced by the source data batch norm statistics while the batch that combines source and target is influenced by both domains batch norm statistics. Two loss terms are then computed:
• The source loss term is responsible for predicting correct source labels and for aligning source and target logit domains. We first combine logits for the source images using random logit interpolation, which encourages the model to produce the same label for the hyperspace connecting source logits obtained from 1) only source examples and 2) a combination of source and target examples. In practice, this creates an implicit constraint to align the source and target domains in logit space. The newly obtained source logits are then used to compute the cross-entropy loss for the source data.
• The target loss term is responsible for predicting the correct target labels and for aligning the target predictions to a desired class distribution. Since we don’t assume access to labels for the target images, we create a pseudo-label for these images as follows. First, we rectify
the class distribution obtained from weakly augmented target images to a desired class distribution using distribution alignment. If the target class distribution is known, it can be used directly. In the general case where it is not known, we use the source class distribution instead. We then select entries of the batch for which the rectified probabilities of the weakly augmented target image predictions are above a user-defined confidence threshold. A pseudo label is then made for these outputs by selecting the most confident class, and these pseudo-labels are used with a standard cross-entropy loss applied to the logits of the strongly augmented images.
Augmentation. For a dataset D ∈ {SL, TU}, we augment each image batch XD fed into AdaMatch twice, once with a weak augmentation and once with a strong augmentation, using the same types of weak and strong augmentations as (Berthelot et al., 2020). This forms a pair of batches XD,w and XD,s respectively, which we denote together as X aug D = {XD,w, XD,s}. From these pairs of batches, we then compute logits Z ′SL, Z ′′ SL and ZTU as follows:
{Z ′SL, ZTU} = f({X aug SL , X aug TU }; θ) (1)
Z ′′SL = f(X aug SL ; θ) (2)
That is, we compute logits by calling the model twice for both the strong and weakly augmented images. The first time we pass both source and target inputs together in the same batch so the batch normalization statistics are shared, and the second time we only pass the source label data. We only update batch normalization statistics when computing {Z ′SL, ZTU} = f({X aug SL , X aug TU }; θ) to avoid double counting source labeled data. Note that without batch normalization, we would have Z ′SL ≡ Z ′′SL. But batch normalization may make them slightly different. Random logit interpolation. The role of random logit interpolation is to randomly combine the joint batch statistics from the source and target domains with the batch statistics from the source domain, which has the effect of producing batch statistics that are more representative of both domains. More precisely, during training, we obtain logits ZSL by randomly interpolating the logits Z ′SL and Z ′′ SL computed with different batch statistics:
ZSL = λ · Z ′SL + (1− λ) · Z ′′SL (3) where we sample λ ∼ UnSL·k(0, 1). Note that each individual logit gets its own random factor. Our underlying goal here is to minimize the loss for every point between Z ′SL and Z ′′ SL, which can be accomplished by either 1) having Z ′SL and Z ′′ SL be equal to each other, or 2) having the whole line between Z ′SL and Z ′′ SL be a minima. Rather than picking one of the two ways, we formulate the problem as minimizing the loss for all connecting points, which gives the model the freedom to find the best possible solution. We then use random logit interpolation as a cost effective implementation of this loss formulation. By sampling a random point along the line between Z ′SL and Z ′′ SL at each batch, over time, we implicitly allow the model to minimize the loss for all points on the line.
Distribution alignment. Distribution alignment (Berthelot et al., 2020) can be seen as an additional form of model regularization that helps constrain the distribution of the class predictions to be more aligned with the true distribution. Without it, the classifier could just predict the most prevalent class or exhibit other failure modes. Ideally, if the target label distribution is known, we would use it directly. However, when the target label distribution is unknown, we approximate it using the only available distribution – the source label distribution. A limitation of this approach is that the more the source label distribution differs from the target distribution, the more incorrect the approximation will be, which may cause the model performance to degrade. However, in practice, we find that aligning the target pseudo-labels to match the source label distribution helps significantly.
Unlike ReMixMatch (Berthelot et al., 2020), we estimate the source label distribution from the output of the model rather than using the true labels. We make this change since the model may not be capable of matching the ground truth source label distribution (particularly when source accuracy is low) but matching the source output distribution is a more attainable goal. To implement this, we first extract the logits for weakly and strongly augmented samples from the batch (by indexing):
ZSL = {ZSL,w, ZSL,s} ZTU = {ZTU,w, ZTU,s} (4) Then, we compute pseudo-labels for labeled sources and unlabeled targets:
ŶSL,w = softmax(ZSL,w) ∈ RnSL·k ŶTU,w = softmax(ZTU,w) ∈ RnTU ·k (5)
Using distribution alignment, we rectify the target unlabeled pseudo-labels by multiplying them by the ratio of the expected value of the weakly augmented source labels E[ŶSL,w] ∈ Rk to the expected value of the target labels E[ŶTU,w] ∈ Rk, obtaining the final pseudo-labels ỸTU,w ∈ RnTU ·k:
ỸTU,w = normalize ( ŶTU,w
E[ŶSL,w] E[ŶTU,w]
) (6)
normalize ensures that the distribution still sums to 1. As could be seen E[ỸTU,w] = E[ŶSL,w], which confirms that distribution alignment makes the target pseudo-labels follow the source label distribution. If the target label distribution is known, one can simply replace the term E[ŶSL,w] with it in the formula above.
Relative confidence threshold. A confidence threshold is typically used to select which predicted labels are confident enough to be used as pseudo-labels (Lee, 2013). However, since machine learning models are poorly calibrated (Guo et al., 2017), especially on out-of-distribution data Ovadia et al. (2019), the confidence varies from dataset to dataset depending on the ability of the model to learn its task. To address this issue, we introduce a relative confidence threshold which adjusts a user-provided confidence threshold relative to the confidence level of the classifier on the weakly augmented source data.
Specifically, we define the relative confidence threshold cτ as the mean confidence of the top-1 prediction on the weakly augmented source data multiplied by a user provided threshold τ :
cτ = τ
nSL nSL∑ i=1 max j∈[1..k] (Ŷ (i,j) SL,w) (7)
We then compute a binary mask ∈ {0, 1}nTU by thresholding the weakly augmented target images with the relative confidence threshold cτ :
mask(i) = max j∈[1..k] (Ỹ (i,j) TU,w) ≥ cτ (8)
Loss function. The loss L(θ) sums Lsource(θ) for the source and Ltarget(θ) for the target.
Lsource(θ) = 1
nSL nSL∑ i=1 H(Y (i) SL , Z (i) SL,w) + 1 nSL nSL∑ i=1 H(Y (i) SL , Z (i) SL,s) (9)
Ltarget(θ) = 1
nTU nTU∑ i=1 H ( stop_gradient(Ỹ (i)TU,w), Z (i) TU,s ) ·mask(i) (10)
L(θ) = Lsource(θ) + µ(t)Ltarget(θ) (11) whereH(p, q)=− ∑ p(x) log q(x) is the cross-entropy loss and stop_gradient is a function that prevents gradient from back-propagating on its argument. Prevention of gradient back-propagation on guessed labels is a standard practice in SSL works that favors convergence. µ(t) is a warmup function that controls the unlabeled loss weight at every step of the training. In practice we use µ(t) = 1/2 − cos ( min(π, 2πt/T ) ) /2 where T is the total training steps. This particular function smoothly raises from 0 to 1 for the first half of the training and remains at 1 for the second half.
As can be noted, Lsource(θ) is the cross-entropy loss typically used in fully supervised learning which the nuance that we call it twice: once on weakly augmented samples and once on strongly augmented ones. Similarly, Ltarget(θ) is the masked cross-entropy loss, where entries for which the confidence is less than cτ are zeroed. This loss term is exactly the same as in FixMatch (Sohn et al., 2020).

heading: 3.2 HYPER-PARAMETERS
text: AdaMatch only requires the following two hyper-parameters: (1) Confidence threshold τ (set to 0.9 for all experiments). (2) Unlabeled target batch size ratio uratio (set to 3 for all experiments) which defines how much larger is the unlabeled batch, e.g. ntu = nsl · uratio.

heading: 3.3 EXTENSION TO SSL AND SSDA
text: SSL takes as input two types of data: labeled samples and their labels XL and unlabeled samples XU . AdaMatch can be used without change for the SSL task by feeding XL in place of XSL and XU in place of XTU . The same trick is provided for other methods in the experimental study.
SSDA data differs from UDA by the presence of targeted labeled data XTL. AdaMatch can be used without change by feeding the concatenated batch XSTL = {XSL, XTL} in place of XSL. As in SSL, the same trick is provided for other methods in the experimental study. Note there’s a subtle effect on the uratio which ends up being implicitly halved since the labeled batch is twice bigger in SSDA than in standard UDA.

heading: 4 EXPERIMENTAL SETUP
text: We evaluate AdaMatch on the SSL, UDA, and SSDA tasks using the standard DigitFive (Ganin et al., 2016) and DomainNet (Peng et al., 2019) visual domain adaptation benchmarks and we compare to various existing methods. DigitFive experiments and ablation studies were run on a single V100 GPU, other experiments were run on a single TPU.
Datasets. Digit-Five is composed of 5 domains, USPS (Hull, 1994), MNIST (LeCun et al., 1998), MNIST-M (Ganin et al., 2016), SVHN (Netzer et al., 2011), and synthetic numbers (Ganin et al., 2016). We resize all images into 32×322 and convert them to RGB. DomainNet (Peng et al., 2019) is composed of six domains from 345 object categories. Unlike prior work (Ganin et al., 2016; Saito et al., 2018; Peng et al., 2019), unless otherwise stated, we train models from scratch to focus on evaluating the efficacy of the algorithms themselves.
Democratic research. For DomainNet, we experiment with two image resolutions, 64×64 and 224×224. Though Peng et al. (2019) only evaluate with resolution 224×224, we include 64×64 to make future experiments using DomainNet more accessible to researchers with limited compute budget. Moreover, we find that AdaMatch can outperform the previous SOTA for DomainNet at 224×224 resolution even when trained with 64×64 resolution images.
Network and training hyperparameters. We use ResNetV2-101 (He et al., 2016) for resolution 224×224 , WRN-34-2 (Zagoruyko & Komodakis, 2016) for 64×64, and WRN-28-2 for 32×32. We tried three confidence thresholds (0.8, 0.9, 0.95) on 64×64 UDA clipart→infograph and picked the best. Additionally, we picked the largest uratio allowed on 8 GPUs for a batch size of 64 using 224×224 images. We coarsely tuned (using 3 candidate values) the learning rate, learning rate decay, and weight decay using only the 64×64 supervised source data. For weight decay, we used a empirical rule: halving it for 32×32 neural networks and doubling it for the 224×224 to coarsely adapt the regularization to the capacity of the network (we used a weight decay of 0.0005 for WRN-28-2, 0.001 for WRN-34-2 and 0.002 for ResNetV2-101). We set learning rate to 0.03 and learning rate cosine decay to 0.25. We trained DigitFive for 32M images, and DomainNet for 8M images.
Baselines. We compare against popular SOTA methods from UDA and SSL, including maximum classifier discrepancy (MCD) (Saito et al., 2018), FixMatch with distribution alignment (FixMatch+), (Sohn et al., 2020), and NoisyStudent (Xie et al., 2019). For UDA, we include MCD since it is the current single-source SOTA on DomainNet (Peng et al., 2019). For SSL, we include FixMatch+ since it achieves SOTA performance on standard SSL benchmarks and our algorithm borrows many components from it. We also include NoisyStudent since it is a popular approach to SSL that achieves competitive results on large-scale datasets like ImageNet (Deng et al., 2009). Finally, we include our own baseline, BaselineBN, which uses fully supervised learning on the source domain but additionally feeds unlabeled target data through the network to update batch norm statistics (see Appendix B).
Evaluation. For each source and target dataset pair, we calculate the final accuracy as the median accuracy reported over the last ten checkpoints (we save a checkpoint every 216 samples). In summary tables, we report the average final accuracy over all target datasets. Unlike SSL, most prior work in UDA evaluates using ImageNet pre-training. As explained in Section 2, we choose to report the majority of our results without pre-training. However, for easier comparison with prior work, we also evaluate ImageNet pre-trained AdaMatch, FixMatch+, and MCD for UDA in Section 5.1.1.
2Images of USPS are resized from 16×16 with bi-cubic interpolation. For MNIST, we pad an image with two zero-valued pixels on all sides.

heading: 5 RESULTS
text: We now summarize our results for the UDA, SSL, and SSDA tasks. We ran all five algorithms on all source-target pairs for every dataset (20 for DigitFive and 30 for DomainNet64/224), varying the number for labels in SSL/SSDA tasks, for a comprehensive total of 2,420 experiments. This represents a major experimental undertaking and we hope that the results (especially the finer grained results on individual dataset pairs in Appendix E and the democratized setting) will serve future work.

heading: 5.1 UNSUPERVISED DOMAIN ADAPTATION (UDA)
text: Table 2 shows the average target accuracy achieved by each algorithm for the UDA task on the DigitFive, DomainNet64, and DomainNet224 benchmarks. For the UDA task, we additionally compare to a fully-supervised learning setup that uses an oracle model that has access to all labels from both the source and target datasets (Oracle).
We find that AdaMatch outperforms all other algorithms we compare against, and the improvement is highest for the larger dataset size, DomainNet224, where AdaMatch achieves an average target accuracy of 28.7% compared to 23.9% for NoisyStudent. Additionally, on all three benchmark datasets, AdaMatch, FixMatch+, and NoisyStudent all significantly outperform both MCD and BaselineBN. This success illustrates that SSL methods can be applied out-of-the-box on UDA problems, but we can also significantly improve upon them if, as in AdaMatch, the distributional differences between the source and target data are accounted for.
For DomainNet64, we also observe that both AdaMatch and FixMatch+ without pre-training on DomainNet64 are able to out-perform the previous state-of-the-art accuracy of 21.9% set by MCD with pre-training on DomainNet224 (Peng et al., 2019). Overall, the accuracy gap between AdaMatch on DomainNet64 and DomainNet224 is relatively low – only 2% – and, for the purposes of democratizing research, we encourage future UDA research to evaluate on DomainNet64 since it is significantly less computationally intensive to run experiments on the smaller image size.

heading: 5.1.1 UDA WITH PRE-TRAINING
text: In order to compare with prior UDA work which uses ImageNet pre-training, in Table 3 we evaluate AdaMatch versus MCD on the DomainNet224 benchmark when initializing from pre-trained ImageNet ResNet101 weights. As is typical with pre-training, we also found in our experiments that early stopping is necessary to get the best results for AdaMatch. Thus, for the pre-training results in Table 3, we report the final target accuracy achieved after training for 8 million images (our standard training protocol), the final accuracy achieved after early stopping at 2 million images, and the maximum median accuracy (over a window of 10 checkpoints) achieved across the entire run.
Using the standard training protocol of 8 million images, AdaMatch with pre-training is able to outperform MCD with pre-training, a difference of 28.2% versus 22.3%, respectively (we also confirm that the accuracy we are able to achieve on MCD closely matches the reported accuracy of 21.9%
from (Peng et al., 2019)). However, we also observe that pre-trained AdaMatch peaks early on in training, and a significantly higher accuracy of 35.6% is achievable if an Oracle were to tell when to stop training. To account for this, we recommend early-stopping AdaMatch in the pre-trained setting.

heading: 5.2 SEMI-SUPERVISED LEARNING (SSL)
text: In the SSL setting, we only train on a single dataset (since there is no notion of source nor target) which we randomly divide into two groups: labeled and unlabeled. Figure 2 shows the average target accuracy for each algorithm on the SSL task as we vary the number of target labels (we also include the corresponding numerical results in Appendix D). We find that AdaMatch achieves state-of-the-art performance for DomainNet224, and competitive performance to FixMatch+ on the DigitFive and DomainNet64 benchmarks. Additionally, we observe that increasing the number of target labels generally results in better accuracy for all methods, and on the DomainNet224 dataset in particular, the gap in performance between AdaMatch and FixMatch+ widens as we increase the number labels.
Overall, AdaMatch’s improvement over FixMatch+ is smaller on the SSL task compared to the SSDA or DA tasks, which is expected since AdaMatch by design is an extension of FixMatch+ to handle distribution shifts. This also suggests that the additional components of AdaMatch, i.e. random logit interpolation and relative confidence thresholding, improve accuracy more in settings where the unlabeled and labeled data are not drawn from the same distribution.

heading: 5.3 SEMI-SUPERVISED DOMAIN ADAPTATION (SSDA)
text: In the SSDA setting, we randomly sample a subset of the target dataset and treat it as labeled. Figure 3 shows the performance of AdaMatch compared to all other algorithms on the SSDA task as we vary the number of target labels (corresponding numerical results are in Appendix D). AdaMatch outperforms all other algorithms on both DomainNet64 and DomainNet224. As expected, increasing the number of target labels improves the accuracy for all methods. In very low label regime, AdaMatch further improves its lead over other methods.
Method Accuracy
AdaMatch 26.3 w/o random logit interpolation 25.2 w/o distribution alignment 17.7 w/o relative confidence threshold 23.2

heading: 6 ABLATION STUDY
text: In this section, we perform an ablation analysis on each component of AdaMatch to better understand their importance and we provide a sensitivity analysis of hyperparameters, such as uratio, weight decay, confidence threshold, or augmentation strategies. We conduct our study on DomainNet using 64×64 as input and report the average accuracy across 6 domain adaptation protocols.3
Exclude-One-Out Analysis. AdaMatch improves upon FixMatch with several innovative techniques: random logit interpolation, adaptive confidence thresholding and distribution alignment (initially introduced by ReMixMatch and discussed in FixMatch as an extension but not per-se part of it). To better understand the role of each component, we conduct experiments by excluding one component at a time from AdaMatch. As we see in Table 4 all components contribute to the success of AdaMatch.
Sensitivity Analysis. AdaMatch only has two hyper-parameters to tune: uratio and confidence threshold. While it is not specific to AdaMatch, we also measure the sensitivity to weight decay as it was shown to be important to achieve a good performance in (Sohn et al., 2020). Results are in Figure 4. Similarly to the findings from FixMatch (Sohn et al., 2020), AdaMatch requires high confidence threshold. Also, we observe improved accuracy with higher uratio at the expense of more computes. For L2 weight decay, we find fairly reliable performance between 0.0001 and 0.001.

heading: 7 CONCLUSION
text: Machine learning models still suffer a drop in accuracy on out-of-distribution data. However, if we have access to unlabeled data from a shifted domain (the UDA setting) or even a small amount of labeled data from the shifted domain (the SSDA setting), we can greatly improve accuracy. In this work, we present AdaMatch, a general method designed to boost accuracy on domain shifts when given access to unlabeled data from the new domain. AdaMatch unifies the domains of UDA, SSL, and SSDA, demonstrating that one method can perform well at all three.
Overall, our work shows that it is possible to apply SSL algorithms out-of-the-box to the domain adaptation problem. By taking into account the distribution shift, AdaMatch can significantly improve upon SSL methods. This suggests that a promising direction for future work is to take new advances from SSL and translate or modify them for the domain adaptation setting. While AdaMatch outperforms prior work by a large margin, there is still an even larger margin left for improvement on out-of-distribution shifts. This leaves open an important question: which is more important for future progress, making use of unlabeled data more efficiently, or attempting to better model domain shifts?
3The 6 protocols include clipart→infograph, infograph→painting, painting→quickdraw, quickdraw→real, real→sketch, and sketch→clipart.

heading: ETHICS STATEMENT
text: AdaMatch is designed to make ML models robust to a domain shift between train and test time using a limited amount of labeled data and a large amount of unlabeled data. The development of data efficient ML algorithms are of utmost importance in democratization of ML methods. However, the confirmation bias of self-training (Arazo et al., 2019) could be a concern for building a fair ML model across major and minor classes. AdaMatch partially resolves the issue with the distribution alignment, but more in-depth investigation on the fairness and robustness of data-efficient ML algorithms should be done in the future.

heading: REPRODUCIBILITY
text: We wrote our experimental codes using the open-source Objax library (Objax Developers, 2020) and used the same hyperparameters for most of our experiments, which we specify in Section 4. Moreover, we include an anonymized source code in the supplementary material of the submission for interested readers. As mentioned in the abstract, we will make the code publicly available for reproducible research.

heading: B BASELINEBN
text: For BaselineBN, the same model and hyper-parameters are used, but the loss is computed differently. Both source and target are concatenated as a batch, and the loss is only computed on the (source) labeled logits for both weak and strong augmentations.
{ZSL, ZTU} = f({XSL, XTU}; θ)
Lbaseline(θ) = 1
nSL nSL∑ i=1 H(Y (i) SL , Z (i) SL,w) + 1 nSL nSL∑ i=1 H(Y (i) SL , Z (i) SL,s)

heading: C UDA MODEL EVAL CURVES WITH AND WITHOUT PRE-TRAINED WEIGHTS
text: We randomly select a dataset pair from DomainNet224 (Sketch→Clipart) and plot the target accuracy versus the number of training images for both pre-trained and randomly initialized AdaMatch and MCD in Figure 5. For all pre-trained runs, we set weight decay to 0 and initialize model weights using an ImageNet pre-trained ResNet101 architecture.
Figure 5 shows that AdaMatch benefits significantly from early stopping. Although not plotted, we observe a similar pattern in other dataset pairs. Using a consistent early stopping rule of 2 million images, we found that, on average across all dataset pairs, AdaMatch with pre-training and early stopping outperforms randomly initialized AdaMatch by 4.7% (see Table 3).

heading: D SUMMARY RESULTS
text: In this section, we include the corresponding tables for the SSL and SSDA summary figures (Figure 2 and 3, respectively).
D.1 SSL
D.2 SSDA
E INDIVIDUAL DATASET RESULTS In this section, we evaluate all methods on the individual dataset pairs from the DigitFive, DomainNet64, and DomainNet224 benchmarks for the UDA, SSL, and SSDA tasks.

heading: SSL DigitFive AdaMatch (num_target_labels=25)

heading: SSL DigitFive AdaMatch (num_target_labels=10)

heading: SSL DigitFive AdaMatch (num_target_labels=5)

heading: SSL DigitFive AdaMatch (num_target_labels=1)

heading: SSL DigitFive BaselineBN (num_target_labels=25)

heading: SSL DigitFive BaselineBN (num_target_labels=10)

heading: SSL DigitFive BaselineBN (num_target_labels=5)

heading: SSL DigitFive BaselineBN (num_target_labels=1)

heading: SSL DigitFive FixMatch+ (num_target_labels=25)

heading: SSL DigitFive FixMatch+ (num_target_labels=10)

heading: SSL DigitFive FixMatch+ (num_target_labels=5)

heading: SSL DigitFive FixMatch+ (num_target_labels=1)

heading: SSL DigitFive MCD (num_target_labels=25)

heading: SSL DigitFive MCD (num_target_labels=10)

heading: SSL DigitFive MCD (num_target_labels=5)

heading: SSL DigitFive MCD (num_target_labels=1)

heading: SSL DigitFive NoisyStudent (num_target_labels=25)

heading: SSL DigitFive NoisyStudent (num_target_labels=10)

heading: SSL DigitFive NoisyStudent (num_target_labels=5)

heading: SSL DigitFive NoisyStudent (num_target_labels=1)

heading: SSL DomainNet224 AdaMatch (num_target_labels=25)

heading: SSL DomainNet224 AdaMatch (num_target_labels=10)

heading: SSL DomainNet224 AdaMatch (num_target_labels=5)

heading: SSL DomainNet224 AdaMatch (num_target_labels=1)

heading: SSL DomainNet224 BaselineBN (num_target_labels=25)

heading: SSL DomainNet224 BaselineBN (num_target_labels=10)

heading: SSL DomainNet224 BaselineBN (num_target_labels=5)

heading: SSL DomainNet224 BaselineBN (num_target_labels=1)

heading: SSL DomainNet224 FixMatch+ (num_target_labels=25)

heading: SSL DomainNet224 FixMatch+ (num_target_labels=10)

heading: SSL DomainNet224 FixMatch+ (num_target_labels=5)

heading: SSL DomainNet224 FixMatch+ (num_target_labels=1)

heading: SSL DomainNet224 MCD (num_target_labels=25)

heading: SSL DomainNet224 MCD (num_target_labels=10)

heading: SSL DomainNet224 MCD (num_target_labels=5)

heading: SSL DomainNet224 MCD (num_target_labels=1)

heading: SSL DomainNet224 NoisyStudent (num_target_labels=25)

heading: SSL DomainNet224 NoisyStudent (num_target_labels=10)

heading: SSL DomainNet224 NoisyStudent (num_target_labels=5)

heading: SSL DomainNet224 NoisyStudent (num_target_labels=1)

heading: SSL DomainNet64 AdaMatch (num_target_labels=25)

heading: SSL DomainNet64 AdaMatch (num_target_labels=10)

heading: SSL DomainNet64 AdaMatch (num_target_labels=5)

heading: SSL DomainNet64 AdaMatch (num_target_labels=1)

heading: SSL DomainNet64 BaselineBN (num_target_labels=25)

heading: SSL DomainNet64 BaselineBN (num_target_labels=10)

heading: SSL DomainNet64 BaselineBN (num_target_labels=5)

heading: SSL DomainNet64 BaselineBN (num_target_labels=1)

heading: SSL DomainNet64 FixMatch+ (num_target_labels=25)

heading: SSL DomainNet64 FixMatch+ (num_target_labels=10)

heading: SSL DomainNet64 FixMatch+ (num_target_labels=5)

heading: SSL DomainNet64 FixMatch+ (num_target_labels=1)

heading: SSL DomainNet64 MCD (num_target_labels=25)

heading: SSL DomainNet64 MCD (num_target_labels=10)

heading: SSL DomainNet64 MCD (num_target_labels=5)

heading: SSL DomainNet64 MCD (num_target_labels=1)

heading: SSL DomainNet64 NoisyStudent (num_target_labels=25)

heading: SSL DomainNet64 NoisyStudent (num_target_labels=10)

heading: SSL DomainNet64 NoisyStudent (num_target_labels=5)

heading: SSL DomainNet64 NoisyStudent (num_target_labels=1)

Review summaries:
This paper proposes to extend semi-supervised learning techniques to address domain adaptation problems and proposes a unified method, AdaMatch, that can handle unsupervised domain adaptation (UDA), semi-supervised domain adaptation (SSDA) and semi-supervised learning (SSL). While extensively borrowing techniques from SSL, AdaMatch proposes three unique techniques, i.e., distribution alignment,  random logit interpolation, and relative confidence threshold. Extensive experiments verify the efficacy of AdaMatch for the significant improvement over existing methods for different problems.

This paper presents a unified solution for unsupervised domain adaptation (UDA), semi-supervised learning (SSL), and semi-supervised domain adaptation (SSDA). The authors extend the state-of-the-art SSL method, that is FixMatch, to make it capable of handling DA setting where target data stem from a different distribution with source data. Experimental results show that the proposed method performs on par or significantly better than respective state-of-the-art methods in UDA, SSL, and SSDA. 

The paper extends FixMatch as a unified approach for semi-Supervised Learning, Unsupervised Domain Adaptation, and Semi-supervised Domain Adaptation.  The proposal learns representations with a cross-entropy loss between labels and logits for a strong and weak augmentation versions of the source data while the target data helps with regularization.  The proposal uses a random mixture the logits of the two augmentations, and normalizes the distribution of the pseudo-labels with the ratio of the expected logits of the augmentations.  The contribution is the mixture of aligning the label distributions by normalizing them through the ratio between the source and target predicted labels, using relative thresholds to compare source and target label distributions, and the random logit interpolation to regularize the representations.  This combinations let the method outperform the reported methods in the three tasks on two datasets.