Progressive Mixup Augmented Teacher-Student Learning for Unsupervised Domain Adaptation
Authors: Aotian Zheng
Jie Mei
Farron Wallace
Craig Rose
Rania Hussein
Jenq-Neng Hwang
Venue: Submitted to ICLR 2023 (2023)
Keywords: Unsupervised Domain Adaptation
Progressive Mixup Augmentation
Teacher-Student Learning

Abstract: Unsupervised Domain Adaptation (UDA) aims to transfer knowledge learned from a labeled source domain to an unlabeled target domain, mostly through learning a domain invariant feature representation.  Currently, the best performing UDA methods use category level domain alignment to capture fine-grained information, resulting in significantly improved performance over global alignment.  While successful, category level UDA methods suffer from the unreliable pseudo-labels for target data.  Additionally, most UDA methods directly adapt from source to target domain without regard for the large domain discrepancy.  In this paper, we propose an UDA approach with teacher-student learning where the teacher network is used to provide more reliable target pseudo-labels for the student network to train with. Furthermore, we use a progressive mixup augmentation strategy which generates intermediate samples that become increasingly target-dominant as training progresses.  Aligning the source and intermediate domains allows the model to gradually transfer fine-grained domain knowledge from the source to the target domain while minimizing the negative impact of noisy target pseudo-labels.  This progressive mixup augmented teacher-student (PMATS) training strategy along with class subset sampling and clustering based pseudo-label refinement achieves state-of-the-art performance on two public UDA benchmark datasets: Office-31, and Office-Home.

Under review as a conference paper at ICLR 2023
PROGRESSIVE MIXUP AUGMENTED TEACHER- STUDENT LEARNING FOR UNSUPERVISED DOMAIN ADAPTATION
Anonymous authors Paper under double-bling review
ABSTRACT
Unsupervised Domain Adaptation (UDA) aims to transfer knowledge learned from labeled source domain to an unlabeled target domain, mostly through learn- ing domain invariant feature representation. Currently, the best performing UDA methods use category level domain alignment to capture fine-grained information, resulting in significantly improved performance over global alignment. While suc- cessful, category level UDA methods suffer from the unreliable pseudo-labels for target data. Additionally, most UDA methods directly adapt from source tar- get domain without regard for the large domain discrepancy. In this paper, we propose an UDA approach with teacher-student learning where the teacher net- work is used to provide more reliable target pseudo-labels for the student during training. Furthermore, we use a progressive mixup augmentation strategy which generates intermediate samples that become increasingly angeet-domiant as train- ing progresses. Aligning source and intermediate domains allows the model to gradually transfer fine-grained domain knowledge from the source to the target domain while minimizing the negative impact noisy target pscudo-labels. This progressive mixup augmented teacher-student (PMATS) training strategy along with class subset sampling and clustering based pseudo-label refinement achieves tate--o-the-and performance two public UDA benchmark datasets: Office-31 and Office-Home.
INTRODUCTION
Unsupervised Domain Adaptation (UDA) has become popular research topic due to its necessity in applying deep learning models real world scenarios. Often, there exists domain gap (Quinonero- Candela et al. 2008) between training data and real world testing data that negatively affects model performance during test time. Collecting and labeling data from various domains impractical due to being both time consuming and labor intensive. Though semi-supervised learning (Berthelot cct al. 2201 Sohn et al.) 2020) and unsupervised learning (Gidaris et al. (0188) have been studied improve generalizability to unseen and unlabeled data, it is under the assumption that both the labeled and unlabeled data existed a similar domain.
UDA specifically tackles the problem of data distribution shift, or domain shift, between the labeled source data and the unlabeled target data through transferring knowledge learned from the source domain the target domain. UDA methods can mostly be split into two different categories. Ad- versarial adaptation methods (Ganin & Lempitsky 2015 Tzeng et al. 2017). inspired by generative adversarial networks (GANs). introduce a domain discriminator encourage domain confusion in the feature generator through domain-ad adversarial objectives. This minimizes the gap between source and target distributions through learning domain invariant features. Statistical adaptation methods (Long et al., 2015 2017) align source and target domain distributions through minimizing a statis- tical discrepancy measure, such as maximum mean discrepancy (MMD) and joint MMD (JMMD), between the two domains. Most domain adaptation methods in these two categories directly align source and target domain distributions (Ganin & Lempitsky 2015) Tzeng et al. 2017 Long et al.) 2015 2017) without consideration for the large domain discrepancy. e.g. from synthetic images to real images (Peng et al. (017) or from art to real images (Venkateswara et al., 2017). Even when
Na al. (2021) and Hua & Guo (202 attempt address the large domain gap, they only create

---

Under review as a conference paper at ICLR 2023
small number of augmented intermediate domains (4 or less) based arbitrarily fixed mixup ratio between source and target images.
Recent domain adaptation research (Zhu et al. 2020 Long et al. 2018 Pei et al. 2018 Kumar et al, 2018) has found performing domain alignment at the category level, taking into account the source and target sample class information, to be more effective than naively learning global domain shift. Since target data are unlabeled in the UDA setting, category level alignment methods rely on producing pseudo-labels for target samples. These generated pseudo-labels are noisy and unreliable, presenting a problem for deep convolutional neural networks (CNNs) which lack robustness to such pseudo-labels (Morerio al. 2020, Jiang et al. 2020). Replacing the CNN backbone with a vision transformer (Xu et al. (022)) has been shown to improve model robustness to noisy pseudo-labels.
In this paper. we address the two issues of large domain gap and noisy pseudo-labels mentioned above by proposing a progressive mixup training strategy with teacher-student learning. We con- struct an intermediate augmented domain that becomes progressively more target-like as training continues. During different periods of training, the intermediate domain has different characteris- ties. For example, the intermediate domain is more source-like initially, with more reliable label information but lower correlation with target domain. As training progresses, the intermediate do- main becomes more target-like, with less reliable label information but higher similarity to target domain. By gradually changing the intermediate domain from source-like to target-like, we're 're able to train one model that retains the benefits of both perspectives without using an ensemble of models as in Na et al. (2021). For category level subdomain alignment, we use teacher model to generate pseudo-labels for training our student model. Our teacher model, as a temporal ensemble (Tarvainen) & Valpola, 2017) of the student model, produces less noisy target predictions by averaging together the predictions of many student models at previous time steps. Furthermore, we use a clustering based pseudo-late -aauel refinement method to obtain more reliable soft pseudo-labels
We evaluate the performance of our Progressive Mixup Augmented Stacher-Student (PMATS) algo- rithm on two different UDA benchmarks with varying degrees of domain shift. Experiments prove the effectiveness of our approach since we achieve state-of-the-art performance on both datasets. The contributions of this paper are summarized as follows.
We propose an algorithm that effectively combines teacher-student learning with a progres- sive mixup to efficiently bridge the source and target domains utilizzng a gradually shifting intermediate domain. This not only effectively combines knowledge from both source source-like and target-like perspectives, but also increases the model robustness to noisy pseudo-labels.
Target predictions from our classifier are further refined through spherical K-means clus- tering and Gaussian Mixture Model (GMM) to produce more reliable soft pseudo-labels.
We validate the effectiveness of our approach through extensive ablation studies and eval- uation three standard benchmarks.
RELATED WORKS
2.1 UNSUPERVISED DOMAIN ADAPTATION
fea- ture learning methods. Statistical methods train a deep adaptation network (DAN) (Long et al.] 2015) to minimize the maximum mean discrepancy (MMD) between source and target features over domain specific layers. Building on top of MMD, joint adaptation networks (JAN) (Long et al. 2017] aligns the joint distributions of multiple domain-spectie discrepancy (JMMD). Weighted MMD (Yan et al. 2017) alleviates class weight bias by assigning class specific weights to source data. Moving from global domain alignment to category level, or subdomain, alignment, deep subdomain adaptation network (DSAN) (Zhu et al.) 2020) uses local maximum discrepancy (LMMD) to align the subdomain distributions. The Contrastive Adaptation Network (CAN) (Kang al. 2019) uses a similar discrepancy measure called the Contrastive Do- main Discrepancy (CDD) which not only minimizes the intra-class subdomain discrepancy, but also maximizes the inter-class subdomain discrepancy in a contrastive manner. Completely replacing the MMD based loss, cross-domain contrastive learning (CDCL) (Wang et al., 2022) instead uses contrastive learning to align source and target distributions through modified noise-contrastive
Most recent UDA methods are based on domain alignment and discriminative
layers using a joint maximum mean
2

---

Under review as a conference paper at ICLR 2023
Student source feature, 4n
source Image
prediction 3%
loss
Backbone
mikup image
mixup Tenturr. Zm
LMMD loss
groonnd truth
paocula lubel miixup(y..3
exponential moving
Teacher
target
Backbone
target feature,
cis
prodtetion
Pseuco-label
refined %
Image
ft
Refinement
An overview of our proposed pipeline. Source data and augmented mixup data will be input into the student feature extraction backbone while target data will be input into the teacher feature extraction backbone to obtain corresponding embedding features Z., Zm. and Z,. Target feature Z, will go through the teacher classification head obtain target prediction Dt. which will then go through clustering-based pseudo-label refinement to obtain refined probability distribution V/- The mixup label y/m calculated from target refined pseudo-label 1/, and source ground truth label y/s will be used in the LMMD loss to align source and mixup feature distributions at the subdomain level. Finally, source feature Z, will also go through the student classification head to obtain source prediction Ds for classification (cross-entropy) loss. The student network weights updated based on these two loss functions while the teacher network an exponential moving average of the student network.
Figure I:
timation (infoNCE) loss (Oord al. 2018), which has the anchor sample from domain and the corresponding positive and negative samples be from another domain. Adversarial methods, such the domain adversarial neural network (DANN) (Ganin Lempitsky 2015). learn do-
main invariant feature representation through back-propagating the reverse gradients of a domain classifier. Building on top of DANN, dual mixup regularized learning (DMRL) (Wu al.) 2020) guides the classifier to enhance consistent predictions between samples and enriches the intrinsic structures of the latent space through two mixup regularizations, FixBi (Na et al. (022] uses mixup to bridge the large domain gap between the source and target domains by creating two intermediate representations through fixed ratio-based mixup between source and target samples.
2.2 PSEUDO-LABELING
For category level domain alignment, generating pseudo-labels (Lee et al. 2013) is necessity due to the lack of target labels in the UDA setting. For domain adaptation, pseudo- labels have been generated directly from the model prediction (Zhu et al.] 2020; Long et al., 2018) for conducting conditional or subdomain distribution alignment. Caron et al. a (2018) propose deep self-supervised method by generating pseudo-labels via K-means cluster, which is also used in Kang et al.) (2019). Cross-doman transformer (CDTrans) (Xu al. (202I) combines both methods by first initializ- ing cluster centroids with weighted K-means clustering using the model probability output. Our pseudo-labes refinement method takes this step further by using the K-means clustering results as initialization for Gaussian Mixture Model (GMM) to produce refined soft pseudo-label.
2.3 SELF-ENSEMBLING AND MEAN TEACHER
Self-ensembling was originally proposed for generating pseudo-labels in the semi-supervised learn- ing scenario. The temporal ensemble (Laine & Aila 2016) averaged the predictions of model over time for a better approximation of the true label compared to the model prediction at single time. Self-ensembling was extended to the Mean Teacher model (Tarvainen & Valpola 2017) which directly averaged the model weights rather than the model predictions. A student model would
3

---

Under review as a conference paper at ICLR 2023
be trained with cross-entropy loss labeled data and to match the teacher model predictions unlabeled data. The Mean Teacher model has been directly used for domain adaptation on image classification (French al.] 2017) and medical imaging segmentation (Perone et al. 2019) tasks, training the student model to match the teacher model prediction for unlabeled target data. Rather than directly performing domain adaption with the Mean Teacher method, the better teacher pseudo-labels for subdomain feature alignment.
2.4 VISION TRANSFORMER
The transformer proposed in Vaswani al. (2017) model sequential data in the field of Natural Language Processing (NLP). It has since been adapted to computer-vision tasks with great success, resulting in transformer based models becoming more and more popular. For example, the Vision Transformer (ViT) (Dosovitskiy et al., 2020) uses sequences of image patches as the transformer input. The Data-efficient Image Transformer (DeiT) (Touvron al., 2021) introduces distillation strategy for transformers to help with VIT training. VIT variants have been shown to outperform their CNN counterparts in various tasks beyond image classification, such as object detection (Liu et al. 2021). semantic segmentation (Yuan et al.) [021], and object RelD (He et al.) 2021). This trend is also seen in domain adaption for image classification, where the transformer based method (Xu et al. (021) outperforms other CNN based UDA methods most benchmarks. We conduct experiments using both CNN and transformer backbones of similar size to show that method is effective regardless of the backbone architecture.
3 METHOD
In unsupervised domain adaptation, we are given a source domain D, = {xx], vf) nf, source samples and a target domain D, = ) of n, target samples where y E R j=1
the one-hot label of zf and C is the number of classes. D. and D, sampled from distributions p and respectively, where p + q. Our goal in the UDA scenario is to train a network on labeled source domain data x and unlabeled target domain data x² make accurate predictions ÿ' on x'.
In this section, we will go over the components of our PMATS algorithm as shown in Figure
3.1 TEACHER NETWORK
The teacher network is identical to the student network in architecture and is updated iteration t based on the equation
8, = 80_ a + 0, a),
(1)
where o' and 0 are the teacher and student model parameters respectively and a is the exponential moving average (EMA) decay.
3.2 CLUSTERING-BASED PSEUDO-LAREL REFINEMENT
Given target features 2 from our feature generator backbone and target probability distribution ø) from our classifier as the initial pscudo-label, we perform series of clustering operations to refine the pseudo-label as shown in Figure 2 This is based on the assumption that even with domain shift, target features of the same class tend to be close together. We use this assumption to refine our pseudo-labels based on the K-means clustering of target samples. K-means clustering gives us a hard label that may not be suitable for samples close to multiple cluster centers. Therefore, we use GMM to convert the hard label to soft probability distribution. Details of our pseudo-label refinement process are discussed Appendix
3.3 SUBDOMAIN ALIGNMENT
To perform the category level domain alignment, we use the local maximum mean discrepancy (LMMD) loss proposed in Zhu et al. (2020), which takes advantage the target probability distri- bution for calculating discrepancy. unlike the similiar CDD (Kang al. 2019) loss, which uses hard target pseudo-labed We will go over the concept MMD before formulating the LMMD loss as extension of MMD.

---

Under review as a conference paper at ICLR 2023
target feature, Zt
K-means clustering
GMM
GMM
refined
target images features
cluster centroids
initielized with 9t
Figure 2: Pseudo-label refinement. Target predictions from classifier head will be used to initialize cluster centroids for K-means clustering. The results of the K-means clustering will then be used to initialize GMM for optimization with the EM algorithm. Finally, the GMM will output refined pseudo-label 8 given target feature Ze
3.3.1 MAXIMUM MEAN DISCREPANCY
Maximum mean discrepancy (MMD) Gretton al. 2012 is kernel two-sample test, which eval- uates the null hypothesis that = based on the idea that if two distributions are identical, then all their statistics should be the same. MMD defines the difference between two distributions the difference in their means in the reproducing kernel Hilbert space (RKHS) H:
du (p. 4) [o]=())) - Eg(o(x')]11
(2)
where 66+) is some functional mapping that maps the original samples to the RKHS with some kernel. In practice, MMD is estimated with the empirical kernel mean embeddings
du q) =
1
Σ ((x))
Σ o(z') ZjED,
(3)
Tse ZLED.
3.3.2 LOCAL MAXIMUM MEAN DISCREPANCY
To align the distributions relevant subdomains rather than the global source and target distribu- tions, LMMD (Zhu et al. 2020) calculates difference values for each of the C' categories using class weight w representing the probability that the sample belongs to class c.
dr(p.g)
Σ F.DD
w b(x*)
Σ wf(((x) IGED,
Where weight uf for sample x; is caleulated as:
wf
Vic
(5)
(e
where Yie is the e-th entry for vector yi. For source samples, one-hot true label y is used to calculate weight w For target samples, we use the soft pscudo-label 0/ to calculate weight wfs.
Note that in our PMATS algorithm, align the source distribution p with an augmented interme- diate distribution of rather than with the target distribution q directly. To adapt the feature layers, we the features 2 and 27th corresponding to source sample x; and mixup sample 2 respectively. The mixup sample xm and corresponding mixup label y are obtained through the progressive mixup augmentation covered in the following section. Finally, thanks to the characteristics of the
5

---

Under review as a conference paper at ICLR 2023
mixup images
source image
target image
training epoch
Figure 3: Progressive Mixup Augmentation. Augmented mixup images will gradually transition from source-like to target-like as training progresses.
RKHS, reformulate Eq.
44
(p
1
Σ
Th Rx
C
j=l
w 2m w k( ((<))
(6)
am
w w
mc
During training, we combine LMMD loss Limend with cross-entropy loss Log on the labeled source data.
3.4 PROGRESSIVE MIXUP AUGMENTATION
Mixup Zhang et al. 2017 data augmentation method generally used to increase neural network robustness to noisy pseudo-labels. In the context of UDA, mixup has been used to construct virtual samples by taking the convex combination of labeled and unlabeled data (Na et al. 2021 Wu al., 2020). Most methods use mixup ratio sampled from the beta distribution: X ~ Beta(a, a), where hyperparameter. FixBi (Na et al. 202I) instead uses two fixed ratios to avoid ran- domness. PrDA Hua & Guo (2020) uses four fixed ratios for what they call progressive domain augmentation. We also sample our mixup ratio from the beta distribution, but with different hyper- parameters:
A N Beta(d + eo, e),
where a, and 3 are hyperparameters and the epoch number. By keeping 3 and fixed, we use to decide how quickly we transition from source-like to target-like and d controls how close the source sample initially. Given pair of samples (x*, and their labels (v", j'*) from the source and target domains respectively, the corresponding mixup sample and label is given as
the mixup sample
Ttth (1 A)x" + Xx
(8)
= A)y + Xj"
Another key difference between our method and PrDA that PrDa only creates virtual samples xn and not their corresponding label Td as they perform domain adaptation through multiple subspace alignment on the Grassmann manifold. Our progressive mixup augmentation is visualized in Figure

---

Under review as a conference paper at ICLR 2023
Algorithm 1: PMATS Training Procedure
Input unlabeled target dataset D2 X,, source dataset D2 = (X,, Ye), student model f.
teacher model f', max epochs E, iterations per epoch K, warmup epochs W, set of all
classes C
for to E do
Initialize cluster centers with target class predictions using Eq. (TO
Perform K-means clustering target data Xx, obtain intermediate pseudo-labels y**;
Fit GMM model to target data X, while using yd to initialize parameters based on Eqs.
(14). [55. (16). (17) to obtain refined pseudo-labels y":
for k to do
Randomly choose subset of classes C' from the set of all classes C;
if S W then
Sample batches (xf uf) and (/)) belonging classes C from D. and Dú: end
else
Sample batches (a%, vt) and v()) belonging to classes C' from D2 and Dy; end
Calculate loss
LCE Le based on Eq. 16:
Back- propagate and update model f;
Update model J' based on Eq. 47;
end
end
3.5
IMPLEMENTATION DETAILS
The training process of our PMATS algorithm is show in Algorithm For further implementation settings, please see Appendix A
4
EXPERIMENTS
We evaluate our proposed method PMATS on two domain adaptation benchmarks and compare with sate-of-the-art UDA methods. Additionally, validate the contribution each of our components through extensive ablation studies.
4.1 DATASETS
Office-31 (Saenko et al. 2010 is one of the most popular datasets for real-world domain adaptation, containing total of 4,1 images of 31 different categories. Office-31 had different domains: the Amazon (A) domain with 2817 images, the Webcam (W) domain with 795 images, and the DSLR (D) domain with 498 images. Despite being popular, the relatively small size of the dataset and similar domains makes it less useful for comparing SoTA methods when most can achieve similarly high accuracies.
Office-Home (Venkateswara et al., 2017) is a newer dataset consisting of 15,588 images over 65 different categories. Office-Home has 4 different domains: the Artistic (A) domain, the Clipart (C) domain, the Product (P) domain, and the Real-World (R) domain. Besides being much larger than Office-31, 1, the domain combinations Office-Home offer a nice mix easy and difficult adaptation tasks.
4.2 FEATURE VISUALIZATION
We use t-SNE Van der Maaten & Hinton 2008 visualize features from the source and target domain before and after alignment in Figure 4 Before alignment, we can see that the source and target features separated, indicating domain gap. After alignment with PMATS, the target features mostly form compact clusters around the source features, except for the target feature near the center which still remains spread out. These spread out feature difficult classify and thus

---

Under review as a conference paper at ICLR 2023
(a) Features without adaptation
(b) Features aligned by PMA
The t-SNE (Van der Maaten Hinton 2008) visualization of features before and after alignment on task A - W. Red points are source samples and blue points are target samples
Figure 4:
is not forced by our LMMD loss into distinct cluster. By using soft pseudo-labels from GMM rather than hard pseudo-label from K-means clustering, we are able to avoid aligning these hard to classify samples with the incorrect subdomain distribution.
Table 1: Comparison with SoTA on Office-31 dataset.
The best performance marked in bold. indicates that Deil-small was used backbone instead of ResNet50.
Method
A
W
A
D
W
A
W
Avg
ResNet50 (He 2016
68.9
68.4
96.7
60.7
76.1
DANN (Ganin Lemptisky 2015
79.7
82.0
67.4
CDAN+E Long et al. 2018
92.9
94.1
87.7
DSAN
2020
90.2
93.6
98.3
100.0
88.4
CAN KKng et 9
94.5
78.0
99.1
90.6
SRDC Tang et al.
95.7
99.2
77.1
100.0
90.8
RSDA-MSTN
95.8
96.0
99.3
78.9
100.0
91.1
FixBi Na al.
95.0
99.3
100.0
D (Touvron atal. 2021
87.6
86.9
97.7
73.5
86.7
CDTrans-S*
al. 2021
94.6
PMATS
96.4
100.0
PMATS*
97.4
97.7
80.7
99.8
92.5
Table 2: Comparison with SoTA on Office-Home dataset. The best performance is marked in bold. indicates that Deit-small used as backbone instead of ResNet50.
Melhod
R
Ave
elal.
DANN Caniin
Lemglssy 2015
45.6
59.3
70.1
47.0
58.5
60.9
46.1
43.7
68.5
63.2
51.8
76.8
57.6
CDANEE
20133
$4.6
74.1
78.
63.0
72.2
74.1
61.6
$2.3
79.
72.3
57.3
82.8
68.5
DSAN
20227
70.8
75.4
67.8
$5.9
78.5
60.3
83.1
67.6
SRDC Tang cl (000)
76.3
81
69.5
76.2
53.8
81.7
76.3
57.1
71.3
RSDA-MSTN Larr in 20210
53.2
77.7
81.3
66.4
74
76.5
67.9
53
75.8
57.8
K5.4
70.
FiaBi 2a A1d
58.]
77.3
80.1
67.7
79.5
78.1
65.8
$7.9
81.7
76.4
62.9
86.7
72.7
DeicS*
CDTransS (Xi sil 2021
60.6
79.5
82.4
75.6
81.0
82.3
72.5
56.7
84.4
77.0
59.1
85.5
74.7
PMETS
57.8
7901
81.6
66.3
75.9
80.4
68.8
57.2
81.0
76.5
03.4
83.0
73.1
PMATS
80.2
84.0
75.1
82.1
81.5
75.5
64.9
84.6
78.5
66.7
88.5
77.0
8

---

Under review as a conference paper at ICLR 2023
4.3 COMPARISON
Results on the Office-31 dataset are presented in Table [ Our method with ResNer50 backbone obtains results similar other best performing SoTA methods FixBi and RSDA-MSTN, around 91.4% average accuracy over all domain transfer tasks. Switching to the Deit-Small backbone, our accuracy improves by 1.0% on average, beating all other SoTA methods. Even CDTrans-S, which uses the same transformer backbone, only achieves a 90.4% accuracy compared to our 92.4%.
Thole our results on the Office-Home dataset. On this dataset, our method with the ResNet50 backbone performs slightly better than the SoTA CNN based method FixBi at 73.1% % average ac- curacy. When we switch in accuracy across all categories. In fact, our PMATS* method has the best accuracy in most cat- egories. The
transformer backbone, however, we saw a significant improvement
C' and R - C categories show the largest improvement over the other SoTA methods, increasing by around 8% from the second best performing CDTrans, which also uses the Deit-Small backbone. Overall, our method beats the other SoTA methods by at least 2.3% With large improvement in categories with lower baseline accuracies, this shows that progressive mixup augmentation method achieves our goal bridging large domain gaps to aid in the adapta- tion.
4.4 ABLATION STUDY
We conduct ablation studies on 4 of the 6 adaptation task of the Office-31 dataset to investigate the effectiveness of each of our components. In table [44] we show improvement through adding progressive mixup (PM), teacher-student (TS), and pseudo-label refinement (PR) over the baseline DSAN Zhu et al. (2020) with class aware sampling. Our progressive mixup improves the accuracy by average of 1.5% while teacher-student learning with two-stage pseudo-label refinement strategy provides an additional 0.9% improvement on average. This shows that each component is effective in improving performance.
Table 3: Ablation study the effectiveness of components the four non-trivial tasks of the Office-31 dataset. We use Deit-Small as our backhone for this experiment.
DSAN PM
TS
PR
A
+
D
A
W
D
+
A
W
A
Avg
94.4
96.0
77.7
78.6
86.7
96.0
97.7
78.9
80.1
88.2
96.6
97.7
79.5
80.6
88.6
97.4
97.7
80.4
80.7
89.1
5 CONCLUSION
In this paper, we propose PMATS algorithm that aims to create intermediate augmented domains between the source and target domains, as to aid in domain adaptation for cases with large do- main discrepancy. We achieve this through progressive mixup between random source and target sample pairs to create intermediate samples that gradually shifts from source-like target-like. By minimizing the discrepancy between our source and intermediate distributions, we able to take advantage of multiple perspectives between the source and target domain during training. Further- more, we propose a clustering -based pseudo-label refinement strategy combined with temporal ensemble teacher network to generate more reliable target pseudo-labels for subdomain alignment. From experimental results, show our method to be superior other SoTA UDA methods difficult domain adaptation benchmarks.
9

---

Under review as a conference paper at ICLR 2023
REFERENCES
David Berthelot, Nicholas Carlini, Ian Goodfellow, Nicolas Papernot, Avital Oliver, and Colin A Raffel. Mixmatch: A holistic approach semi-supervised learning. Advances in neural informa- fion processing systems, 32, 2019.
Mathilde Caron, Piotr Bojanowski, Armand Joulin, and Matthijs Douze. Deep clustering for un- supervised learning of visual features. In Proceedings of the European conference on computer vision (ECCV), pp. 132-149, 2018.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale scale hi- erarchical image database. In 2009 IEEE conference on computer vision and pattern recognition, pp. 255-255.11 leee, 2009.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x 16 words: Transformers for image recognition seale. arXiv preprint arXiv:2010. 1929, 2020.
Geoffrey French, Michal Mackiewiez, and Mark Fisher. Self-ensembling for visual domain adapta- tion. arXiv preprint /20.00000005208, 2017.
Yaroslav Ganin and Victor Lempitsky. Unsupervised domain adaptation by backpropagation. In International conference on machine learning, pp. 1180-1189, PMLR, 2015.
Spyros Gidaris, Praveer Singh, and Nikos Komodakis. Unsupervised representation learning by predicting image rotations. arXiv preprint 1803.0078 2018.
Arthur Gretton, Karsten M Borgwardt, Malte J Rasch, Bernhard Schölkopf, and Alexander Smola.
A kernel two-sample test. The Journal of Machine Learning Research, 13(1):723-773, 2012.
Xiang Gu, Jian Sun, and Zongben Xu. Spherical space domain adaptation with robust pseudo-label loss. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pp. 9101-9110, 2020.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 770-778, 2016.
Shuting He, Hao Luo, Pichao Wang, Fan Wang, Hao Li, and Wei Jiang. Transreid: Transformer- based object re-identification. In Proceedings of the IEEE/CVF international conference on com- puter vision. pp. 15013-15022, 2021.
Kevin Hua and Yuhong Guo. Unsupervised domain adaptation with progressive domain augmenta- tion. arXiv preprini arXiv:2004.0173:: 2020.
Xiang Jiang, Qicheng Lao, Stan Matwin, and Mohammad Havaei. Implicit class-conditioned do- main alignment for unsupervised domain adaptation. International Conference on Machine Learning, pp. 4816-4827. PMLR, 2020.
Guoliang Kang, Lu Jiang, Yi Yang, and Alexander G Hauptmann. Contrastive adaptation network for unsupervised domain adaptation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp.
2019.
Abhishek Kumar, Prasanna Sattigeri, Kahini Wadhawan, Leonid Karlinsky, Rogerio Feris, Bill Free- man, and Gregory Wornell. Co-regularized alignment for unsupervised domain adaptation. Ad- vances Neural Information Processing Systems, 31, 2018
Samuli Laine and Timo Aila. Temporal ensembling for semi-supervised learning. arXiv preprint 161iv:1610.022 2016.
Dong-Hyun Lee al. Pseudo-label: The simple and efficient semi-supervised learning method for deep neural networks. In Workshop on challenges representation learning, ICML, volume 3, pp. 896, 2013.
10

---

Under review as a conference paper at ICLR 2023
Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer. Hierarchical vision transformer using shifted windows. In Proceedings of the IEEE/CVF International Conference Computer Viston. pp. 10012-10022, 2021.
Mingsheng Long, Yue Cao, Jianmin Wang, and Michael Jordan. Learning transferable features with deep adaptation networks. In International conference machine learning, pp. 97-105, PMLR, 2015.
Mingsheng Long, Han Zhu, Jianmin Wang, and Michael Jordan. Deep transfer learning with joint adaptation networks. In International conference on machine learning, pp. 2208-2217. PMLR, 2017.
Mingsheng Long, Zhangjie Cao, Jianmin Wang, and Michael Jordan. Conditional adversarial domain adaptation. Advances in neural information processing systems, 31, 2018.
Pietro Morerio, Riccardo Volpi, Ruggero Ragonesi, and Vittorio Murino. Generative pseudo-label refinement for unsupervised domain adaptation. In Proceedings of the IEEE/CVF Winter Confer- ence on Applications of Computer Vision, pp. 3130-3139, 2020.
Jaemin Na. Heechul Jung, Hyung Jin Chang, and Wonjun Hwang. Fixbi: Bridging domain spaces for unsupervised domain adaptation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 1094-1103, 2021.
Aaron van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predic- tive coding, arXiv preprint arXiv:1807.03771 2018
Zhongyi Pei, Zhangjie Cao, Mingsheng Long, and Jianmin Wang. Multi-adversarial domain adap- tation. In Thirty-second AAAI conference on artificial intelligence, 2018.
Xingchao Peng, Ben Usman, Neela Kaushik, Judy Hoffman, Dequan Wang, and Kate Saenko. Visda: The visual domain adaptation challenge. arXiv preprint arXiv: 2017.
Christian S Perone, Pedro Ballester, Rodrigo C Barros, and Julien Cohen-Adad. Unsupervised domain adaptation for medical imaging segmentation with self-ensembling. Neurolmage, 194: 1-11, 2019.
Joaquin Quinonero-Canded Masashi Sugiyama, Anton Schwaighofer, and Neil D Lawrence. Dataset shift in machine learning. Mit Press, 2008.
Kate Saenko, Brian Kulis, Mario Fritz, and Trevor Darrell. Adapting visual category models to new domains. In European conference on computer vision, pp. 213-226, Springer, 2010.
Kihyuk Sohn, David Berthelot, Nicholas Carlini, Zizhao Zhang, Han Zhang, Colin A Raffel, Ekin Dogus Cubuk, Alexey Kurakin, and Chun-Liang Li. Fixmatch: Simplifying semi-supervised learning with consistency and confidence. Advances in neural information processing systems, 33:596-608, 2020.
Hui Tang, Ke Chen, and Kui Jia. Unsupervised domain adaptation via structurally regularized deep clustering. In Proceedings of the IEEE/CVF conference computer vision and pattern recognition, pp. 8725-8735, 2020.
Antti Tarvainen and Harri Valpola. Mean teachers are better role models: Weight-averaged con- sistency targets improve semi-supervised deep learning results. Advances in neural information processing systems, 30, 2017.
Hugo Touvron, Matthieu Cord, Matthijs Douze, Francisco Massa, Alexandre Sablayrolles, and Hervé Jégou. Training data-efficient image transformers & distillation through attention. In International Conference on Machine Learning, pp. 10347-10357. PMLR, 2021.
Eric Tzeng, Judy Hoffman, Kate Saenko, and Trevor Darrel.. Adversarial discriminative domain adaptation. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 7167-7176. 2017.
11

---

Under review as a conference paper at ICLR 2023
Laurens Van der Maaten and Geoffrey Hinton. Visualizing data using t-sne. Journal of machine learning research, 9(11), 2008.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention all you need. Advances in neural informa- tion processing systems, 30, 2017.
Hemanth Venkateswara, Jose Eusebio, Shayok Chakraborty, and Sethuraman Panchanathan. Deep hashing network for unsupervised domain adaptation. In Proceedings of the IEEE conference on computer vision and patlern recognition, pp. 5018-5027, 2017.
Rui Wang, Zuxuan Wu, Zejia Weng, Jingjing Chen, Guo-Jun Qi, and Yu-Gang Jiang. Cross-domain contrastive learning for unsupervised domain adaptation. IEEE Transactions on Multimedia, 2022.
Yuan Wu, Diana Inkpen, and Ahmed El-Roby. Dual mixup regularized learning for adversarial domain adaptation. In European Conference on Computer Vision, pp. 540-555, Springer, 2020.
Ri-Hua Xiang and Run-Sheng Wang- A range image segmentation algorithm based on gaussian mixture model. Journal of Software, 14(7):1250-1257) 2003.
Tongkun Xu, Weihua Chen, Pichao Wang, Fan Wang, Hao Li, and Rong Jin. Cdtrans: Cross-domain transformer for unsupervised domain adaptation. arXiv preprint urXiv:2109.06166 2021.
Hongliang Yan, Yukang Ding, Peihua Li, Qilong Wang, Yong Xu, and Wangmeng Zuo. Mind the class weight bias: Weighted maximum mean discrepancy for unsupervised domain adaptation. In Proceedings of the IEEE conference computer vision and pattern recognition, pp. 2272-2281, 2017.
Li Yuan, Qibin Hou. Zihang Jiang, Jiashi Feng, and Shuicheng Yan. Volo: Vision outlooker for visual recognition. arXiv preprint arXiv:2106. 13112, 2021.
Hongyi Zhang, Moustapha Cisse, Yann N Dauphin, and David Lopez-Paz. mixup: Beyond empirical risk minimization. arXiv preprint 11Xf::1710.09412, 2017.
Yongchun Zhu, Fuzhen Zhuang, Jindong Wang, Guolin Ke. Jingwu Chen, Jiang Bian, Hui Xiong, and Qing He. Deep subdomain adaptation network for image classification. IEEE transactions on neural networks and learning systems, 32(4):1713-17) 2020.
APPENDIX
A IMPLEMENTATION SETTINGS
For our backbone, we use ImageNet Deng et al. (2009) pretrained weights as initialization. Based on the class aware sampling procedure in Kang al. (2019), in each iteration we randomly choose subset of 10 classes to sample our our data from. For target data, we use our generated pseudo-label for the class aware sampling. Because our network initially cannot provide good target pseudo- labels, we define warmup period of W epochs during which we sample data normally.
In our experiments, we report our results using both ResNet50 He et al. (2016) and Deit-Small Touvron et al. (202T) as backbones for the Office-31 Saenko et al. (2010) and Office- Venkateswara et al. (2017) datasets to have a fair comparison other SoTA methods. More specifi- cally, since the ResNet50 and Deit-Small have comparable model size, this yields fair comparison with other methods that use either as a backbone. For both datasets, we set a, 3, and to be 0.5, 1, and 2 respectively for the mixup ratio beta distribution in Eq. 77. We use the same progressive where is the training process Iinearly changing from O to and 7 is 1. We use the SGD optimizer for training with a momentum of 0.9, an initial learning rate of 0.01, and a weight decay of 0.0005. Our learning rate schedule is defined as no (1+0070 where no is the initial learning rate, is set to 2 for Office-31 and 3 for Office-Homee and b is 0.75.
schedule as in Zhu et al. (2020) for the trade-off term Ay: deo = (I+emp(-70))
12

---

Under review as a conference paper at ICLR 2023
B CLUSTERING BASED PSEUDO-LABEL REFINEMENT
B.1 K-MEANS CLUSTERING
After sending all of the target data through our model to obtain 2} and ijj, we compute initial cluster centroids Og:
Eje UJ4
O,
(10)
Ejer Dj
where c is the category number and is the set all target samples of size Nr. The one-hot pseudo- labels can then be reassigned via nearest neighbor classifier, shown in Eq. [1] using cosine distance in Eq. 12).
03 = {
if c argminy dist(Ok:=") otherwise,
(11)
where
1 < a, L,,b.
dist(a, b) = - |a|||bb|]
(12)
Based on the new pseudo-labels y", we can calculate new cluster centroids:
O² Ejer 05 ΣSET 07
(13)
Equations and (13) are repeated iteratively until convergence or the maximum number of clus- tering steps is reached.
B.2 GAUSSIAN MIXTURE MODEL
Performing K-mean gives us a hard pseudo-label since each target sample is directly assigned to one cluster with no probability of belonging to multiple clusters/classes. To convert this hard label to soft probability distribution, we use GMM with parameters initialized from our K-means clustering results from Section B.I For the e-th Gaussian component, we can initialize mean Me. converiance Cou., and influcence factor Ne with the following equations.
Pe = Ejer D
(14)
Ejer Dic
Cov, = ΣjET Uf((2) (a) HO)
(15)
Ejer 076
Te Σjer D5t N,
(16)
To fit the Gaussian model to our data, we use the expectation maximization (FM) algorithm Xiang & Wang (003)..
In the estimation (E) step, we use the currently estimated parameter values to estimate the probability that the data æt is generated by each component:
Te TeN Nar |ites Cove)
Vie
(17)
Ei=1 (INlll Cove)
where N (zll, Cov) is the Gaussian density function.
In the maximization (M) step, we calculate the parameter values that would give the maximum likelihood expectation using Eqs. 14 15 16 to update 9fc with Y5c- The and steps are repeated until the value of the likelihood function converges or the maximum number of iterations is reached. With the finalized GMM parameters, we can use Eq (I7) to also calculate the refined soft target pseudo-label i
13


Reviews:
[{"review_id": "xzzG01Nzlt", "reviewer": "Reviewer_XSGj", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper presents a framework to reduce the domain gap between the source and target domains in the unsupervised domain adaptation task. The teacher network is used to increase the reliability of pseudo-labels, and the progressive mixup augmentation strategy is employed to generate the intermediate samples to bridge the large domain gap. This training strategy is called PMATS, which shows the state-of-the-art performance of the two UDA benchmark datasets.", "summary_of_the_review": "Even with the state-of-the-art performance, the insufficient description and the incremental framework are improper for the publication. The authors should improve the description of the motivation and the detailed framework for the next submission.", "strength_and_weaknesses": "The abstract and the experimental validation is written well, so the motivation of the proposed algorithm and its validation were successfully delivered. However, the detailed explanation of the methodology contains many typos and errors, which reduces the reliability of the designed framework. In addition, I have several questions and feedback as follows:\n\n1. Lack of experiments for large-scale dataset\n The proposed algorithm was validated by using two UDA benchmarks including Office-31 and Office-Home. Even though we can validate the domain-variant robustness of the UDA algorithms by using the two datasets, the datasets have an insufficient number of training samples to show the scale robustness. I recommend the authors perform additional experiments to show its scale robustness by using large-scale datasets such as the VisDA dataset or UDA segmentation scenarios. If the algorithm is validated through the segmentation task, we can verify the real-world applicability of the proposed algorithm also.\n\n2. Qualitative validation of gradual change\n  The authors insist that the proposed algorithm can generate the intermediate images and the intermediate images change gradually during the training phase. Unfortunately, it is hard to imagine the gradual change of intermediate samples just through the quantitative results. Although several graphical examples are visualized in some frameworks, qualitative examples would be more helpful to understand the meaning of the gradual change of the intermediate sample.\n\n3. Mechanism to increase the model robustness to noisy pseudo-labels\n  As I understand, the model becomes robust to the noisy pseudo-labels by using LMMD, but the LMMD is referred to in the previous study. Without the LMMD, only the intermediate sample using mixup augmentation is a novel approach to this framework. Then, which new module is helpful to increase the robustness of the noisy pseudo-labels? Or, the synergy between the LMMD and PMATS is special? If then, why?\n\n4. Difference with \"UDA with progressive domain augmentation\", 2020\n The overall approach and framework are very similar to those of \"UDA with progressive domain augmentation\". The only difference with the previous work is to utilize the mixup augmentation. I hope to recognize the novelty of the previous work.\n\n5. Role of p and q' in Eq. 6\n  Even though the function d_H is fed by p and q', there is no related notation in the following equation. We need more explanation for the equation.\n\n6. Errors in Eq 2 and Eq 5\n  In Eq.2, there is no \"=\", and the right part of the norm parenthesis is missing in Eq. 5. The first character after Eq.5 should be lower-cased (w).\n\n7. Motivation and analysis for d, alpha, and beta\n  Even though the authors present the previous approaches to determine the hyperparameters of mixup, the motivation of the proposed sampling mechanism is not exactly explained. Why is the new sample mechanism necessary? With this motivation, the empirical analysis for its validity should be added also.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "ckkEySDarzX", "reviewer": "Reviewer_dVDV", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper address UDA with the mean teacher model where the teacher model generate pseudo labels to supervise the student model, and the teacher model is updated as the moving average of the student model. Mixup is adopted to help enhance generalization performance. Experiments on two datasets verify the effectiveness. ", "summary_of_the_review": "This paper is below bar of this conference for the limited technical contribution and unconvincing experimental evaluation.", "strength_and_weaknesses": "### Strength\n1. The proposed method is technically sound. \n2. Experiments on two datasets verify the effectiveness\n\n### Weakness\n1. The novelty is limited. Mean teacher models have been used by quite a few works to address the UDA problem, e.g., [1,2,3,4]. I do not see a clear novelty in the proposed method of using a mean model to address UDA. Mixup is also a common technique to enhance generalizablity, and has been used to address UDA before [5,6]. I do not see a strong novelty here as well. \n2. The evaluation is not sufficient and convincing. Only two datasets are employed, while established practices in this area use at least three datasets for evaluation. \n\n\n- [1] MTTrans: Cross-Domain Object Detection with Mean Teacher Transformer\n- [2] Unbiased Mean Teacher for Cross-domain Object Detection\n- [3] Exploring Object Relation in Mean Teacher for Cross-Domain Detection\n- [4] CMT: Cross Mean Teacher Unsupervised Domain Adaptation for VHR Image Semantic Segmentation\n- [5] Adversarial Domain Adaptation with Domain Mixup\n- [6] Dual Mixup Regularized Learning for Adversarial Domain Adaptation", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "1: The contributions are neither significant nor novel.", "empirical_novelty_and_significance": "1: The contributions are neither significant nor novel."}, "extra_text": {}}, {"review_id": "c4UfeTRSdB", "reviewer": "Reviewer_6xGt", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes the progressive mixup augmented teacher-student (PMATS) training strategy to tackle unsupervised domain adaptation (UDA). In specific, a momentum encoder based teacher network is employed to assign pseudo labels to target domain samples, and a student network is trained to classify source domain samples and align the distribution of the source and the intermediate mixup domains. A mixup scheduling scheme is designed to progressively move the mixup domain from source to target along training. Authors demonstrate the performance gains of PMATS on Office-31 and Office-Home datasets.", "summary_of_the_review": "In summary, the current manuscript does not meet the high standard of ICLR on technical novelty, experimental verification and presentation. I give a rejection currently and recommend improving the paper according to the weakness. ", "strength_and_weaknesses": "Strength:\n\n1. In general, the proposed techniques (i.e., scheduled domain mixup and pseudo labeling with teacher network) are valid to perform progressive domain alignment between source and target domains. \n\n\nWeakness:\n\n1. Most of the proposed techniques have been studied in previous works, including domain mixup [a], momentum encoder for pseudo labeling [b] and clustering-based classification refinement [c]. Therefore, authors should better present which parts of the proposed method are novel and which parts follow previous works. Authors are suggested to add a subsection to the current draft to discuss the connection and difference against these technically related works. \n2. The experimental verification is not sufficient. It is glad to see improvements on Office-31 and Office-Home. However, considering the development of UDA at scale, the audience would definitely like to see the results on larger-scale UDA datasets under diverse domain shifts. It is unknown how the proposed method performs on the datasets like DomainNet and VisDA?\n\n\n\n[a] Xu, Minghao, et al. \"Adversarial domain adaptation with domain mixup.\" Proceedings of the AAAI Conference on Artificial Intelligence. Vol. 34. No. 04. 2020.\n\n[b] Huang, Jiaxing, et al. \"Category contrast for unsupervised domain adaptation in visual tasks.\" Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2022.\n\n[c] Tang, Hui, Ke Chen, and Kui Jia. \"Unsupervised domain adaptation via structurally regularized deep clustering.\" Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 2020.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "FbPcIq1zqR", "reviewer": "Reviewer_kMVd", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes a method for unsupervised domain adaptation wherein pseudo-labeling, teacher-student approaches, augmentation (mix-up) and MMD based alignment are put together in one framework. Given a target sample and source sample, a mix-up sample is first constructed. The label for the mix-up sample is created using a pseudo-label obtained from the teacher network. With the source, mixup data-target pair, an alignment loss based on LMMD and a source classification loss is used for updating the student network. The teacher network is a EMA of student network.  This approach seems to achieve good performance on some benchmark datasets.", "summary_of_the_review": "I feel like this is one of the papers that merely combines several ideas already used in domain adaptation, but does not do a good job of thoroughly studying their approach. Experimental validation is also weak. So, I am inclined towards rejecting the paper.", "strength_and_weaknesses": "The approach seems to combine several ideas from UDA which seems to work well and get the SOTA on some benchmark datasets.\n\nThis paper is what seems like several ideas from UDA put together in one framework. All these ideas - pseudo labeling, mixup augmentation, teacher-student approaches and LMMD have all been proposed separately and used in other papers. This paper merely combines all these approaches. While there is some merit in doing that, I feel like the paper has not done a good job of comparing how and why their combined approach improves over each of the independent methods. There is an ablation showing that the performance improves, but other than that, there is no proper visualization / justification as to why this is better. I can think of other ways of combining the approaches, but why this specific way? What happens to the feature space due to each of the loss terms? Why does each loss term help? Some visualizations explaining these might help the readers understand the method a bit better.\n\nI feel like the paper has not compared with some of the best SOTA approaches on these benchmarks. For instance, [1] achieves a performance of 73% and [2] achieves 72.2% accuracy on Office-Home benchmark with Resnet-50 backbone. These papers are not compared. Compares to these numbers, this paper only achieves a 0.1% performance improvement. The paper achieves 77% with DeiT backbone, claiming that as SOTA is not a but fair because with improved backbone, these approach might also get good performance.\n\nOne of my main concerns with this paper is the lack of evaluation on some hard domain adaptation benchmarks such as DomainNet. The performance on Office-31 is already saturated and the current methods only obtain marginal gains. DomainNet, on the other hand, is a comprehensive benchmark for evaluating adaptation algorithms.\n\nThe paper has so many components which make the system a bit complicated. I wonder whether this system would help in practice, or if there would be many hyperparameters to tune.\n\n[1] Liu et al., \"Cycle Self-Training for Domain Adaptation\", NeurIPS 2021\n[2] Prabhu et al., \"Selective entropy optimization via committee consistency for unsupervised domain adaptation\", ICCV 2021.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "CS8TL4P0uY", "reviewer": "Reviewer_CyYH", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes a strong engineering solution for the task of Unsupervised Domain Adaptation (UDA). The proposed method includes a student model and a teacher model. The latter is obtained by performing an exponential moving average on the former. The pseudo labels of the images in the target domain are obtained by refining the prediction of the teacher model. The authors further leverage the mixup augmentation to improve the performance.", "summary_of_the_review": "I think the current paper is not ready to be published on ICLR. See the weaknesses above.", "strength_and_weaknesses": "Strength:\n- The empirical performance is strong.\n\nWeaknesses:\n- This paper actually proposes a combination of some existing engineering techniques (e.g., EMA [*3], mixup [*1, *2], pseudo-label refinement [*4], Kmeans, and GMM), and tunes a group of hyper-parameters, yielding a strong performance on common benchmarks. The novelty and scientific contributions are trivial. This paper does not provide sufficient insights on the design of the proposed method. Personally, it is more like an experimental report instead of a paper.\n- At least several questions need to be answered. Why do we need the teacher-student architecture? Why do we need to refine the pseudo label? Why do we need to perform mixup? The simple \"it improves the accuracy\" is not a proper answer to these questions.\n- After reading the reviews from other reviewers and carefully reading the related works [*1-*4], I find that the aforementioned questions have been largely answered in [*1-*4]. Hence, now my major concern lies in the novelty of this paper. I think at least the authors should discuss the difference between their work and [*1-*4]. If there are some differences, the authors may consider investigating replacing their design with the techniques in [*1-*4], reporting the empirical changes, and analyzing why this happens.\n\n[*1] Adversarial domain adaptation with domain mixup. AAAI 2020.\n\n[*2] Dual Mixup Regularized Learning for Adversarial Domain Adaptation. ECCV 2020.\n\n[*3] Category contrast for unsupervised domain adaptation in visual tasks. CVPR 2022.\n\n[*4] Unsupervised domain adaptation via structurally regularized deep clustering. CVPR 2020.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "1: The contributions are neither significant nor novel.", "empirical_novelty_and_significance": "1: The contributions are neither significant nor novel."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject