Bidirectional View based Consistency Regularization for Semi-Supervised Domain Adaptation
Authors: Yuntao Du
娟 江
Hongtao Luo
Haiyang Yang
MingCai Chen
Chongjun Wang
Venue: Accepted by TMLR ()

Abstract: Distinguished from unsupervised domain adaptation (UDA), semi-supervised domain adaptation (SSDA) could access a few labeled target samples during learning additionally. Although achieving remarkable progress,  target supervised information is easily overwhelmed by massive source supervised information, as there are many more labeled source samples than those in the target domain. In this work, we propose a novel method BVCR that better utilizes the supervised information by three schemes, i.e., modeling, exploration, and interaction. In the modeling scheme, BVCR models the source supervision and target supervision separately to avoid target supervised information being overwhelmed by source supervised information and better utilize the target supervision. Besides, as both  supervised information naturally offer distinct views for the target domain, the exploration scheme performs intra-domain consistency regularization to better explore target information with bidirectional views. Moreover, as both views are complementary to each other,  the interaction scheme introduces inter-domain consistency regularization to activate information interaction bidirectionally. Thus, the proposed method is elegantly symmetrical by design and easy to implement. Extensive experiments are conducted, and the results show the effectiveness of the proposed method.

Published in Transactions on Machine Learning Research (03/2023)
Bidirectional View based Consistency Regularization for Semi-Supervised Domain Adaptation
Yuntao Du
National Key Laboratory for Novel Software Technology Deportment of Computer Science and Technology, Nanjing University
Juan Jiang
National Key Laboratory for Novel Software Technology Deportment of Computer Science and Technology, Nanjing University
Hongtao Luo
National Key Laboratory for Novel Software Technology Department of Computer Science and Technology, Nanjing University
Haiyang Yang
National Key Laboratory for Novel Software Technology Department of Compuler Science and Technology, Nanjing Universily
Mingcai Chen
Chenmc@@@maiilnjuu.cct...
National Key Laboratory for Novel Software Technology Deportment of Computer Science and Technology, Nanjing University
Chongjun Wang
chjwang@rju.edde edu.cn.
National Key Laboratory for Novel Software Technology. Deportment of Computer Science and Technology, Nanjing University
Jorum Reviewed on OpenReview: hMpp:: /tt///ppeereeeeeeeee www.:neteettootttttmmemmmmm W WVwnceBJJ.
Abstract
Distinguished from unsupervised domain adaptation (UDA), semi-supervised domain adap- tation (SSDA) could access few labeled target samples during learning additionally. Al- though achieving remarkable progress, target supervised information is easily overwhelmed by massive source supervised information, as there are many more labeled source samples than those in the target domain. In this work, we propose a novel method BVCR that better utilizes the supervised information by three schemes, i.e., modeling, exploration, and interaction. In the modeling scheme, BVCR models the source supervision and target su- pervision separately avoid target supervised information being overwhelmed by source supervised information and better utilize the target supervision. Besides, as both supervised information naturally offer distinct views for the target domain, the exploration scheme per- forms consistency regularization within each domain to better explore target information with bidirectional views. Moreover, as both views are complementary to each other, the interaction scheme performs consistency rogularization across domains to exchange comple- mentary information for belter learning. The proposed method elegantly symmetrical by design and easy to implement. Extensive experiments are conducted, and the results show the effectiveness of the proposed method.
*Corresponding author.
1

---

Published in Transactions on Machine Learning Research (03/2023)
1 Introduction
Deep neural networks have achieved remarkable progress in many fields when the train and the test samples are drawn from the same distributions Krizhevsky et al. (2 He el (2ITG). However, existing deep learning methods need many labeled samples to train the model while annotating the samples is difficult
and time-costing in some situations Chen all (201M). Besides, the model usually suffers from performance degradation when applying a trained model to a different environment because of domain shift Donahic of al (201));; Pann & Yang (20I). To solve such a challenge, unsupervised domain adaptation (UDA) has been proposed to address this distribution mismatch by transferring knowledge from the labeled source domain to the fully unlabeled target domain. UDA methods obtain model that generalizes well on the target domain
with no labeled examples from the target domain Pan & Yang (2010); Long et all (2015).
Recently, Semi-Supervised Domain Adaptation (SSDA) Saito al (2014) Jiang et al (2122) has attracted a
lot of attention due to its remarkable performance compared with Unsupervised Domain Adaptation (UDA)
Pan & Yang ( Long et al. ). As few labeled samples of the target domain are available in SSDA
and these labeled samples could effectively boost the performance by offering more supervised information.
To better utilize labeled larget samples and avoid the learned model being biased towards the source domain, line of methods learn invariant features to reduce domer- discrepancy via minimax entropy Saito et al
(2019). joint representations and risks alignment Li all (2021a), and bidirectional a adversarial training Jiang (220). Besides, Kim & Kim (2020) further identifies the intra-domain discrepancy problem within the larget domain, and solves this problem by adversarial perturbation. To better reduce the domain discrepancy, following methods reduce both the intra-domain and the inter-domain discrepancy for good adaptation by contrastive learning all (2021e); Singh (202T). samplo-to-same sclf-distillation Yoon eL AL (2021), and cross-domain adaptive clustering ei all (202113. Beyond these works, DECOTA Yang ( proposes to decompose the SSDA task into semi-supervised learning and unsupervised
et all
gt
domain adaptation task.
Although achieving remarkable progress, there are still some issues that are needed to be addressed to further improve the adaptation performance. For example, existing SSDA methods have not adequately utilized the labeled target samples and target supervised information is easily overwhelmed by massive source supervised information. As there are many labeled samples in the source domain but a few labeled samples in the target domain, therefore the learned discriminative features with a single model are biased towards the source domain Yang et al] (202T).. To deal with these issues, aim to emphasize the importance of the target supervised information and better utilize larget supervised information together with source supervised information.
Besides, we find the supervised information from two domains are both useful for the predictions on the unlabeled larget samples and complementary to each other. The large amount of labeled source samples is useful for learning discriminative features, while there are domain shifts between the source and the target domain. On the contrary, the labeled target samples share similar distribution with unlabcled target samples (with smaller intra-domain shifts), while the number of labeled samples is limited. As we can see, both supervised information naturally offer two complementary views for the target unlabeled target samples. Thus, we aim to explore the supervised information beneficial to the unlabeled target samples from bidirectional views and combine both information for better learning.
In this work, we propose simple yet effective method Bidirectional View based Consistency Regularization (BVCR) to achieve the above goals. BVCR is composed of three schemes, i.e., modeling, exploration, and interaction. In the modeling scheme, to avoid the target supervised information being overwhelmed by source supervised information, we propose to model both supervised information independently. We learn classification model for cach domain with the same but unshared structure such that the supervised information is modeled by the corresponding model in each domain. In the exploration scheme, we further explore both supervised information from different views and we resort to consistency regularization to achieve this goal as a simple yet effective solution to various label scare problems Sohn et all (2020). We design intra-domain consistency regularization (Intra-CR), which is applied to the unlabeled target samples and performed within each domain (model). Especially, the consistency regularization within
2

---

Published in Transactions on Machine Learning Research (03/2023)
the source model could help the source model explore the information relevant to the target domain. And the consistency regularization within the target model could help the target model further explore the internal structure within the unlabeled target samples, In the interaction scheme, as both views offer complementary information to each other, we design inter-domain consistency regularization (Inter-CR) to encourage information interaction between both views, which is also applied on the unlabeled larget. samples but is performed across domains (models). The Inter-CR is performed with an alternative supervised direction such that could not only promote the learning of each model but also achieve bidirectional transfer across domains.
As we can see, BVCR is symmetric method in the model architecture, the losses, and the importance of information by design. It is easy to implement without complex adversarial training as previous meth Saito et all (2 Li et all (202TT);; Kim & Kim (2n20). Extensive experiments are conducted, and the results show the effectiveness of the proposed method. To sum up, the main contributions are as follows:
We propose an SSDA framework based on bidirectional views to better utilize the supervised infor- mation via three schemes, i.e., modeling, exploration, and interaction.
In the modeling scheme, BVCR independently models both supervised information to avoid the target supervisod information being overwhelmed by source supervised information.
In the exploration scheme, we propose Intra-CR to explore the information beneficial to unla- beled target samples from both views separately.
In the interaction scheme, we propose Inter-CR to exchange complementary information from bidirectional views to promote the learning of cach model.
We conduct extensive experiments on three common datasets. The proposed method obtains better results than baseline methods in these real-world datasets and we also systematically study the components BVCR.
2 Related Work
2.1 Unsupervised Domain Adaptation
Existing unsupervised domain adaptation methods are divided into moment matching and adversarial domain adaptation. The goal of the former is to reduce the statistical distribution discrepancy across domains. The widely used statistical measurements include the first-order moment Long el. all (201)) the second-order
momentSum.et all (2IIr), and other statistical measurements all (2I))) Lee et all (2I19) Adversarial
domain adaptation reduces the distribution discrepancy in an adversarial manner. DANN Ganin ct all (20Th)
introduces a domain discriminator which plays a min-max game with the feat ure extractor by the domain
adversarial loss. MCD Saito et all (20185) introduces two classifiers as a discriminator to play a min-max game with the feature extractor. Considering the practical multi-class problem, MDD Zhang et all (2111 proposes a margin based theory and a new method based on this theory proposed. Following methods Cicek & Soatid (2019) adopt the multi-class discriminator, which considers the domain and class information simultaneously. However, recent works show that the well-known UDA approaches may not generalize well in SSDA scenarios Saito et all (2019) as they could not effectively utilize the supervised information in the target domain.
2.2 Semi-Supervised Domain Adaptation
Semi-supervised domain adaptation is practical scenario where few labeled samples available the target domain. Early methods focus on reducing inter-domain discrepancy. MME Saito et all (201g) proposes to align the distribution by minimax entropy. BiAT Jiang el al. (2020) aims to generate bidirectional adversarial examples to fill the domain gap. Furthermore, APE Kim & Kim (2020) introduces the intra- domain discrepancy issue within the target domain the SSDA problem and solves this issuc based on adversarial perturbation. The following methods aim to reduce both inter-domain discrepancy and intra- domain discrepancy simultaneously. ECACL Li el all (202Tc) proposes enhanced categorical alignment to
3

---

Published in Transactions on Machine Learning Research (03/2023)
deal with inter-domain discrepancy and consistency alignment to deal with intra-domain discrepancy. CLDA Singl (2121) designs a contrastive learning framework, where class-wise contrastive learning is adopted to reduce the inter-domain gap and instance-level contrastive alignment are used to minimize the intra- domain discrepancy. CDAC of (202TE) propose adversarial adaptive chustering loss to minimize both discrepancy. Another line of methods is inspired by the recent process of semi-supervised learning. DECOTA Yang et all (20241) decomposes SSDA into semi-supervised learning (SSL. sub-task and UDA sub-task, and it adopts co-training- based framework to learn from each other. MCL Yan ehal (2112) performs consistency learning multi-level to fully utilize consistency regularization in SSDA.
2.3 Semi-Supervised Learning
Deep semi-supervised learning (SSL.) methods include pseudo-labeling Berthelot el all (201);; Iscen et all (21 and consistency regularization. In this work, the most related methods are based on consistency regularization, a variant of self-training. It constrains the learned model to make consistent predictions the same example under variants of noises. Methods falling into this category differ in the level of noise, models for making two predictions, and the distance measurement. For example, VAT Miyato et all (201H)
generates noise in an adversarial manner. II-Model Laine & Aila (201 performs Gaussian noise, dropout, etc., to augment images. Mean Teacher Tarvainen & Valpola (2 instead maintains an Exponential Moving Average (EMA) of the models parameters. Further, Fixmatch Sohn ef all (2020) proposes to augment the sample with weak and strong augmentation to produce two variants of samples. The learning strategy of
our work is inspired by Fixmatch, but the goal is different. Fixmatch is used for improving the performance of supervised learning, while the goal of our method is to transfer knowledge from one domain to the other
domain.
3 Method
In SSDA, we have datasets sampled from two different domains. Let us consider three datasets in this content: a source dataset contains labeled source samples D. = {(1f, (() a labeled target datasct which contains labeled target samples D, = {()) = and an unlabeled target dataset that contains large number of images without any corresponding labels D. {(4)) Usually, we have N₂ < N₂ and N2 & N.. Labels vf and of of the samples from source and labeled larget sets correspond to the same label space, ie., y = (1,2, K}. SSDA seeks to learn classification model using D., Dr, and Dy and evaluate on D..
As shown in Figure II the proposed method contains two symmetric models, i.e., the source model Fs and the target model Fr. Both models are composed of the feature extractor and classifier, i.e., F== = fa o gs and F, = f2 9r- Different from previous methods Saito ct all (2MB); if all (20210), the structures of both models are the same but the parameters are not shared across the domain. In in the modeling scheme (section [.) by learning separate model for each domain on supervised samples, we could avoid target supervised information getting overwhelmed by source supervised information. In the exploration scheme (section B before exchanging the information from these two models, we use the unlabeled target samples to adapt the models by intra-domain consistency regularization, such that the source model could concentrate more on the domain-shared information (features) and less on source-pris pprivtt information (features), and the target model could learn better representations by utilizing the structural information within the unlabeled target samples. In the interaction scheme (section B exchange complementary information for better learning by inter-domain consistency regularization.
Especially, following the principle of the recent successful semi-supervised learning (SSL) algorithm Sohn all (2021), this proposed method utilizes image augmentation techniques with a "weak and strong augmen- tation" " stratogy to implement consistency regularization. uses prodictions on weakly augmented images, which are relatively more accurate Chen all (2022), correct the predictions on strongly augmented images. In the next subsections, we introduce each component in detail.
4

---

Published in Transactions on Machine Learning Research (03/2023)
Labeled source data
Classification loss
Pa Cyluu ( P=(Ul u()
Is
fs
Intra-CR
Ct
111
Unlabeled target data
Weak augmentation
Odd epoch
Inter-CR
Even epoch
A
Strong augmentation
Pe uz)
Labeled target data
St
ft
pr(ulu))
Intra-CR
Classification loss
F;
Figure 1: The model structure of BVCR. There are two symmetric models F, and F; in BVCR. The Intra-CR is performed within each domain (model) at the first stage, while the Inter-CR performed across domains (models) at the socond stage and it changes the supervised direction every epoch. Note that these two models are trained with different samples, thus, they would not become similar soon.
3.1 Modeling: Supervised Training
To model both the source and target supervised information independently, the corresponding models in both domains are trained to minimize the empirical risks on labeled samples as conventional supervised methods. Taking the source model F, as an example, the feature extractor Is maps a source sample x into the feature and the classifier f. would classify the feature into K categories, i.e., (ol=;)) F,(xf) aff (xt))) where is the softmax function. Then, cross-entropy loss Loe(-.) is adopted to minimize the empirical risk:
Lere W/D fce (vl P. xFf)
(1)
Similarly, the classification loss on the labeled target samples of the target model F, is,
Lar = (B Lee (ui (o(lt))
(2)
where pr(vl=)) F1(xt) (()( is the prediction target sample of of target model F, For the labeled samples in both domains, besides adopting the cross-entropy loss on the original images, also apply this loss on both the weakly augmented and strongly augmented images additionally such that the learned model can be more accurate.
3.2 Exploration: Intra-domain Consistency Regularization
To better explore the target information from bidirectional views, we introduce intra-domain consistency regularization (Intra-CR) in both domains.
For the target domain, the target model learned with few labeled target samples only contains partial information about the target domain, while the large amount unlabeled target samples contain more useful information (e.g., internal structure) that not learned by the model. Thus, performing the consistency regularization within the target domain could help the target model further explore the internal structure within the unlabeled target samples. Besides, by forcing the target model learned with labeled target samples to produce consistent predictions on the unlabeled target samples, the model could achieve implicit aligument between labeled and unlabeled target samples. Thus, it reduces the intra-domain discrepancy within the target domain.
Specially, we follow previous methods Sohn et (2020) and adopt the "weak and strong augmentation" strategy to implement consistency regularization. The weak and strong augmentation functions differ in the

---

Published in Transactions on Machine Learning Research (03/2023)
degree of image augmentation and more noise is added in strong augmentation. For each unlabeled target sample úi, both the weak augmentation a(.) and strong augmentation A(.) are applied:
u = a(us; of = A(us)
(3)
Generally, the predictions on weakly augmented images are more accurate than those on strong augmented images as the latter was more severcly damaged Chen all (2022). So we use the predictions of weakly augmented images supervise the predictions on the strong augmentation samples. The pseudo label of the weakly augmented image 0/ with target model F, is obtained by,
9
arg max Pe (y"))(")) yE(1.....K
(4)
where pr(ylu?") (((u)) is the prediction on the weakly augmented sample of target model Fr. And the consistency loss within the target domain is,
S Σ E..E. [[(mx(p((y))u")) 2 ce "0."", p(())
(5)
where I(() is the indicator function and (e(ylu)) (2)) = F((u)) is the prediction on the strongly augmented sample. To mitigate the affect of incorrect pseudo labels, only the samples with high confident predictions are used for loss computation and y is the confidence threshold.
For the source domain, could simply train source model with only labeled source samples without using any target samples and then apply this model for unlabeled target samples. Such strategy is similar to single-domain domain generalization Zhou et all (21zTa); [aag et all (20211); Saa_et_al (2122). While the results show that the performance of domain generalization methods worse than that of domain adaptation methods as difficult to expect the trained source model to generalize better in the target domain without socing any target samples Zhou al (2021a); [Wang et al. (202T).
Thus, we implement the Intra-CR the source model with unlabeled target samples instead of the source samples. Besides, the learned source model contains both information specific to the source domain and information shared with the target domain, the Intra-CR within the source domain could help the source model concentrate more on the information relevant to the larget domain. Similarly, we also use the predictions weakly augmented images to supervise the predictions on the strongly augmented samples and the consistency loss within the source domain is,
C.o Σ E,,,,, [(max(p.(y)lu)) 2 Cee (0i" (vlyy)())
(6)
where Ps (v (vvll(") (u,",, (vlu F. Fuf),, and 0.th arg ma..)e (vll""". Combining these two losses, the intra-domain loss is,
Cintro -.oo + from
(7)
3.3 Interaction: Inter-domain Consistency Regularization
Based on the observation that both views are useful for the target domain and are complementary to each other, we propose to make the interaction between them to promote the learning each model. Thus, we propose inter-domain consistency regularization (Inter-CR), which applied across models (domains) with unlabeled larget samples.
After augmenting original images with weak and strong augmentations, the weakly augmented images are fed to one model and the strongly augmented images are fed to the other model. As the predictions on weakly augmented images are more accurate and we pay equal importance lo each model (view), we change the supervised direction every epoch. Namely, we feed the weakly augmented images to the source model in even epoch and foed the weakly augmented images the target model in odd epoch. The same strategy is also applied to strongly augmented images and note that only the model fed with strongly augmented samples is optimized by Inter-CR.

---

Published in Transactions on Machine Learning Research (03/2023)
Taking the latter scenario as an example, the weakly augmented image uy is fed to the target model Ft, and we get the pseudo label by Equ El The consistency loss across domains at odd epoch is,
Cinter
E
[[(max(p(())=4") 2 Cce (0 P. (o(ylu)))
(8)
com
At even epoch, we change the supervised direction, and the consistency loss across domains is,
Linter
Σ _ [[(ma((p((vlu")) 2 22) Cee -p((llu)))
The proposed Inter-CR has several benefits. Firstly, to some content, the inter-domain consistency regular- ization can be seen as variant of co-training Blum Mitchell ((89н)). Through this mutual promotion, both models could learn complementary information from each other and be trained better. However, our method is different from the standard co-training framework, the bidirectional teaching is performed si- multaneously and repeated every epoch in co-training. Another strategy to perform Inter-CR is to train the models with simultaneous bidirectional training, ie., training with Eq & and g simultaneously every epoch just like standard co-training. Although this stratogy is also a possible training method, it may make the two models become similar quickly while BVCR could mitigate this problem by training alternately. Besides, we also experimentally find that alternative teaching achieves better than simultaneous bidirectional training and the comparison is shown in section experiments. Secondly, the Inter-CR can be seen as an implicit alignment across domains, thus reducing the inter-domain discrepancy across domains. Thirdly, previous SSDA methods concentrate on unidirectional knowledge transfer, i.e., from the source domain to the target domain, while the Inter-CR achieves the bidirectional transfer across domains Lin et all (202T)) Xie et al (2022). Thus, the domain shifts could be better bridged.
3.4 Overall
Strong and Weak Augmentation Strategy. Data augmentation is an effective way to prevent overfit- ting and plays an important role in machine learning. In this work, the weak augmentation functions apply random horizontal flip and random crop. RandAugment Cubnk et all (2022) and Cutout Devries & Taylor (2 constitute the strong data augmentation. More details about them are reported the appendix.
Training Objective. The overall learning process is composed of two ordinal stages in epoch and the pseudo code of BVCR is shown in appendix. At the first stage, the objective of our method is a weighted combination of the classification loss in both domains and the intra-domain consistency loss, The optimization problem is formulated as:
min Lare + Ltar + X
(10)
Se.Se-
where À is the trade-off syper-parameter. At the second stage, the learning objective is only the inter-domain consistency loss and the optimization problem at odd (even) epoch is formulated as Equ [ (Equ
min Cinter
(1la)
9 con
min Cinter St.ft
(11b)
con
3.5 Comparison with Existing Methods
In this subsection, we compare BVCR with related methods including Fixmatch Sohn cf all (2020), co- training Blum & Mitchell (1998), and MCL Yan all (2022).
Comnection with Fixmatch Sohn of all (2020): Fixmatch is a representative SSL approach based on consistency regularization. However, such hod only implements consistency within one do- main. Instead, we follow the same strategy to implement consistency while we not only implement consistency within each domain but also extend cross-domain scenario by Inter-CR effec- tively exchange the bidirectional supervised information. It is also noticed that although Intra-CR. is similar with Fixmatch, while it is the cornerstone of Inter-CR.
7

---

Published in Transactions on Machine Learning Research (03/2023)
Connection with Co-training Blum & Mitchell (1198): To some degree, the training strategy of Inter- CR could be seen variant of co-training- While co-training performs simultaneous bidirectional training but our strategy performs alternative bidirectional training. The proposed strategy could avoid the learned two models becoming similar quickly and the results in section experiments also verifies the superiorily of the proposed stralegy.
Comparison with MCL Nan et al (2122). MCL also adopts consistency regularization for SSDA and it implements consistency at multi level. However, needs extra class prototypes and complex optimal transport mapping Courly el al. (2 to perform intra and inter domain consistency. Instead, BVCR adopts unified sample-level consistency, which is more simple.
4 Experiments
4.1 Setups
Datasets: We evaluate the proposed method on several latest SSDA benchmarks including Office3
Saenko et. all (201I), Office-Home Venkateswara et al (2u1z)) and DomainNet Peng et al. (2014))
Office31 is a relatively small datasot and it is composod of three domains with 31 classes: Dslr (D), Webcam (W), and Amazon (A). Office- Home is a middle-size dataset, which contains domains including Real (R),
Clipart (C), Art (A), and Product (P) with 65 classes. DomainNet is the largest benchmark and there are domains with 126 classes. The four domains are Real (R), Painting (P), Clipart (C), and Sketch (S).
Baselines: We compare BVCR with previous methods, including,
S+T, which trains a single model by labeled source and target samples only.
SSL methods: including ENT Grandvalet & Bengid (2M1) which trains a single model with
labeled source and target and unlabeled target using standard entropy minimization.
UDA methods: including DANN Ganin et al ( ADR Saito et all 2018 CDAN Long et al.
(201 and BNM CD of (2020)
SSDA methods: including MME Saito et all (2019) Mcta-MME
Li & Hospedales (2020), BiAT
Jiang et al. (2020). APE Kim & Kim (2020), UODA Qin et all (2020),
CLDA Singh (2021)), DECOTA
Yang et all
(2021), and MCL Yan of (2022).
Implementation: Our method is implemented by PyTorch Paszke et all (201)) with open-source
library Zhou el all (20216). The experiments are conducted on Linux operal ing system with Tesla V100 (32G memory) and GeForce RTX 3090 (24G memory). We adopt ResNet-34, Alexnet, and VGG-16 pretrained on ImageNet Dong ct all take a two-layer MLP with randomly initialed weights. All the methods are evaluated under one-shot and three-shot settings, i.e., there are one or three labeled samples in each class. The top-1 accuracy is reported over all the unlabeled samples. At the inference time, we make prediction by equally averaging the source model F. and target model F,. The results of MCT is implemented by ourselves using the open-source code and the result of baselines are taken from their respective publications if the evaluation protocol is the same.
(20DH) as the backbone network of the fcature extractor. The two classifiers fs and f.
For the fair comparison, parts of target samples are split as valitaion set, and they are not used for training and only used for selecting the models and soper-parameters. The split stargey of Office-Hone and Domain- Net dataset following previous work Saito el all (2019) released on Github while the Office31 dataset is
hitpps/ www.:/wwwwcooo http://wwwww....eeeccce/////////
2 hitpp: http//wwww.... http://wwweaaaabbboooo//o/oooeee
3 http: www./.....uuuuube M3tiedddMM//DA/ M3SDA/
https:/ http://gthtbbcoo /tggg//ubbboooo/aaaaa/////////// Kaiyaagaaaaaauuuuaaa Dassi, pytorct
httpp::/gtthuub.com/ http://aoooooocceeeee//////////
ittps:/ hwtp://..gghuboommmmmmmm /ww.:oomm//iiioleeiia
/SDDA MMF
8

---

Published in Transactions on Machine Learning Research (03/2023)
Table Accuracy (%) on DomainNet under 1-shot and 3-shot settings using Alexnet as backbone.
R-C
R--P
P-C
C-4S
S-sP
R-+
P-I
Mean
Methed
I-shot
3-shot
I-shot
3-shot
L-shot
3-sbot
I-sact
3-shot
1-shot
3-shos
1-shot
3-shot
1-shot
3-shot
1-shot
3-shor
S+I
43.3
17.1
12.4
15.0
10.1
41.9
33.6
36.1
35.7
38.2 1
29.1
33.3
55.8
58.7
10.0
43.4
DANN
43.3
06.1
11.6
13.8
39.1
11.0
35.9
36.5
36.9
38.9
32.5
33.4
53.5
57.3
40.1
42.4
ADR
43.1
06.2
11.4
44.4
39.3
43.6
52.8
36.4
$3.1
38.9
29.1
92.4
55.9
57.3
39.2
42.7
CDAN
46.3
46.8
45.7
45.0
38.3
42.3
27.5
29.5
301.2
33.1 7
28. 8
31.3
56.7
38.7
39.1
41.0
ENT
37.0
46.5
35.6
42.6
26.8
40.4
18.9
31.1
15.1
29.0
18.0
29.6
52.2
60.00
29.1
39.8
MME
48.9
55.6
18.0
19.0
16.7
51.7
36.3
39.1
39.1
13.0
33.3
$7.9
56.8
60.7
41.2
28.2
Meta-MME
56.4
50.2
51.9
39.6
03.7
35.7
60.7
28.8
BIAT
51.2
58.6
49.2
50.6
44.0
32.0
97.7
41.9
39.6
42.1
37.2
42.0
56.9
58.8
45.5
49.4
APE
47.7
54.6
49.0
50.5
46.9
52.1
38.5
42.6
38.5
42.2
33.8
38.7
87.5
61.4
44.8
48.9
CLDA
56.3
50.9
56.0
57.2
50.8
54.6
42.5
47.3
46.8
51.4
38.0
42.7
64.4
67.0
50.7
54.3
DVCR
61.6
62.7
56.7
58.4
54.8
36.5
47.5
50.3
46.8
40.8
51.5
51.8
61.4
62.8
54.3
56.0
Table 2: Accuracy (%) on Office Home for under 1-shot and 3-shot settings using VGG-16 16G--66- as backbone.
Maon
75.3
10.5
51.6
72.7
ADB
29.7
60.2
71.8
68.7
57.4
75.7
60.9
09.8
371-
41.5
39.5
81.8
61.0
55.8
25.7
71.5
71.8
21.5
36.8
22.4
70.8
62.1
51.6
78.7
74.4
72.2
62.7
BNM
79.5
62.5
72.8
68.4
62.7
66.1
76.4
72.6
70.8
DECOTA
4033
64.8
75.5
36.8
71.1
42.5
73.1
57.8
63.3
78.9
66.1
78.6
88.8
72.3
38.8
73.6
56.5
61.9
BVCR
$5.821.1
81.910.44
68.0::0.52
78.750.1
48.9218
$7.751.1
7022608
SLTEL2
7299604
$7.85:03
13.42
65.3:20.1
Saler
S:T
45.5
78.6
63.0
72.1
BA3
09.4
$1.55
71.4
1652
70.4
62.3
DANN
56.1
77.9
63.7
71.6
52.4
06.3
09.5
58.0
72.8
68.7
06.4
69.8
63.9
ENT
48.3
81.6
65.5
78.6
66.8
66.9
75.0
45.6
78.8
72.9
99.1
77.0
64.8
APE
56.0
81.0
66.2
73.7
61.4
69.3
75.0
64.4
73.7
71.4
61.7
76.1
66.5
MME
56.0
82.9
66.7
70.7
53.0
69.2
76.7
54.9
71..8
72.9
61.1
78.3
67.6
UODA
57.6
83.6
67.5
77.7
64.9
61.0
77.7
36.4
70.7
73.8
61.9
78.4
68.9
DECOTA
59.9
83.9
67.7
77.5
57.77
80.7
766.0
$1.9
76.0
74.3
63.2
76.1
69.5
MCL
56.0
81.7
68.5
78.7
51.5
62.5
75.6
$2.0
74.6
73.0
64.0
77.7
67.7
BVCR
65.6100.3
84.210.14
69.1100.1
76.9:003
80.52:0.5
SL.800.7
79.9:10.9
58:10.8
75.64008
73:4:0.1
6L7780.2
79:0.8
70.3:003
randomly split by ourself and the split strategy is shown in the code. The specific number of train, validation, and test sets used in three datasets are present in Table in appendix.
Following ReMixMatch (Berthelot et all (2020), we also employ distribution alignment to modify the output of the model for encouraging the unlabeled target distribution of pseudo-labels to follow the source label distribution. We experimentally set the temperature parameter of softmax function as initially and increase it uniformly to 1.5 over the training process to make the predicted probability distribution smoother, avoiding the overconfidence of the model with training.
The learning rale,
X, and the initial value confidence threshold 7 are searched from {le-5, 5e-5, le-4, 5e-4, 5e-3), {0.5, 1.0, 2.0}, and (0.5, 0.8, 0.85, 0.9, 0.95) on validation set, respectively. The value of the confidence threshold is 0.9 and set as for all datasets. The number of epoch ans the random seed is all set to be 100 and 123 three datasets. We set À as 1 for all datasets and batchsize B to be 32, 16, and 16 for fffice-Home,, Office31, and DomainNet, respectively. We use SGD with momentum for optimization and set the momentum as 0.95. The learning rate is set to be 5e-4 and adjusted with cosine annealing strategy.
4.2 Results
This subsection shows the main results, and more results are shown in the appendix.
DomainNet. We summarize the performance of tasks on the DomainNet dataset in Table MI Using Alexnct backbone, our method outperforms the the sccond-best method CLDA by 3.6% and 1.7% under 1-shot and 3-shot settings on average. Our approach surpasses the well-known semi-supervised domain adaptation benchmarks methods in most tasks of the NomainNet dataset. Such improved performance
9

---

Published in Transactions on Machine Learning Research (03/2023)
Table Accuracy (%) on Office31 under 1-shot and 3-shot settings with Alexnet.
W-+
D-+A
Mean 1-shot
Method
1-shot
3-shot
1-shot
3-shot
3-shot
S+T
50.4
61.2
50.0
62.4
50.2
61.8
DANN
57.0
64.4
54.5
65.2
55.8
64.8
ADR
50.2
61.2
50.9
61.4
50.6
61.3
CDAN
60.3
48.5
61.4
49.5
ENT
50.7
64.0
50.0
66.2
50.4
65.1
MME
57.2
67.3
55.8
67.8
56.5
67.6
BIAT
57.9
68.2
54.6
68.5
56.3
68.4
APE
67.6
BVCR
62.6
66.3
58.3
70.0
60.5
shows that our approach better utilizes the supervised information in SSDA. In some tasks (e.g., P-R),, BVCR does not achieve the best performance. Tt. may be because that the domain discrepancy wetw task P-R is smaller than other tasks (with better S+T accuracy) and supervised information across domains are less complementary. Thus, BVCR gets less gain in these tasks. But with the proposed Intra- CR and Inter-CR, BVCR also achieves the second-bers best results.
FificeHHome, Table 0 reports the results of the Office- Home dataset under 1-shot and 3-shot settings with VGG-16. Our method shows the best performance in most of the scenarios. Specially, the improvement of mean accuracy is 1.4% and 1.0% compared with the second-best method in 1-shot and 3-shot, respectively. Our method achieves more performance gain when giving less labeled target samples. Similar to the results on DomainNet, our approach surpasses the state-of-the-art SSDA approaches in most of the adaptation tasks. Besides, In some domain adaptation cases under 1-shot, such as R-C and A-*,, the proposed method exceeds DECOTA and MCL by more than 3%.
Office31. The results of the Office31 dataset with Alexnet shown in Table & Our method also outper- forms other baselines in most tasks. The improvement is 4.2% compared with BiAT under the 1-shot setting and the results of 3-shot is slightly lower than BiAT by 0.3% on average. Improvement on this dataset is less than that of Office-Home and DomaiNet may be due to small size of samples.
4.3 Insight Analysis
Ablation study. The proposed method contains three losses (schemes):: the classification loss (modeling scheme), the Intra-CR loss (exploration scheme), and the Inter-CR loss (Interaction) We investigate the effect of each component by different combination during training. Besides, we also analyze the effectiveness of data augmentation (both strong and weak augmentation)
labeled samples. The results A-C and R-PP of Ollic-HHome dataset under 3-shot selting with VGG-16 are shown in Table EL As see, data augmentation is an effective method for improving generalization, and it improves the performance by 5.2% and 2.1% when only using classification losses. Following settings all adopt the data augmentation on labeled samples. As for the losses, after adding the Intra-CR loss, the performance improved by 3.1% and 2.4%, which implies that the Intra-CR can help the models better explore the target information from
the tasks
Table 4: Results (%) of ablation study on two tasks of Office- Home under 3-shot.
Augmentations on labeled samples
Exve
Liar
Cintra com
A-C
R +P
com
42.8
77.5
48.0
79.6
V
V
51.1
82.0
V
V
57.4
84.4
10

---

Published in Transactions on Machine Learning Research (03/2023)
S:T
T-56
Ours
SrT MME a Ours
90
80
85
70
60
30
75
40
30
70
A2C
3
10
20
Figure 2: (a) Analysis of supervised direction. (b) Impact of varying labeled samples.
different views. Moreover, by interacting both views with Inter-CR, the performance is further improved by 6.3% and 2.4% and the combination of all losses achieves the best results.
Effect of backbone. We analyze the network architecture ure with Alexnet, VGG-16, and ResNet-34 on Office-Home with four tasks under 3-shot. As shown in Table E. can that our BVCR outperforms all comparison methods on average accuracy and achieve 55.1%, 69.2%, and 73.4% mean accuracy with Alexnet, VGG-16, and ResNet-34, respectively. The results on other tasks are shown in appendix and our method also achieves better performance. The different results with different backbones demonstrate that our hod works well with multiple network structures.
Table 5: Accuracy (%) on Office-Home under 3-shot setting using Alexnet, VGG-16, and ResNet-34 as backbone networks.
Network
Method
R-P
P-A
A-C
C-R
Mean
S+T
66.7
36.1
38.8
54.3
49.0
Alexnet
APE
74.6
42.1
44.5
58.1
54.8
BVCR
74.0
41.9
45.3
59.4
55.1
S+T
78.6
55.9
47.5
69.7
62.9
VGG-16
APE
81.0
59.3
54.4
71.4
66.5
BVCR
84.4
61.7
57.4
73.4
69.2
S+T
80.8
63.5
54.0
68.3
66.7
ResNet-31
APE
86.2
66.1
63.9
76.8
73.3
BVCR
86.9
66.8
64.7
75.1
73.4
Effectiveness of bidirectional training. Inter- CR is imple- mented in an alternative bidirecitional training manner. Another strategy implement Inter-CR is lo perform Inter-CR with Equ B and Equ simultaneously like co-training and we denote this strat- egy as simultancous bidircctional training. We compare these two strategies on tasks A-C and P-R and the results are shown in Table F. We can see that alternatively training two models achieves better performance as it could effectively avoid these two models becoming similar quickly and encourage better information exchange.
Table Analysis of training strtegy.
A-C
R.+P
Simultaneously Alternatively
53.7
83.2
57.4
84.4
Analysis of supervised direction in Inter-CR. in Inter-CR, we compare our method with two variants on tasks A-CC and R-P of Office-Homn under 3-shot. 1) We always feed the weakly augmented samples the source model (denoted as S T). 2) We always feed the weakly augmented samples to the target model (denoted as T + S). Note that the supervised direction of the former is always from the source domain to the larget domain, and the direction of the latter is always from the target domain to the source domain. While the supervised direction is changed every epoch in BVCR. The results are shown in Figure (a).. As we can see, the proposed strategy achieves better performance for both tasks, which implics that bidirectional transfer across domains could yield better adaptation than unidirectional transfer.
To analyze the effect of changing supervised direction
Impact of the number of labeled samples. We compare the performance changes with the number of labeled target samples from to 20 for each class task R-PP Office-Home with VGG-16. As shown in
11

---

Published in Transactions on Machine Learning Research (03/2023)
Figure ((b), the performance of all methods is increased with more labeled target samples. Besides, BVCR consistently reaches the best performance for all the cases. The results show the benefit of our method for flexibly exploiting a different number of labeled target samples.
Feature visualization. To qualitatively evaluate the results, we adopt the t-SNE kan der Maaten Hintor (2008) to visualize the features produced by our method and the S+T method for adaptation task A A-C the Office-Home dataset under 3-shot setting. The results in Figure (aa)) and (b show that the
learned features exhibit favorable clustering structure, which implies the proposed methods could achieve cross-domain alignment and learn discriminative features.
(a) t-sne by S+T
(b) t-sne by BVCR
Figure 3: Feature visualization by t-sne
Comparison with Fixmatch We compare with Fixmatch on Office-Homi dataset with VGG backbone under 3-shot and the results are shown Table @ As we can see, Fixmatch achieves better performance than S+T but the improvement limited, ie., 0.5%, which implies that simply treating a semi-supervised domain adaptation(SSDA) problem a semi-supervised learning (SSL) problem trivial solution as this strategy could not deal with the domain distribution shifts effectively. BCVR considers the SSDA problem from complementary view, i.c., intra-domain supervised information and inter-domain supervised information, which could better deal with the domain discrepancy and promote model learning.
Table 7: Comparison with Fixmatch Office-Home for under 3-shot settings using VGG-16 as backbone.
Method
R-+C
R-PP
Rt-A
P-R
P-C
P~
A-PP
A-C
A-+Rt
C-RR
C-+A
C-PP
78.6
63.6
72.7
69.4
47.5
73.4
69.7
70.4
62.9
Fixmatch
52.3
78.7
63.7
72.9
47.7
54.9
70.5
19.7
72.8
69.4
57.7
70.0
63.4
BVCR
63.2
84.4
69.1
76.7
60.9
61.7
79.0
57.4
76.1
73.4
62.4
79.4
70.3
5 Conclusions
In this paper, we focus on SSDA. Considering that target supervised information is easily overwhelmed by massive source supervised information in existing SSDA methods, we propose BVCR to model the source supervised information and supervised target information separately. As both of the supervised information are complementary to cach other, we explore the target information from both views and make information interaction bet ween views. With the help of consistency regularization, we design Intra-CR and Inter-CR achieve the above goals. The experimental results show that the proposed method outperforms baselines. As for limitation, our method adopts pseudo labels for learning and it may be affected by noisy pscudo labels. Thus, it is further work to explore more flexible strategy to generate or filter unreliable pseudo labels than fixed threshold.
Acknowledgement
This paper supported by the National Natural Science Foundation of China (Grant No. 62192783, U1811462), the Collaborative Innovation Center of Novel Software Technology and Industrialization Nan- jing University.
12

---

Published in Transactions on Machine Learning Research (03/2023)
References
David Berthelot, Nicholas Carlini, lan J. Goodfellow, Nicolas Papernot, Avital Oliver, and Colin Raffel. Mixmatch: A holistic approach to semi-supervised learning. NeurIPS, 2019.
David Berthelot, Nicholas Carlini, Ekin Dogus Cubuk, Alexey Kurakin, Kihyuk Sohn, Han Zhang, and Colin Raffel, Remixmatch: Semi-supervised learning with distribution matching and augmentation anchoring. Tn International Conference Learning Representations (ICLR). 2020.
Ayrim Blum and Tom. Mitchell. Combining labeled and unlabeled data with co-training. In COLT 98, 1998.
Cheng Chen, Qi Dou, Hao Chen, Jing Qin, and Pheng-Ann Heng- Synergistic image and feature adaptation: Towards cross-modality domain adaptation for medical image segmentation. AAAI conference artificial intelligence, 2019.
Mingeai Chen, Yuntao Du, Yi Zhang, Shuwei Qian, and Chong-Jun Wang. Semi-supervised learning with multi-head co-training. AAAI conference on artificial intelligence, 2022.
Safa Cicek and Stefano Soatlo. Unsupervised domain adaptation via regularized conditional alignment. Proceedings of the IEEE/CVF international conference on computer vision (ICCV), pp. 1416-1425, 2019.
Nicolas Courty, Rémi Flamary, Devis Tuia, and Alain Rakotomamonjy. Optimal transport for domain adaptation. IEEE Transactions on Pattern Analysis and Machine Intelligence,
2017.
Ekin Dogus Cubuk, Barret Zoph, Jonathon Shlens, and Quoc V. Le. Randaugment: Practical automated data augmentation with a reduced search space. CVPRW, pp. 3008-3017, 2020.
Shuhao Cui, Shuhui Wang, et al. Towards discriminability and diversity: Batch muclear-norm maximization under label insufficient situations. IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2020.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, K. Li, and Li Fei-Fei, Imagenet: A large-scale hierarchical image database, In LEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2009.
Terrance Devries and Graham W. Taylor. Improved regularization of convolutional neural networks with cutout. ArXiv, Aas/1700.04452, 2017.
Jeff Donahue, Yangqing Jia, Oriol Vinyals, Judy Hoffman, Ning Zhang, Eric Tzeng, and Trevor Darrell. Decaf: A deep convolutional activation feature for generic visual recognition. In International Conference on Machine Learning (ICML), 2014.
Xinjie Fan, Qifei Wang, Junjie Ke, Feng Yang, Boqing Gong, and Mingyuan Zhou. Adversarially adaptive normalization for single domain generalization. 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 8204 8213, 2021.
Yaroslav Ganin, Evgeniya Ustinova, Hana Ajakan, Pascal Germain, Hugo Larochelle, François Laviolette, Mario Marchand, and Victor S. Lempitsky. Domain-adversarid training of neural networks. JMLR, 2016.
Yves Grandvalet and Yoshua Bengio. Semi-supervised learning by entropy minimization. In Advances in Neural Information Processing Systems, 2004.
Kaiming He, Xiangyu Zhang, et al. Deep residual learning for image recognition. IEEE IEEE/CVF CCF Conference on Computer Vision and Pattern Recognition (CVPR), 2016.
Ahmet Iscen, Giorgos Tolias, Yannis Avrithis, and Ondej Chum. Label propagation for deep semi-supervised lcarning. 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 5065- 5074, 2019.
Pin Jiang, Aming Wu, Yahong Han, Yunfeng Shao, Meiyu Qi, and Bingshuai Li. Bidirectional adversarial training for semi-supervised domain adaptation. In IJCAI, 2020.
13

---

Published in Transactions on Machine Learning Research (03/2023)
Tackyung Kim and Changick Kim. Attract, perturb, and explore: Learning feature alignment network for semi-supervised domain adaptation. In European Conference on Computer Vision (ECCV). 2020.
Alex Krizhevsky, Ilya. Sutskever, and Geoffrey E. Hinton. Imagenet classification with deep convolutional neural networks. Communications of the ACM, 2012.
Samuli Laine and Timo Aila. Temporal ensembling for semi-supervised learning. International Conference on Learning Representations (ICLR), 2017.
Chen- Yu Lee, Tanmay Batra, Mohammad Haris Baig, and Danicl Ulbricht. Sliced wassorstein discrepancy for unsupervised domain adaptation. In FEEE/CVE Conference on Computer Vision and Pattern Recognition (CVPR), 2019.
Bo Li, Yezhen Wang, Shanghang Zhang, Dongsheng Li, Trevor Darrell, Kurt Keutzer, and Han Zhao. Learn- ing invariant represcntations and risks for cuni-supervised domain adaptation. IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2021a.
Da Li and Timothy M. Hospedales. Online meta-learning for multi-source and semi-supervised domain adaptation. In European Conference on Computer Vision (ECCV), 2020.
Jichang Li, Guanbin Li, Yemin Shi, and Yizhou Yu. Cross-domain adaptive clustering for semi-supervised domain adaptation. IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). 2021b.
Kai Li, Chang Liu, Handong Zhao, Yulun Zhang, and Yun Raymond Fu. Ecacl: A holistic framework for semi-supervised domain adaptation. 2021 IEEE/CVF International Conference on Computer Vision (ICCV), pp. 8558 8567, 2021c.
Hong Liu, Jianmin Wang, and Mingsheng Long. Cycle self-training for domain adaptation. Advances in Neural Information Processing Systems, 2021.
Mingshong Long, Yue Cao, Jianmin Wang, and Michael I. Jordan. Learning transferable features with deep adaptation networks. International Conference on Machine Learning (ICML), 2015.
Mingsheng Long, Zhangjie Cao, et al. Conditional adversarial domain adaptation. In Advances in Neural Information Processing Systems, 2018.
Takeru Miyato, Shin ichi Maeda, Masanori Koyama, and Shin Ishii. Virtual adversarial training: A regular- ization method for supervisod and semi-supervised learning. T-PAMI, 41:1979-1998, 2019.
Sinno Jialin Pan and Qiang Yang. A survey transfer learning. TKDE, 22:1345 1359, 2010.
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Köpf, Edward Yang, Zach DeVito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. Pytorch: An imperative style, high perffrrrreeeece deep learning library. In Advances in Neural Information Processing Systems, 2019.
Xingchao Peng, Qinxun Bai, Xide Xia, Zijun Huang, Kate Sacnko, and Bo Wang. Moment matching for multi-source domain adaptation. Proceedings of the IEEE/CVP international conference on compuler vision (ICCV), 2019.
Can Qin, Lichen Wang, Qianqian Ma, Yu Yin, Huan Wang, and Yun Raymond Fu. Opposite structure learning for semi-supervised domain adaptation. SDM, 2020.
Kate Saenko, Brian Kulis, et al. Adapting visual category models to new domains. In European Conference on Computer Vision (ECCV), 2010.
Kuniaki Saito, Y. Ushiku, T. Harada, and Kate Saenko. Adversarial dropout regularization. International Conference Learning Representations (ICLR), 2018a.
14

---

Published in Transactions on Machine Learning Research (03/2023)
Kuniaki Saito, Kohei Watanabe, et al. Maximum classifier discrepancy for unsupervised domain adaptation. In IEEE/CVF Conference Computer Vision and Pattern Recognition (CVPR), 2018b.
Kuniaki Saito, Donghyun Kim, Stan Sclaroff, Trevor Darrell, and Kate Saenko. Semi-supervised domain adaptation via minimax entropy. Proceedings of the TEEE/CVF international conference on computer vision (ICCV), 2019.
Jian Shen, Yanru Qu, Weinan Zhang, and Yong Yu. Wasserstein distance guided representation learning for domain adaptation. In AAAI conference on artificial intelligence, 2018.
Ankit Singh. Clda: Contrastive learning for semi-supervised domain adaptation. Advances in Neural Information Processing Systems, 2021.
Kihyuk Sohn, David Berthelot, and et al. Fixmatch: Simplifying semi-supervised learning with consistency and confidence. Advances in Neural Information Processing Systems, 2020.
Baochen Sun, Jiashi Feng, and Kate Saenko. Return of frustralingly easy domain adaptation. In AAAI conference on artificial intelligence, 2016.
Antti Tarvainen and Harri Valpola. Mean teachers are better role models: Weight-averaged consistency targets improve semi-supervised deep learning results. In Advances in Neural Information Processing Systems, 2017.
Laurens van der Maaten and Geoffrey E. Hinton. Visualizing data using t-sne. Journal of Machine Learning Research, 9:2579 2605, 2008.
Hemanth Venkateswara, Jose Eusebio, et al. Deep hashing network for unsupervised domain adaptation. IEEE/CVF Conference Computer Vision and Pattern Recognition (CVPR), 2017.
Zijian Wang, Yadan Luo, Ruihong Qiu, Zi-Y Yu Huang, and Mahsa Baktash. Learning to diversify for single domain generalization. 2021 IEEE/CVF International Conference on Computer Vision (ICCV), pp. 814- 823, 2021.
Renchunzi Xie, Hongxin Wei, Lei Feng, and Bo An. Gearnet: Stepwise dual learning for weakly supervised domain adaptation. AAAI conference on artificial intelligence, 2022.
Zizheng Yan, Yushuang Wu, Guanbin Li, Yipeng Qin, Xiaoguang Han, and Shuguang Cui. Multi-level consistency learning for semi-supervised domain adaptation. In IJCAI, 2022.
Luyu Yang, Yan Wang, Mingfei Gao, Abhinav Shrivastava, Kilian Q. Weinberger, Wei-Lun Chao, and Ser- Nam Lim. Deep co-training with task decomposition for semi-supervised domain adaptation. Proceedings of the IEEF/CVF international conference on computer vision (ICCV), 2021.
Jeongbeen Yoon, Dahyun Kang, and Minsu Cho. Semi-supervised domain adaptation via sample-to-sample self-distillation. WACV, 2021.
Yuchen Zhang, Tianle Liu, et al. Bridging theory and algorithm for domain adaptation. In International Conference on Machine Learning (ICML), 2019.
Kaiyang Zhou, Ziwei Liu, Yu Qiao, Tao Xiang, and Chen Change Loy. Domain generalization: A survey. arXiv preprint arXiv:2103.0265 2021a.
Kaiyang Zhou, Yongxin Yang, Yu Qiao, and Tao Xiang, Domain adaptive ensemble learning. Transactions Image Processing (TIP), 2021b.
LEEE
15

---

Published in Transactions on Machine Learning Research (03/2023)
Table 8: The number of train set, validation set, and test set when each domain selected as target domain under 3-shot sctting.
Office31
Offie-Horee
DomainNet
Webcam
Amazon
Real
Clipart
Art.
Product
Real
Painting
Clipert
Sketch
Labeled target samples train
93
93
195
195
195
195
378
378
378
378
# Unlabeled target sumples in train set
312
600
2631
3967
3975
2037
4049
696022
30746
17947
23148
# Validation Test
312
609
2631
3967
3975
2037
0019
69602
30746
17947
25148
A
Setup
A.1
Augmentation and pseudo code
functions differ in the degree of image augmentation and more noise is added in strong augmentation. Specifically, the weak image transformation function applics random horizontal flip and random crop. Two augmentation techniques, namely RandAngment Cubuk et. all (2022) and Cutout Devries & 'Taylor (201Z), constitute the strong transformation function. In Rand.Aggment given number of operations are randomly selected from fixed of geometric and photometric transformations, such as affine transformation, color adjustment. Then they are applied images with stochastic magnitude. The operations of RandAugment are shown in Table MD The range is similar to the original version, so we do not elaborate on their meaning here. Cutout randomly masks out square regions of images to gray. Both of them are applied sequentially in the strong augmentation. Besides, the pscudo codc of BVCR shown in algorithm M
In this paper, the weak and strong augmentation
B More Results
More results DomainNet. We summarize the performance of 7 tasks the DomainNet dataset in Table g It is obvious that the proposed method outperforms bascline methods on most tasks. On average, our method outperformed the best-performed method APE by 1.2% under the 3-shot setting when ResNet-34 is adopted as the backbone network.
Analysis of training cost. We Lest the training time MME, UODA, and BVCR on task A-C of Office-Home dataset under 3-shot with the
running steps. The com- parison of running time for different methods is shown in Figure M As can see, BVCR needs more training time, namely 2.2 times of MME and 1.07 times of UODA. It because that BVCR has two models and the samples are fed into the models some times. But recent. SSDA methods also need much training time as they rely on contrastive learning adversarial training.
LODA
Figure Training cost of different methods
Table 9: Performance comparison in DomainNet under 3-shot setting using ResNet-34 as backbone network.
Method
R-C
R-P
P-C
C-S
S-P
R-$
P-R
Mean
S+T
60.0
62.2
59.4
55.0
59.5
50.1
73.9
60.0
DANN
59.8
62.8
59.6
55.4
59.9
54.9
72.2
60.7
ADR
60.7
61.9
60.7
54.4
59.9
51.1
74.2
60.4
CDAN
69.0
67.3
68.4
57.8
65.3
59.0
78.5
66.5
ENT
71.0
60.0
62.1
61.1
78.6
67.6
MME
72.2
69.7
71.7
61.8
66.8
61.9
78.5
68.9
UODA
73.2
69.4
64.2
80.8
71.2
Meta-MME
73.5
70.3
72.8
68.0
63.8
79.2
70.1
BIAT
74.6
67.5
62.1
78.6
69.7
APE
76.6
72.1
76.7
66.1
67.8
79.4
71.7
BVCR
76.4
73.9
71.3
69.2
71.2
69.4
79.0
72.9
16

---

Published in Transactions on Machine Learning Research (03/2023)
Algorithm Bidirectional View based Consistency Regularization for SSDA (BVCR)
Input: Source samples a., Target labeled samples Di Target unlabeled samples Du, Batch-size B, trade- off hyper-paranter X, and running epoches T.
Output: Trained target model F; and source model F..
1:
Initialize =
2: while T do
// The first stage
Sample mini-batch of B examples from D, and D; separately, B examples from Du. Train the feature extractors Osr gt. and classifiers Ss, fe by Equ 10.
// The second stage
7: Sample mini-batch of B examples from D..
8: if odd epoch then
9: Train the source feature extractor 9, and source classifier fa by Linter (Equ 8).
10: else
11: Train the target feature extractor 9r and target classifier fo by Linter (Equ 9).
12:
end if
13:
end while=0
Table 10: List of operations for strong transformations of the RandAugment. Two transformations are randomly chosen and performed with stochastic magnitude.
Operation
Range
Identity
[0.0, 1.0)
ShearX
[0.0, 0.3]
ShcarY
[0.0, 0.3]
TranslateX
[0.0, 0.33]
TranslateY
[0.0, 0.33)
Rotate
[0, 30]
AutoContrast
[0, 1]
Invert
[0, 1]
Equalize
[0, 1]
Solarize
[0, 110]
Posterize
[4,
Color
[0.1, 1.9
Brightness
Sharpness
[0.1, 1.9
17


Reviews:
[{"review_id": "ulmaHb4d9I", "reviewer": "Reviewer_MqnC", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "\nStrengths:\n1. Overall the paper is well written except for some minor issues ( listed below). \n2. The proposed method brings together interesting ideas like consistency regularization using weak and strong augmentation, training two models separately one for each domain, to address various challenges in semi-supervised domain adaptation. \n3. The method is empirically evaluated on several standard domain adaptation tasks and the results show that it achieves superior target domain accuracy in comparison to various baselines. Ablation studies justify the use of three losses viz. classification loss, inter and intra consistency regularization losses. \n\nWeaknesses:\n1. I did not get a good understanding of the loss functions and how exactly are they helping? More intuition and details in section 3.1, 3.2 and 3.3 could be helpful.\n2. What is the interaction among the losses? In particular the modeling scheme is supposed to help in avoiding the target supervised information being overwhelmed by source supervised information, do other losses defeat this purpose? \n3. The paper lacks novelty, the main ideas are from (Sohn et al. 2020). \n4. No error bars in the experiments makes it hard to judge the effectiveness of the method.\n5. Writing related : \n    a) $s$, $t$ in the superscript are used for source and target domain. Using superscript $s$ with $u_i$  to denote strong augmentation is a bit confusing in the beginning. \n    b) I found the use of loose cumbersome phrases (without formal/informal definitions) a bit uneasy to follow, especially in the abstract before I read the whole paper. e.g. “information interaction bidirectionally” , “source/target supervised information” , “inter/intra domain consistency regularization”. \n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper gives a semi-supervised domain adaptation method called BVCR. The setting considered has a large amount of labeled data in the source domain and small amount of labeled data in the target domain. The goal is to learn a model that has good performance on the unlabeled pool in the target domain.  Their method trains two models separately ( one for each domain). In addition to usual classification loss on the labeled samples, their method also adds consistency regularization to minimize intra-domain and inter-domain discrepancies that help in getting a better adapted model. For consistency regularization they utilize weak and strong augmentation from (Sohn et al. 2020). The proposed method is backed by empirical evaluation on benchmark datasets and compared against standard baselines. \n", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "1. Please provide error bars in all the experiments.\n\n2. More intuition and details on the loss functions will be helpful. \n\n3. One of the main pitches of the paper is that “target supervised information gets overwhelmed by source supervised information and the proposed method somehow address this issue” Can this claim be demonstrated more directly using some other evaluation metrics? An experiment in the earlier sections of the paper highlighting this problem with current methods can make the motivation well-grounded. \n\n4. Please address writing related concerns listed above. ", "broader_impact_concerns": "No", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "SuRdpQziMx", "reviewer": "Reviewer_m1Ua", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n1. Well written paper and has the required flow which explains the concept with ease.\n2. Good ablation studies on the analysis of the supervised direction, training duration of the method, effect of backbone architectures and performance of the method regarding the number of labeled target samples. \n3. Showcases the better usage of self-supervised consistency learning in comparison with adversarial training used in most domain adaptation methods.\n\nWeaknesses:\n1.  Lack of novel ideation for the problem statement. The algorithm seems to be inspired from a combination of multiple methods in conjuncture like Semi-supervised learning (FixMatch) and Co-training and applying the concept on multiple model outputs to achieve Inter-CR and Intra-CR adaptation stages in the SSDA problem. \n2. Lack of proper reporting of results for the main experimental results. The results for DomainNet dataset using AlexNet backbone compares with CLDA and misses out on comparing with DeCOTA[1] and MCL[4] while the results using ResNet34 backbone does not compare with them. Moreover there are related works which should have been used in the comparison of the results like DeCOTA[1], ATDOC[2], STAR[3] and MCL[4]. Resnet34 backbone results can also include the 1-shot results. Similarly results for Office-31 and Office-Home are evaluated on different backbones of AlexNet and VGG respectively. More results on different backbones would be required for better comparison. \n3.  Comparison to FixMatch is essential since the consistency regularization component is inspired from FixMatch. \n4. Interesting observation is that the method performs poorly when the source and the target domains are comparable to each other. While the method seems to work well when the shift between both the domains are significant, it should also have the capability to do so otherwise. \n\nRelated works:\n1. Yang L, Wang Y, Gao M, Shrivastava A, Weinberger KQ, Chao WL, Lim SN. Deep co-training with task decomposition for semi-supervised domain adaptation. In Proceedings of the IEEE/CVF International Conference on Computer Vision 2021.\n2. Liang J., Hu D., Feng J. Domain adaptation with auxiliary target domain-oriented classifier IEEE conference on computer vision and pattern recognition (CVPR) (2021)\n3. Improving semi-supervised domain adaptation using effective target selection and semantics. CVPRW 2021. \n4. Multi-level consistency learning for semi-supervised domain adaptation. IJCAI 2022. \n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "Summary: The authors propose an algorithm for semi-supervised domain adaptation using a combination of intra and inter consistency regularization using three different strategies consisting of modeling, exploration and interaction. While semi-supervised domain adaptation has shown immense improvement with the availability of small amount of labeled target data, they are overwhelmed with the large number of labeled source samples. This undermines the importance and usage of labeled target samples in an efficient way. In this paper, the authors propose to have complimentary viewpoints by using two different models for source and target data. In the modeling stage, the source model is trained on the fully labeled source data while the target data is trained on the labeled target data. In the exploration stage, the unlabeled target samples are augmented into weak and strong augmented versions and are passed through both the models. The weaker augmented samples are used as target labels while the stronger augmented sample predictions are used as model predictions. Then the intra domain consistency regularization is performed by both, the source and target models on the unlabeled target samples. The interaction stage involves using the weakly augmented sample predictions of one model and the strongly augmented sample predictions from the second model to perform inter-domain consistency regularization and this utilizes the complimentary viewpoint towards developing a better generalized model. ", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Addressing the above weakness points of [1] and [2] is critical for acceptance of the paper while the points [3] and [4] would make the paper stronger with better convincing evidence for the algorithm.  ", "broader_impact_concerns": "No.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "XUSkaya8xr", "reviewer": "Reviewer_E8Hz", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Pros**\n- The paper is easy to follow and well-organized.\n- Experiments on several datasets and analyses are sufficient.\n\n**Cons**\n- Limited Novelty: this paper is kind of like a combination of Fixmatch + co-training. Recent works have explored intra-domain consistency[1] and co-training[2]. Even though the authors specify the difference in the main text, I still consider this work as incremental. \n- Lack of Justification on Inter-domain Consistency. The source model and target model are trained in the same way at stage 1 Eqn 10. Then, the authors design a co-training loss for inter-domain consistency and claim it could ** exchange complementary information from\nbidirectional views**. However, I cannot see how these two models hold complementary information, considering they are trained exactly the same at stage 1. No further analysis or justification for verifying this claim. Also, the authors do not explain why iterative optimization (odd, even epochs) is necessary. Essentially, it is like making the model consistent with its moving average (In this case, the moving average only considers the model from the last epoch).\n- Experiment:\n1. Intra-domain consistency: authors should at least compare to a Fixmatch baseline and [1], which share large similarity with the proposed component. Also, the authors claim that they also do source domain consistency compared to Fixmatch. But the benefits of it do not reflect in Table 4.  How $L_{Intra-source}$+$L_{Intra-target}$ is better than $L_{Intra-target}$?\n2. Inter-domain Consistency: First, I recommend the authors show $L_{Inter} + L_{supervised}$ in Table 4. Although the performance of this paper is better than DECOTA in Table 1-2, it is unclear how the co-training part of this paper compares to DECOTA. Because $L_{Intra}$ plays a very important role in the performance. We cannot see how the proposed Inter-domain Consistency is better than other co-training strategies. \n\n[1] Kai Li, Chang Liu, Handong Zhao, Yulun Zhang, and Yun Raymond Fu. Ecacl: A holistic framework\nfor semi-supervised domain adaptation. 2021 IEEE/CVF International Conference on Computer Vision\n(ICCV)\n[2] Luyu Yang, Yan Wang, Mingfei Gao, Abhinav Shrivastava, Kilian Q. Weinberger, Wei-Lun Chao, and SerNam Lim. Deep co-training with task decomposition for semi-supervised domain adaptation. Proceedings\nof the IEEE/CVF international conference on computer vision (ICCV), 2021.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper deals with the problem of semi-supervised domain adaptation (SSDA). The authors propose a method BVCR that utilizes the supervised information by three schemes, i.e., modeling, exploration, and interaction. Specifically, it proposes Intra-CR to explore the information beneficial to unlabeled target samples and Inter-CR to exchange complementary information from bidirectional views to promote the learning of each mode", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "1. Clarification on Inter-domain Consistency w.r.t why you claim it could exchange complementary information from bi-views.\n2. Recommend adding a Fixmatch baseline\n3. Verify whether $L_{Intra-source}$+$L_{Intra-target}$ is better than $L_{Intra-target}$\n4. For ablation table, recommend adding  $L_{Inter} + L_{supervised}$.", "broader_impact_concerns": "None", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "tUF26NDmrI", "reviewer": "Reviewer_rief", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Pros\n1 The paper is well-organized.\n2 Clear and concise image of overall model structure.\n3 Sufficient analysis and ablation study.\n\nCons\n1 Novelty : This paper is like a combination of Fixmatch and co-teaching or a co-teaching version of MCL[1].\n2 In section 3.5, author say that BVCR’s unified sample-level consistency is more effective than MCL’s. Further experiment and analysis is needed.\n3 Results : The results are less convincing because the authors didn’t report the results of other methods include MCL in Table 3 and Table 8. It seems that MCL surpasses BVCR by up to 3.6% in the setting of dataset DomainNet and backbone ResNet34. Further analysis and experiment of this result is required. And more experiments in deeper models like ResNet34 and ResNet50 are recommended.\n\nMissing Reference\nSome papers related to semi-sipervised domain adaptation are missing.\n\nSemi-supervised heterogeneous domain adaptation: Theory and algorithms (TPMAI),\nSemi-supervised Domain Adaptation via Minimax Entropy (ICCV),\nFrustratingly easy semi-supervised domain adaptation (ACL),\n\n\n\n[1] Zizheng Yan, Yushuang Wu, Guanbin Li, Yipeng Qin, Xiaoguang Han, and Shuguang Cui. Multi-level consistency learning for semi-supervised domain adaptation. In IJCAI, 2022.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper focuses on the problem of semi-supervised domain adaptation (SSDA). The authors propose a new method called BVCR which model the source supervised information and supervised target information separately. BVCR designs Intra-CR and Inter-CR to explore the target information from both views and make information interaction between views.\n\n", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "1.  In section 3.5, author say that BVCR’s unified sample-level consistency is more effective than MCL’s. Further experiment and analysis is needed.\n\n2.  Further analysis and experiment of this result is required. And more experiments in deeper models like ResNet34 and ResNet50 are recommended.", "broader_impact_concerns": "No other impact concerns.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The authors work in the area of semi-supervised domain adaptation, where the goal is to perform domain adaptation with only a small number of labeled datapoints from the target domain. This is a hard problem in general, with a lot of moving parts. The authors put together several approaches that are often useful in semi-supervised learning in general. These include a mix of consistency regularization (for obtaining useful information from unlabeled data), both within and across domains. The outcome is a large pipeline that tries to get the most out of the available information.

The authors perform a substantial number of experiments, with reasonable performance improvements in a number of settings. The claims in the paper are substantiated to a solid degree.

All reviewers agree that there is some creativity in the way the authors brought multiple techniques, and that the experimental results are strong. Several reviewers liked the writing and presentation.

On the downside, the most common concern for reviewers is novelty. However, these concerns are not warranted for TMLR. Once these worries are removed from account, I believe the paper is in good shape.

For this reason, I believe the paper should be accepted. The authors are encouraged to clarify in writing any remaining points that tripped up some reviewers in terms of writing.
Decision: Accept (poster)