Cycle-consistent Masked AutoEncoder for Unsupervised Domain Generalization
Authors: Haiyang Yang
Xiaotong Li
SHIXIANG TANG
Feng Zhu
Yizhou Wang
Meilin Chen
LEI BAI
Rui Zhao
Wanli Ouyang
Venue: ICLR 2023 poster (2023)

Abstract: Self-supervised learning methods undergo undesirable performance drops when there exists a significant domain gap between training and testing scenarios. Therefore, unsupervised domain generalization (UDG) is proposed to tackle the problem, which requires the model to be trained on several different domains without supervision and generalize well on unseen test domains. Existing methods either rely on a cross-domain and semantically consistent image pair in contrastive methods or the reconstruction pair in generative methods, while the precious image pairs are not available without semantic labels. In this paper, we propose a cycle cross-domain reconstruction task for unsupervised domain generalization in the absence of paired images. The cycle cross-domain reconstruction task converts a masked image from one domain to another domain and then reconstructs the original image from the converted images. To preserve the divergent domain knowledge of decoders in the cycle reconstruction task, we propose a novel domain-contrastive loss to regularize the domain information in reconstructed images encoded with the desirable domain style. Qualitative results on extensive datasets illustrate our method improves the state-of-the-art unsupervised domain generalization methods by average $\textbf{+5.59\%}, \textbf{+4.52\%}, \textbf{+4.22\%}, \textbf{+7.02\%}$ on $1\%, 5\%, 10\%, 100\%$ PACS, and $\textbf{+5.08\%}, \textbf{+6.49\%}, \textbf{+1.79\%}, \textbf{+0.53\%}$ on $1\%, 5\%, 10\%, 100\%$ DomainNet, respectively.

Published as a conference paper at ICLR 2023
CYCLE-CONSISTENT MASKED AUTOENCODER FOR UNSUPERVISED DOMAIN GENERALIZATION
Haiyang Yangl.b. 15-- Xiaotong Li Shixiang Tang Feng Zhu?, Yizhou Wang*.
Meilin Chen 4.5, Lei Bai", Rui Z 8.6, Wanli Ouyang' NNanjing University Pekingg University 3The University of Sydney *Zhejiang University "Sensetime Research QQing Yuan Research Institute 'Shanghai Al Laboratory
hyyangesmail njju www..nu..uuuuu.ee... edw..n..nn cn, lixiaotong@stu. pkug edu.cn cn bai srisanshi@@maaln com, (yizhouwang, merlinis zzjuz.uea edu.e.ccccc stan39066uni..o wyw.ee....dddyyeeeeee...e edu.yyuduuuu.. (zhufeng, zhaorui @aenset ime. com wanli, www.ouusyannnnggnooeeeee edu.aa edu.au
ABSTRACT
Self-supervised learning methods undergo undesirable performance drops when there exists a significant domain gap between training and testing scenarios. Therefore, unsupervised domain generalization (UDG) proposed to tackle the problem, which requires the model to be trained on several different domains with- out supervision and generalize well on unseen test domains. Existing methods ei- ther rely on cross-domain and semantically consistent image pair in contrastive methods or the reconstruction pair in generative methods, while the precious im- age pairs are not available without semantic labels. In this paper, we propose a cycle cross-domain reconstruction task for unsupervised domain generalization the absence of paired images. The cycle cross-domain reconstruction task con- verts a masked image from one domain to another domain and then reconstructs the original image from the converted images. To preserve the divergent do- main knowledge of decoders in the cycle reconstruction task, we propose a novel domain-contrastive loss regularize the domain information in reconstructed im- ages encoded with the desirable domain style. Qualitative results on extensive datasets illustrate method improves the state-of-the-art unsupervised domain generalization methods by average +5.59 %. +4.52% +4.22 %. +7.02% on 1%. 5%, 10%, 100% PACS, and +5.08% +6.49% +1.79% % +0.53% on 1%, 5%. 10%, 100% DomainNet, respectively.
INTRODUCTION
Recent progresses have shown the great capability of unsupervised learning in learning good repre-
sentations without manual annotations (Doersch et al. 2015 Noroozi & Favaro 2016 Gidaris al.
2018 Chen et al.
2020b He et al. 2020 Chen et al. 2021 Zbontar et al. 2022 Caron et al. 2021
Tian et al. 2020;
Henaff 2020 Oord et al. 2018 Wu et al. 2018 Misra Maaten 2020 Caron
2023 However, they mostly rely the assumption that the testing and training domain should follow an independent and identical distribution. In many real-world situa- tions, this assumption is hardly held due to the existence of domain gaps between the training set and testing set in the real world. As a result, significant performance drops can be observed when deep learning models encounter out-of-distribution deployment scenarios (Zhuang al. 2019 Sariyildiz
al. 2020
Li et
2022
et al. 2021 Wang et al. 2021 Bengio et al.) 2019 Engstrom et al. 2019 Hendrycss & Dietterich, 2018 Recht et al. 2019 Su al. (019). A novel setting. unsupervised domain generalization (UDG) (Zhang et al. 2022 Harary et al. 2021 Yang et al. (022). is therefore introduced to solve the problem, in which the model trained on multiple unlabeled source domains and expected to generalize well unseen target domains.
"Equal Contribution
1 "oorrsponndig Author

---

Published as a conference paper at ICLR 2023
Existing unsupervised domain generalization methods rely on constructing cross-domain but mantically consistent image pairs to design the pretext tasks, i.e,, contrastive-base (Zhang et al. 2022 Harary al. 202T) and generative-baset based methods (Yang et al. 2022). The contrastive- methods aim to push the cross-domain positive pairs (samples of the same classes but from different domains) together and pull the negative pairs (the samples of different classes) apart (Fig. Ta)). In contrast, the generative-base method proposes a new cross-domain masked image reconstruction task recover the original image based on its syyle-transferre counterpart, which aims disentan- gle the domain information and obtain a coomain-nnvariann content encoder (see Fig. Tb)). Although they achieve great success in unsupervised domain generalization, how to fully exploit the multiple domain information and establish better input-target reconstructed pairs is still fundamental chal- lenge. fact, the reconstructed pairs are expected to cover more realistic and diverse cross-domain sample pairs, but without image annotations, those pairs can not be accurately obtained.
To tackle the challenge above, we propose generative- Masked AutoEncoder (CycleMAE). Our method designs novel cycle cross-domain reconstruction task for unsupervised domain generalization in the absence of paired images, which reconstructs multiple images from different domains in self-circulating manner with masked autoencoder. Specifically, the cycle cross-domain reconstruction task is to reconstruct an image under randomly masking from one domain to another domain, and then bring this generated counterpart back to its original domain, forming the self-circulating approach (as illustrated in Fig. [c)) In this way, can establish the cross-domain counterparts of multiple domains by the forward reconstruction, and construct high-quality input-target pairs for the backward reconstruction. Since the high-qualit, input-target pairs are from the model outputs, they can be more realistic than cross-domain pairs designed by manual rules. With these more realistic pairs, the models can be taught to better disen- tangle the domain invariiat features.
model named Cycle-consistent
We further observe that directly applying the cycle reconstruction tasks may underestimate the model's s ability disentangle the style information in the domain-specifii decoders. Without any su- pervision of the reconstructed images in the forward reconstruction, the model tends to take "short- cuts" that domain-specific decoders generate images in a similar domain to reduce the learning difficulty of encoders to extract content information in the backward reconstruction. To this end, we additionally introduce a domain contrastive loss to keep different decoders capturing divergent information. Specifically, we accomplish this regularization by pulling the samples with the same domain labels close and pushing the samples with different domain labels apart, which can force the decoder to capture less redundant information from each other and thus preferably disentangle the domain information.
To demonstrate the effectiveness of CycleMAF, massive experiments are conducted on the com- monly used multi-domain UDG benchmarks, including PACS eL 2017) and DomainNet (Peng al. 2019p. The experiment results demonstrate that CyCleMAE achieves new state-of-the-art and obtains significant performance gains on all correlated unsupervised domain generalization set- tings (Zhang et al., [2022). Specifically, CycleMAE improves the states-of-the of-the-art unsupervised do- main generalization methods by Average +5.59 %, +4.52% % +4.22%. +7.02% on 1 1%, 5%, 10%, 100% data fraction setting of PACS, and +5.08% % +6.49 %, +1.79% +0.53% % on 1 1%, 5%, 10%, 100% data fraction setting of DomainNet.
Our contributions are two-fold: (I) We propose a self-circulating cross-domain reconstruction task to learn domain-invarien features. (2) We propose a domain contrastive loss to preserve the domain discriminativeness of the transformed image in the cycle cross-domain reconstruction task, which features. Extensive experiments validate the ef- fectiveness of our proposed method by improving the ofate-of-the-art generative bassed methods by large margin. Related works will be elaborated in the Appendix 5.1
regularizes the encoder to learn
2 CYCLE-CONSISTENT MASKED AUTOENCODER
We now introduce our cycle-consistent Masked Autoencoder (CycleMAE) to learn domain-invariant features for unsupervised domain generalization based on a simple generative baseline method Di- MAE. Given images in a set of N different domain X {X1, X2 .... Xv ), our proposed CycleMAE consists of a transformer- based content encoder E and multiple Nomain-specifii transformer based decoders D= (Di, D2, ..., Dx J. where N is the number of domains in the training dataset and D,
2

---

Published as a conference paper at ICLR 2023
Pull
Reconstruction
Share weights
(a) Previous contrastive methods
Reconstruction
network
Y2
network
(b) Previous generative methods
(c) Our proposed methed
Figure I: Comparison of previous and the proposed methods for UDG task. (a): Contrastive methods aim pulling cross-domain but semantic similar images together. (b): Previous generative- based methods aim at reconstructing the original image based on the cross-domain images (generated by handcraft style-mix manners). (c): Our proposed CycleMAF leverages a self-circulating cross- domain reconstruction task for unsupervised domain generalization in the absence of paired data. (X and Y are a cross-domain but semantically the same image pair.)
the domain-specinn decoder of the image domain X, The key innovation of our proposed method lies in easing the difficulty of constructing the cross-domain but semantically same image pairs by proposing a novel cycle cross-domain reconstruction task and a domain contrastive loss. Given an image from the domain X, the image reconstruction cycle should be able bring back the original image, i.e.,
x x
- =x= s x,
where y = ( .,...., Y } the generated image encoded with the style of X, and & is the reconstructed image in X; after the cycle tross-domale reconstruction task. The domain contrastive loss is used to preserve the domain difference among y to regularize the domain-specifet decoder to learn domain style information.
the images generated by the forward transformation, y; D, (x)))
2.1 OVERVIEW OF CYCLEMAE
As shown in Fig. 2 the proposed CycleMAE is based DiMAE (Yang al. 2022 but differently undergoes two consecutive and reversible processes: the forward transformation process (illustrated by blue arrows) and the backward transformation process (illustrated by green arrows). For the forward transformation (Step 1), we mostly follow the process in DiMAE, which transforms a style- mixed image to images from different domains, i.e., - y. For the backward transformation process (Step 2), we transform the generated images y to & to reconstruct x. The cycle consistency loss, the domain contrastive loss along with the original cross-domain reconstruction loss in DiMAE are imposed in Step 3.
Stepl: Transform an image x to the an image x in X, we implement style-mix in (Yang et al.) (2022) to generate its style-mixed image v. Then we randomly divide into visual patches v, and masked patches Vm- We feed the visual patches V. into the encoder-decoder structure to generate the forward-transfore images y (y1, Y2,,
images Y1.y2:-. .... ...y..... NN (blue arrow). Given
Step 2: Transform the generated images y to the image & (green arrow). Given set of forward- transformed images y, we randomly divide each of them into visual patches yn and mask patches Ym- Then use the visual patches y. to reconstruct the image x, where & and x belong to the same domain.
Step 3: Network optimization with the cross-domain reconstruction loss, the proposed domain con- structive loss, and the cycle reconstruction loss. The parameters the encoder and domain- specific decoders D {D1, Dz...., Dx are optimized by the syclesreconttructionn loss (Eq. ( the domain contrastive loss (5) and the cross-domain reconstruction loss (Eq, 6 used in (Yang et al. 2022).

---

Published as a conference paper at ICLR 2023
Feccous Lcycle
772
Decoder (Real)
Masking
Content
StyleMix
Encoder
Decoder
(Sketch)
Ym
Yv
Y2
Masking
Forward Transformation Process
Backward Transformation Process
Figure 2:
The illustration of CycleMAE with the cross-domain Reconstruction task. The training process includes the forward and backward transformation process. For an image x, the forward transformation process transforms it to reconstruct the images in multiple domains. The backward transformation process then brings them to the original domain. use the cross-domain recon- struction loss, the cycle cross-domann reconstruction loss and the domain contrastive loss to super- vise the model optimization.
2.2 CYCLE CROSS- DOMAIN RECONSTRUCTION TASK
Previous contrastive-based methods and generative-based methods rely on high-quality cross- domain but semantically different image pairs to construct the pretext task. However, we argue that typical algorithms, e.g., using different image augmentations on the same image selecting nearest neighbors, can not preciously define such paired images because of the large domain gap. Therefore, we propose the cycle cross-domain reconstruction task to generate the cross-domain and semantic similar image pairs self-circulating way, i.e., transforming an image to other domains and then bringing the transformed images their original domain. Specifically, the overall cycle cross-domain reconstruction task consists the forward transformation process and backward trans- formation process. The cross-domain cycle reconstruction loss is utilized to minimize the distance between the generated image after the cycle process and the original image.
Forward transformation process. Forward transformation process transforms image into im- ages in multiple domains with the designed encoder and -omain-specific decoder. Specifically, given an image x in the domain X. we leverage the style-mix in Yang et al. (022)) to the style-mixed im- age v, and then randomly divide them into the visual patches and masked patches V, and Vn. The visual patches will be fed into the encoder to extract the content feature z, i.e.,
2 = E(v.).
With the domain-specifie decoders (Di, D., DN). and the masked patches query q, and the content feature z, we reconstruct the images y = (2. y multiple domains = (%1, X2,, ] Given a masked query q. the reconstructed image y; is defined as y, - Dz q).
Backward transformation process. Backward transformation process transforms the generated images (y1. Y2, .'.,y ) from multiple domains the image & in the domain of the original image to reconstruct the original image x. Specifically, we divide every image y; into the visible patches y and masked patches y! where
forward 1, .... N. We follow the implementation transformation by replacing the v. with y i.e., t, = Ely?). With the decoder where D generates images in the same domain X as set of masked patches queries {q'.q} "....... qN) for {y1, Y2, YN }. and the content feature {t1, .2.... tN }, we reconstruct the image & in X. Mathematically, given masked queries (r1, r.,... Fx } the reconstructed image &; from t; can be formulated as
x; D(t,, rz).
(3)
4

---

Published as a conference paper at ICLR 2023
DIMAE input
CycleMAE reconstruction inputs
Target
StyleMixed View
Sketch
Painting
Real
Figure 3: DiMAE leverages the heuristic methods in Yang al. 2022 construct the reconstruc- tion input with lots of artifacts. The CycleMAE utilizes the cross-domain reconstructed images as the reconstruction input, which are more natural and realistic.
Cycle reconstruction loss. The cycle reconstruction loss minimizes the distance between the masked patches and the corresponding masked patches of the original image x. Specifically,
Layele
X x)²
where &; are the outputs of the backward transformation process and x the original image.
Discussion. The high quality of the input-target reconstruction pair should be both natural and diverse. As shown in Fig. 3 the previous method, i.e., DIMAE, constructs the input-target pair by the heuristic style-mix method, which causes artificial defects in the reconstruction input. In contrast, our CycleMAE leverages images generated by domain-speciff decoders as the inputs for reconstructing the original image. Our reconstruction inputs do not rely the heuristic designs and utilize the information in the deep model, which more natural. Furthermore, owing to the multiple domain-specific designs of the generative-based UDG method, we can generate multiple and diverse inputs, instead of one image in DiMAE, for reconstructing the original image.
2.3 DOMAIN CONTRASTIVE LEARNING
Although the cycle cross-domain reconstruction task generate precious input-target reconstruc- tion pairs by the self-circulating process, we observe that directly applying the cycle reconstruction tasks decreases the model's 's ability disentangle the style information in domain domain-specific decoders, which will underestimate the ability of the encoder to learn supervision of the transformed images in the forward transformation process, the model tends to take "shortcuts" that domain-specific decoders generate images in a similar domain to reduce the difficulty of the encoder extract content information the backward transformation process.
features. Without any
Therefore, we propose domain contrastive loss to regularize different decoders to learn divergent domain style information, which could give the encode diverse inputs for the encoder to reconstruct the original image in the backward transformation process. Specifically, domain contrastive learning loss aims at pulling the samples in the same domain close and pushing the samples in different domains apart. As mentioned in (Cao et al.) (022). the semantic information of features increases as the layers go deeper. To effectively regularize decoder to learn divergent domain information the semantic information, utilize the features d from the first
and minimizes the influence decoder transformer layer for domain contrastive learning, where d, = D! (z, q). Here, D denotes the first transformer layer of DD Given intermediate feature d; of the decoder and the content the domain contrastive loss defined as
feature z defined in Eq. 2
N
exp(d: d, /T) log exp(d d: /T) ) exp(d. dT/r)'
Laamain Σ
(5)
where d denotes the features of samples in the same domain with d, and d, denotes the features of samples in the different domains of d, among the training batch. With the domain contrastive loss. the decoder can generate images with divergent domain styles, giving diverse input images for the encoder reconstruct the original images in the backward transformation process.
5

---

Published as a conference paper at ICLR 2023
2.4 OBJECTIVE FUNCTION
Our cycle cross-domain reconstruction task can be compatible to the original cross-domain recon- struction task in (Yang et al. 2022). Therefore, to take the most advantage of reconstruction tasks, we also preserve the original cross-domain reconstruction loss proposed in (Yang et al. (202) in our total objective function. Specifically, given an image from the domain X, the cross-domain reconstruction is formulated as
Crecons = -y;- x)²
(6)
Therefore, our total objective function can be formulated as
Lrecons + Or OL + B
where a and B are hyperparameters that can be empirically set to 2 and The sensitivity analysis of hyperparameters is presented in the Appendix 5.2.1
3 EXPERIMENTS
3.1 EXPERIMENTAL SETUP
Dataset. Two benchmark datasets are adopted to carry through these two settings. PACS, proposed by (Li al., 2017), widely used benchmark for domain generalization. It consists of four domains, including Photo (1,670 images), Art Painting (2,048 images), Cartoon (2,344 images), and Sketch (3,929 images) and each domain contains seven categories. (Peng et al. 2019 proposes large and diverse cross-domain benchmark DomainNet, which contains 586,575 examples with 345 object classes, including six domains: Real, Painting. Sketch, Clipart, Infograph, and Quickdraw.
Following the all correlated setting of DARLING (Zhang et al. 2022), we select Painting, Real, and Sketch as source domains and Clipart, Infograph, and Quickdraw as target domains for DomainNet. In this setting, we select 20 classes out of 345 categories for both training and testing, exactly following the setting (Zhang et al., 2022). For PACS, we follow the common setting domain generalization (Li al.) [2188] Rahman et al., |020 Albuqquuuee al.) (219)) where three domains are selected for self-supervised training. and the remaining domain is used for evaluation. Except above, the remaining experiments will be shown in Appendix 5.2.2
Evaluation protocol. We follow the all correlated setting of DARLING (Zhang et al. 2022) and divide the testing process into three steps. First, train our model in the unlabeled source domains. Second, we use a different number of labeled training examples of the validation subset in the source domains finetune the classifier the whole backbone. In detail, when the fraction of the labeled finetuning data is lower than 10% of the whole validation subset in the source domains, we only finetune the linear classifier for all the methods. When the fraction of labeled finetuning data is larger than 10% of the whole validation subset in the source domains, we finetune the whole network, including the backbone and the classifier. Last, we can evaluate the model on the target domains.
Implementation details. We use ViT-small as the backbone network unless otherwise specified. The learning rate for pre-training is 1.5 x 10-5 and then decays with a cosine decay schedule. The weight decay is set to 0.05 and the batch size is set to 256 x Ná, where N₂ is the number of domains the training set. All methods are pre-trained for 1000 epochs, which is consistent with the implementations in (Zhang et al.] 2022) for fair comparisons. The feature dimension is set to 1024. For finetuning, we follow the exact training schedule as that in (Zhang et al. 2022). We use MAE (He et al., 2021) unsupervised pre-training model in ImageNet for 1600 epochs to ensure labels are not available during the whole pretraining process.
3.2 EXPERIMENTAL RESULTS
We present the results in Tab, g (DomainNet) and Tab. (PACS), which shows that our CycleMAE achieves better performances on most tasks compared to previous methods. Specifically, CycleMAE improves the performance by +3.81 % and +5.08% for DomainNet on overall and average accuracy for 1% fraction setting. For 5% fraction setting, CycleMAE improves the previous methods by +5.53% and +6.49% % for DomainNet on overall and average accuracy. For 10% and 100% fraction settings, which adopts finetuning for the whole network, CycleMAE improves the state-of-art by

---

Published as a conference paper at ICLR 2023
(Linear evaluation)
Fraction
evaluation)
Methods
Clipari
Infograph
Quickdraw
Overall
Avg.
Clipari
Inlograph
Quickdraw
Overall
Aag.
MoCo V2
(Chea
20200
18.55
10.37
6.32
10.05
11.92
28.13
13.79
5.67
14.56
17.20
SimCI R
5.29
11.80
14.74
34.03
17.17
17.32
20.69
BYOL
(Gell al:
2020
6.21
3.48
4.27
4.45
4.65
9.60
5.09
6.02
6.49
6.90
AdCoo E00 ml AM2
16.16
12.26
5.65
9.57
11:36
10.77
18.65
7.75
15.44
19.06
MAE [e
22.38
12.62
10.50
13.51
15.17
32.60
15.28
13.43
17.85
20.44
DARLING 7Zisn ela Ai22
18.53
10.62
12.65
13.29
13.93
39.32
19.09
10.50
18.73
22.97
DIMAF
Tring of mil 20122
26.52
15:47
15.47
17.72
19:15
42.31
18.87
15,001
21.68
25.
CyCReMA
37.54
18.11
17.13
21.53
24.23
55.14
20.87
19.62
27.21
31.88
Label Fraction 10% (Full flactuning)
Label Fraction 100% (Full finetuiing)
Methods
Clipart
Inlograph
Quickdraw
Overall
Avg.
Clipart
Inlograph
Quickdraw
Overall
Aug.
MoCo V2
(Chen A0200
32,40
18.54
5,05
15.92
19.69
64.18
27
15.26
33.76
38.96
SinCI
37.11
19:87
12.33
19.45
23.10
69.72
30.56
37.47
42.29
BYOL
Khell et mi waay
14.55
8.71
5.95
8.46
9.74
54.44
23.70
20.42
28.23
32.86
AdCo
ml A1r1
32.25
17.96
11.56
17.53
20.59
62.84
26. 69
26.76
33.80
38.60
MAF
51.86
24,81
23.94
33.54
59.21
28.53
23.27
32.06
37.00
DARLING
Ziamar AP2
35.15
20.88
15.69
21.08
23.91
72.79
32.01
33.75
41.19
46.18
DIMAR
Taing mir 2012
70.78
38.16
27.39
39.20
45.41
83.87
44,99
39. 19.30
966
56.05
CycEMAL
74,87
38,42
28.32
40.61
47.20
85,39
14.31
40.03
50.46
56.58
Table 1: The cross-domain generalization results on DomainNet. All of the models are trained on Painting, Real, and Sketch domains of DomainNet and tested on the other three domains. The title of each column indicates the name of the target domain. All the models are pretrained for 1000 epochs before finetuned on the labeled data. Results style: best, second best.
Label Fraction 1% (Linear evaluation)
Label Fraction 5% (Linear evaluation)
Methods
Photo
Art.
Cartoon
Sketch
Avg.
Photo
Art.
Cartoon
Sketch
Avg.
MoCo V2
Chen
2020J
22.97
15.58
23.63
25,27
21.87
37.39
25.37
28.11
31.16
30.36
SimCLR VI
L Lien et a
20.433
30.94
17.43
30.16
25.20
25.93
$4.67
35.92
35.31
36.84
40.68
BYOL Grlll al. 2020
14.53
16.21
10.01
12.99
26.55
17.79
19.65
21.47
AdCo
Hü et 2021
26.13
17.11
22.96
23,37
22.39
37.65
28.21
28.52
30.35
31.18
MAE
He al
2021
30.72
23.54
20.78
24.52
24.89
32.69
24.61
27.35
30.44
28.77
DARLING
Zhang ctal.
2022
27.78
19.82
27.51
29.54
26.
44.61
39.25
36.41
36.53
39.20
DIMAE
Yang crai. 2022
48.86
31.73
25.83
32.50
34.23
50.00
41.25
34.40
38.00
40.91
CycleMAB (Ours)
52.63
36.25
35,53
34.85
63,24
39,96
42.15
36.35
45,43
Label Fraction
10% (Full linetuning)
Label Fraction 100%
(Full finetuning)
Methods
Photo
Art.
Cartoon
Sketch
Avg.
Photo
Art.
Cartoon
Sketch
Avg.
MoCo V2
Chen et M0200
44.19
25.85
33.53
24.97
32.14
39,86
28.38
48.89
34,79
43.03
SimCLR VI
54.65
37.65
46.00
28.25
41.64
67.45
43.60
54.48
34.73
50.06
BYOL
Gnll et nl. 2020
27.01
25.94
20.98
19.69
23.40
41.42
23.73
30.02
18.78
28.49
AdCo
46.51
30.21
31.45
22.96
32.78
58.59
50.19
30.45
42.26
MAE
He al 2021
35.89
25.59
33.28
32.39
31.79
36.84
25.24
32.25
34.45
32,20
DARLING
Zhang clal 2023
53.37
39.91
46.41
30.17
42.47
66
41.53
56,89
37.51
51.15
DIMAE
al 2048)
77.87
59.77
$7.72
39.25
58.65
78.99
63.23
59.44
55,89
64.39
CycleMAB (Ours)
85.94
67.93
59.34
38.25
62.87
90.72
75.34
69.33
50.24
71.41
The cross-domain generalization results in PACS. Given the experiment for each target domain run respectively, there is no overall accuracy across domains. Thus we report the average accuracy and the accuracy for each domain. The title of each column indicates the name of the domain used as the target. All the models are pretrained for 1000 epochs before finetuned on the labeled data. Results style: best, second best.
Table 2:
+1.79% and +0.53% for DomainNet on average accuracy and gets +1.41 % and +0.50% perfor- mance gains on overall accuracy. For PACS, CycleMAE improves the previous methods by +5.59% for 1% fraction setting and gets +4.52% performance improvements for 5% fraction setting. For 10% and 100% fraction finetuning settings, CycleMAE improves the state-of-the-art method Di- MAE by +4.22% and +7.02% on average accuracy.
We compare our CycleMAE with previous contrastive-based methods and state-of-the-art generative-bused methods, respectively. Previous contrastive learning methods, such as MoCo V2 (Chen et al.) (220d)). SimCLR V2 (Chen et al., 2020c), BYL (Grill et al, (220)) and AdCo (Hu) eet al. 2021). learn discriminative features using data augmentation to construct positive and neg- ative pairs, which can not explicitly bridge the domain gap by pulling positive pairs together and pushing negative pairs apart. DARLING constructs negative samples for any given queue according to the similarity between different domain samples, but it is still inaccurate and also needs data aug- mentation to construct positive and negative pairs in the same domain. However, the cross- dooss- pairs constructed in this way noisy, which leads to undesirable performance. Compared with contrastive based methods on UDG setting, our method outperforms all current state-of-the-arte and gets +9.49 % and +8.91 % performance improvements DomainNet on average accuracy using
7

---

Published as a conference paper at ICLR 2023
Precons
Lcycle
L domain
Accuracy
Loss Design
Accuracy
X
X
49.42
Cosine Distance
48.87
/
X
54.42
MMD Loss
55.21
/
56.87
Domain Contrastive Loss
56.87
Table 3: Effectiveness of each proposed component of CycleMAE.
Table 4: Comparison of different domain distance regularization loss.
Cross-domain Pairs
FT
BT
Accuracy
Decoder Layer
Accuracy
X
49.21
#1
56.87
J
48.78
#2
56.01
w/wo Heuristic Pairs
V
51.23
55,84
X
56.87
54.21
Table 5: Ablation study on heuristic pairs on forward and backward transformation process. The heuristic pair consists of an image and its style-mixed view.
Table 6: Comparison of features from dif- ferent layers for domain contrastive regu- larization. The decoders consist 8 layers.
linear evaluation protocol and improves the contrastive-batt methods by +23.29% and +10.40% on average accuracy using finetuning evaluation protocol. Furthermore, we find that CycleMAE gets higher performance gains on finetuning evaluation protocol, which consistent with other un- supervised learning researches (He et al. 2021 Xie et al. 2022).
We also compare our CycleMAE with the state-of-the-arr generative-bart method DiMAE, which heuristically generates the artificial style-mixed views to construct the cross-domain images. In contrast, CycleMAE utilizes a cycle reconstruction task to construct cross-domain pairs, where we can obtain diverse image pairs from multiple domains. Our cycleMAE shows -5.08%, +6.49% +1.79% %, +0.53% gains on 1 1%, 5%, 10%, 100% DomainNet.
3.3 ABLATION STUDY
To further investigate the components of proposed CycleMAE, conduct detailed ablation study on the proposed method. Specifically, we train Vit-Tiny for 100 epochs on the combination of Painting, Real and Sketch training set DomainNet, and evaluate the model using the linear evaluation protocol on Clipart.
Effectiveness of Each Component of CycleMAE. As shown in Tab. explore the effectiveness of each component in CycleMAE. The cycle reconstruction improves the accuracy by +5.00 %, and the performance be further improved with domain contrastive loss +2.45 %. Thus, the results verify that the proposed modules can benefit the encoder to learn more Nomain-invarii features.
Comparison of Features from Different Layers. Domain contrastive loss regularized on the output features of decoders. raises the question that the features after which layer should be used obtain the best performances. As shown Tab. 6 the performance decreases from the first layer to the last layer. As mentioned in (Cao et al. 2022). the semantic information of features increases as the layers go deeper, thus adopting the regularization on the first decoder layer can in turn force the features from the different decoders to share less redundancy. Therefore, we set the regularization after first decoder layer as the default protocol.
Effectiveness of Domain Distance Regularization Loss. Domain distance regularization is used to regularize the domain information in different decoders to be different. However, there are many designs of distance regularization. As shown in Tab. the domain contrastive loss regularization gets the best result. That is because the domain contrastive loss regularization pulls the samples from the same domain close and the samples from different domains far away. But other distance metrics only pull the samples from different domains far away without minimizing the intra-doman dis- tance. Thus, the domain contrastive loss improves other domain distance regularization +1.66 %
8

---

Published as a conference paper at ICLR 2023
MoCo V3
MAE
DiMAE
CycleMAE
Figure
The t-SNE Visualization on the feature distributions with different methods.
origin
painting
sketch
real
origin
painting
sketch
real
Figure 5: Reconstruction visualization of different decoders.
Comparison of Cross-domain Pairs for Construction. Paired data is important but absent in cross-domain datasets. Previous generative-based method (Yang et al. (2022) proposes the heuris- tic pairs which consist of an original image and its style-mixed view. Compared them with that proposed by (Yang et al. 2022) in Tab. [5] where FT/BT denote forward/buckwwrr transformation, respectively, we demonstrate that CycleMAE gets a +5.64% performance gain compared with other situations. Compared with using heuristic pairs in both FT and BT, the optimal setting which uses heuristic pairs only FT better because the heuristic images by style mix is not real. However, compared with not using heuristic pairs, using heuristic pairs in FT is important because heuristic pairs in FT can be a good starting point for the encoder in our CycleMAE to learn domain-invariant features and for the decoders learn domain-specific information.
3.4 VISUALIZATION
Feature Visualization. We present the feature distribution visualization of MoCo V3, MAE, Di- MAE, and CycleMAE in Fig. by t-SNE (Van der Maaten & Hinton 2008), where the features part of the combination of Painting, Real, and Sketch training set DomainNet. From the visual- ization, see the features from different domains in CycleMAE more uniform than other methods, which indicates that our CycleMAE shows better
characteristics.
present the reconstruction results of CycleMAE using ViT-Base for better visualization. From the visualization, we know that our CycleMAE can produce more realistic and diverse pairs that are semantically the same but from different domains.
Reconstruction Visualization. In Fig
4 CONCLUSIONS
In this paper, propose the Cycle-consistent Masked AutoEncoder (CycleMAB) to tackle the unsu- pervised domain generalization problem. CycleMAE designs a cycle reconstruction task to construct cross-domain input-target pairs, thus we generate more real and diverse image pairs and help to learn a content encoder to extract domain-invarian features. Furthermore, we propose novel domain contrastive loss help CycleMAE better disentangle the domain information. Extensive experiments on PACS and DomainNet show that CycleMAE achieves statc-of-the performance, verifying the effectiveness of method.

---

Published as a conference paper at ICLR 2023
ACKNOWLEDGMENTS
Majority of this work was completed during Haiyang Yang's internship at Sensetime under the mentorship of Feng Zhu. We also extend our gratitude to Qingsong Xie for his contribution the part of this idea.
REFERENCES
Isabela Albuquerque, João Monteiro, Mohammad Darvishi, Tiago H Falk, and Ioannis Mitliagkas. Generalizing unseen domains via distribution matching. arXiv preprint 191iv:I9111 1.00804, 2019.
Yoshua Bengio, Tristan Deleu, Nasim Rahaman, Nan Rosemary Ke, Sebastien Lachapelle, Olexa Bilaniuk, Anirudh Goyal, and Christopher Pal. meta-transfer objective for learning to disentan- gle causal mechanisms. In ICLR, 2019.
Shuhao Cao, Peng Xu, and David Clifton. How to understand masked autoencoders. arXiv preprint arXiv:2202.03670, 2022.
Mathilde Caron, Ishan Misra, Julien Mairal, Priya Goyal, Piotr Bojanowski, and Armand Joulin. Unsupervised learning of visual features by contrasting cluster assignments. In NeurIPS, 2020.
Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and
Armand Joulin. Emerging properties in self-supervised vision transformers. In ICCV, 2021.
Mark Chen, Alec Radford. Rewon Child, Jeffrey Wu, Heewoo Jun, David Luan, and llya Sutskever. Generative pretraining from pixels. In /CML. PMLR, 2020a.
Ting Chen, Simon Korbblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. ICML. PMLR, 2020b.
Ting Chen, Simon Kornblith, Kevin Swersky, Mohammad Norouzi, and Geoffrey Hinton. Big self- supervised models are strong remi-supervised learners. arXiv preprint MrXiv:2006.10029 2020c.
Xinlei Chen, Haoqi Fan, Ross Girshick, and Kaiming He. Improved baselines with momentum contrastive learning, arXiv preprint rxXi:2003304227, 2020d.
Xinlei Chen, Saining Xie, and Kaiming He. An empirical study of training self-supervised vision transformers. In ICCV, 2021.
Carl Doersch, Abhinav Gupta, and Alexei A Efros. Unsupervised visual representation learning by context prediction. In ICCV, 2015.
Logan Engstrom, Brandon Tran, Dimitris Tsipras, Ludwig Schmidt, and Aleksander Madry. Ex- ploring the landscape of spatial robustness. In ICML. PMLR, 2019.
Spyros Gidaris, Praveer Singh, and Nikos Komodakis. Unsupervised representation learning by predicting image rotations. arXiv preprint 18X.v:1803.07728, 2018.
Jean-Bastien Grill, Florian Strub, Florent Altché, Corentin Tallec, Pierre Richemond, Elena Buchatskaya, Carl Doersch, Bernardo Avila Pires, Zhaohan Guo, Mohammad Gheshlaghi Azar, et al. Bootstrap your own latent-a new approach to self-supervised learning. NeurIPS, 33, 2020.
Sivan Harary, Eli Schwartz, Assaf Arbelle, Peter Staar, Shady Abu-Hussein, Elad Amrani, Roei Herzig, Amit Alfassy, Raja Giryes, Hilde Kuehne, Dina Katabi, Kate Saenko, Rogerio Feris, and Leonid Karlinsky. Unsupervised domain generalization by learning bridge across domains, 2021. URL https: //tp://aaxivvvo.o org/ abs /2112. 02300
Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. Momentum contrast for unsupervised visual representation learning. In CVPR, 2020.
Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick. Masked autoencoders scalable vision learners. arXiv preprint arXiv:2111.0633 2021.
10

---

Published as a conference paper at ICLR 2023
Olivier Henaff. Data-efficient image recognition with contrastive predictive coding. In ICCV, pp- -182-4192 PMLR, 2020.
Dan Hendrycks and Thomas Dietterich. Benchmarking neural network robustness to common cor- ruptions and perturbations. In ICLR, 2018.
Qianjiang Hu, Xiao Wang, Wei Hu, and Guo-Jun Qi. Adco: Adversarial contrast for efficient learning of unsupervised representations from self-trained negative adversaries. In CVPR, 2021.
Gustav Larsson, Michael Maire, and Gregory Shakhnarovich. Learning representations for auto- matic colorization. In ECCV. Springer, 2016.
Gustav Larsson, Michael Maire, and Gregory Shakhnarovich. Colorization as a proxy task for visual understanding. In CVPR, 2017.
Da Li, Yongxin Yang. Yi-Zhe Song, and Timothy M Hospedales. Deeper, broader and artier domain generalization. In ICCV, pp. 5542-5550, 2017.
Xiaotong Li, Yongxing Dai, Yixiao Ge, Jun Liu, Ying Shan, and LINGYU DUAN. Uncertainty modeling for outti--diitributionn generalization. In International Conference Learning Repre- sentations, 2022. URL
net Etrumriruiiii 6HN7 LHyzGgC
Xiaotong Li. Zixuan Hu, Jun Liu, Yixiao Ge, Yongxing Dai, and Ling- Yu Duan. Modeling uncertain feature representation for domain generalization, 2023. URL https://azxiv. orgp:bi.rrrg//s////////
Ya Li, Xinmei Tian, Mingming Gong, Yajing Liu, Tongliang Liu, Kun Zhang, and Dacheng Tao. Deep domain generalization via conditional invariant adversarial networks. ECCV, 2018.
Ishan Misra and Laurens van der Maaten. Self-supervised learning of pretext-invariant representa- tions. In CVPR, 2020.
Mehdi Noroozi and Paolo Favaro. Unsupervised learning of visual representations solving jigsaw puzzles. ECCV. Springer, 2016.
Aaron van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predic- tive coding. arXiv preprint arXhv:1807.03748. 2018
Deepak Pathak, Philipp Krahenbuhl, Jeff Donahue. Trevor Darrell, and Alexei A Efros. Context encoders: Feature learning by inpainting. In CVPR, 2016.
Xingchao Peng, Qinxun Bai, Xide Xia, Zijun Huang, Kate Saenko, and Bo Wang. Moment matching for multi-source domain adaptation. ICCV, 2019.
Mohammad Mahfujur Rahman, Clinton Fookes, Mahsa Baktashmotlagh, and Sridha Sridharan. Correlation-aware adversarial domain adaptation and generalization. Pattern Recognition, 100, 2020.
Benjamin Recht, Rebecca Roelofs, Ludwig Schmidt, and Vaishaal Shankar. Do imagenet classifiers generalize to imagenet? In ICML. PMLR, 2019.
Mert Bulent Sariyildiz, Yannis Kalantidis, Diane Larlus, and Karteek Alahari. Concept generaliza- tion in visual representation learning. In ICCV, 2021.
Jiawei Su, Danilo Vasconcellos Vargas, and Kouichi Sakurai. One pixel attack for fooling deep neural networks. TEVC, 23(5), 2019.
Yonglong Tian, Chen Sun, Ben Poole, Dilip Krishnan, Cordelia Schmid, and Phillip Isola. What makes for good views for contrastive learning? arXiv preprint 200::22005.00243 2020.
Laurens Van der Maaten and Geoffrey Hinton. Visualizing data using t-sne. JMLR, 9(11), 2008.
Yizhou Wang. Shixiang Tang, Feng Zhu, Lei Bai, Rui Zhao, Donglian Qi, and Wanli Ouyang. Revisiting the transferability of supervised pretraining: an mlp perspective. arXiv preprint /2. (0.::2/12.00499 2021.
11

---

Published as a conference paper at ICLR 2023
Zhirong Wu, Yuanjun Xiong, Stella X Yu, and Dahua Lin. Unsupervised feature learning via non- parametric instance discrimination. CVPR, pp. 3733-3742, 2018.
Zhenda Xie, Zheng Zhang, Yue Cao, Yutong Lin, Jianmin Bao, Zhuliang Yao, Qi Dai, and Han Hu. Simmim: A simple framework for masked image modeling. In CVPR, 2022.
Haiyang Yang, Meilin Chen, Yizhou Wang, Shixiang Tang. Feng Zhu, Lei Bai, Rui Zhao, and Wanli
Ouyang. Domain invariant masked autoencoders for self-supervised learning from multi-domains, 2022. BtRt:ttss:///tt
org/ abs httpg/aaaa22.2...ee// 2t1///22500002
Zili Yi, Hao Zhang, Ping Tan, and Minglun Gong. Dualgan: Unsupervised dual learning for image- to-image translation.
Jure Zbontar, Li Jing, Ishan Misra, Yann LeCun, and Stéphane Deny. Barlow twins: Self-supervised learning via redundancy reduction. In ICML. PMLR, 2021.
Xingxuan Zhang, Linjun Zhou, Renzhe Xu, Peng Cui, Zheyan Shen, and Haoxin Liu. Towards unsupervised domain generalization. CVPR, 2022.
T. Zhou, P Krähenbühl, M. Aubry, Q. Huang, and A. A. Efros. Learning dense correspondence via 3d adigudded cycle consistency. In CVPR, 2016.
Jun Yan Zhu, Taesung Park, Phillip Isola, and Alexei A Efros. Unpaired image-to-ime to-image translation using cycle-consistent adversarial networks. In Computer Vision (ICCV), 2017 IEEE Interna- tional Conference on, 2017.
Fuzhen Zhuang, Zhiyuan Qi, Keyu Duan, Dongbo Xi, Yongchun Zhu, Hengshu Zhu, Hui Xiong, and Qing He. A comprehensive survey transfer learning, 2019. URL https: ///ari/axvvvv org 1911. 02685
5 APPENDIX
5.1 RELATED WORK
Self-supervised Learning. Self-supervised learning (SSL) is introduced learn powerful semantic
representations from massive unlabeled data. Recent SSL methods can be divided into two cat-
egories: discriminative (Noroozi & Favaro, 2016: Gidaris et al.] 2018 Chen et al., 2020b Grill et al.) 22200 He al.] 2020|/hhen
al. 2020d] 2021 Zbontar al. [022]] Caron al. (022] and
generative methods (Pathak et al.] 2016 Larsson et al. 2016 2017 He et al. 2021). Among the dis- criminative methods, the early works try to design some auxiliary tasks, like jigsaw puzzle (Noroozi
& Favaro 2016) and rotation prediction (Gidaris et al.) 2018), to learn semantic representations.
The recent works are mainly based on contrastive loss (Chen et al., 2020b Grill et al., 2020 He et al. 20200 Chen et al. 2020d [021] Zbontar et al., | Caron et al. 2021)) which models the similarity and dissimilarity by constructing positive pairs and negative pairs, and learns semantic representations by pulling the positive pairs close and push the negative pairs away.
Generative methods depend on the design of the encoder-decoder structure. Recent methods utilize masked image modeling (MIM) to recover the original images by the masked ones with vision transformer. iGPT (Chen et al., 2020a) predict the pixel value which are from the pixel sequences. MAE (He et al. (021). recent state-of-the the-art method, recovers the input images based on a few patches the images for pre-training the autoencoder, which captures semantic representations in this way.
However, these SSL methods only focus on the situation where the training and testing datasets share the same data distribution and there may be performance drops when training and testing datasets exist domain gap. Thus we propose a generative-base method that takes the domain gap into consideration.
Unsupervised Domain Generalization. Despite the success, domain generalization still relies on the fully labeled data. To ease the annotation burden, Unsupervised Domain Generalization (UDG) is proposed as a novel generalization task that trains with unlabeled source domains and tests with target domains that serve domain shifts with training domains. Derived from contrastive learning.
12

---

Published as a conference paper at ICLR 2023
57
56
56.13
55.25
56
54
55
53
52.6)
5642
54
52
51
53
50
25
0.5
2.5
(a) Performance with different hyper-parameter a
(b) Performance with different hyper-parameter A
Figure 6: Performance with different hyper-parameters. The horizontal axis the value of hyper- parameters and the vertical axis is the top-1 accuracy.
DARLING (Zhang et al., 2022) incorporates domain information into the contrastive loss by re- weighting domain labels. BrAD (Harary et al. 202I) projects inputs into an auxiliary bridge domain and utilizes contrastive learning in this domain to learn Nomain-invarian above works can eliminate the influence of domain shifts to some extent, their performances are still limited due to the difficulty to well define positive pairs in contrastive learning.
features. Although the
Recently, generative -eeeed methods are proposed solve the UDG problem. One of the most rep- resentative methods is DiMAE (Yang et al. (2022), which establishes an MAE-style (He et al. 2021) generative framework for UDG task. DiMAE contains a content encoder and multiple domain- specific decoders. The input images will be transformed into style-view by the proposed Content- Preserved StyleMix module (Yang et al. 2022), and then be masked randomly. The content encoder extracts Nomain-invarian features of the style-view counterparts and then the domain specific de- coders are designed to recover the reconstruction from the features. this way, nomain-invarian features are learned by the content encoder. Different from DiMAE, proposed method does not rely on the heuristic pairs which are generated by stylemix and thus can get more realistic cross- domain pairs.
Cycle-consistency. Cycle-consistency is a common visual trick to preserve the content unchanged. (Zhou al.) (016)) utilizes the consistency across instances of the same category supervised signal to force the model to predict the corresponding relationship between cross instances with the same object. eY al.) and (Zhu al. 2017) both solve the image-to-image translation problem by cycle-consistency. DualGAN is similar to dual learning and trains both primal and dual GANs at the same time. CycleGAN utilizes cycle-consisie conessconsst to make the reconstructed images match closely to their original images.
The previous methods we mentioned use the cycle-consistency to remain the object of images un- changed from one image to another image and make two images share the same similar object. Although we also introduce cycle-consistency to retain the content unchanged, our purpose is not to generate find another image with the same similar object. We utilize the cycle-consistency to push the features extracted by the content encoder to retain more content information and thus we can get better iomain-invariint features.
5.2 MORE EXPERIMENTS
5.2.
PAPER-PARAMETERS SENSITIVITY
As shown in Fig 6 (a), we ablate the loss weight of domain contrastive loss. We observe that the performance is sensitive to the weight of domain contrastive loss, and the Fig 6 (b) shows that cycle reconstruction loss is not sensitive otherwise the weight equals O. We conjecture the reason that cycle reconstruction loss, as the key point of the cycle reconstruction task, always works to guarantee the good performance of CycleMAE whatever the weight of it is except 0. But domain contrastive loss acts as a regularization of the cycle reconstruction task, and its weight influences its regularization ability. From the results, set and 2.
13

---

Published as a conference paper at ICLR 2023
Label Fraction 1% (linear evaluation)
Label Fraction 5% (lincar evaluation)
method
Painting
Real
Sketch
Overall
Avg.
Painting
Real
Sketch
Overall
Avg.
6.68
6,94
7.45
McCo
Chen et al
2020d
11.38
14.97
15.28
14.04
13.88
20.80
24.91
21.44
23.06
22.39
SimCLR
V2
(Chen et 2020c
16.97
20.25
17.84
18.85
18.36
21.35
24.34
27.46
24.15
24.38
BYOI.
Ginll al
2020
5.00
8.47
4.42
6.68
5.96
9.78
10.73
3.97
9.09
8.16
AdCo
Hu et a
2021
11.13
16.53
17.19
15.16
14,95
19.97
24.31
24.19
23.08
22.82
MAEIVIT
smalli
He et al.
2021
17.86
24.57
19:33
21.63
20.59
24.55
30.43
26.07
27.90
27.02
DIUL Zhang
2022
14.45
21.68
21.30
19.59
19.14
21.09
30,51
28.49
27.48
28,19
DIMAD(VIT small) Yang al.
2022
20.18
30,77
20.03
25.63
23,66
27.02
39,92
26.50
33,59
31,15
CycleMAE
(ViT.tiny)
21.24
25,94
22.48
20.87
23.04
25,46
23.80
CycleMAE (NiT small)
30.38
22.31
26.63
25.18
27.64
34,38
32,20
CycleMAF: (ViT_small Supervised Pretmiined)
22.26
26.83
27.21
39.95
27.97
33,96
31.71
Label Fraction 10% (full finetuning)
Label Fraction 100% (full finetuning)
method
Painting
Real
Sketch
Overall
Avg.
Painting
Real
Sketch
Overall
Avg.
FRM
9.90
9.19
5.12
8,56
8.07
31.50
40,21
24.01
34,48
31,91
MoCo V2
Chen al
2020d
25.35
29.91
23.71
27.37
26.32
43.42
58.61
40.38
50.66
47.47
SimCLR R V2
(Chen
2020cc
24.01
30.17
31.58
28.75
28.59
46.79
62.32
51,05
55.71
53.39
BYOL
2020
8.92
34.02
AdCa
29.98
27:57
27.65
26.97
43.55
61.42
51.23
54.37
52.07
MAE (ViT VYT_smally (He et a 202
41.24
54.68
39.41
47.82
45.11
53.13
08.51
48.86
60.21
56.83
DARLING ZZhnn et al. 2022
25.90
33.29
30.77
30.72
29.99
49.64
63.77
54.31
57.91
55.91
DIMAE (VTT, smally Yang et al
2022
50.73
64.89
55.41
59.01
57.01
70.48
82.79
72,10
77.18
75.12
CycleMAE (ViT tiny)
43.42
57.75
41.84
50.51
47.67
56.92
72.52
53.28
64.24
60.91
CycleMAE(VT) (ViT_small) small)
52.81
67.13
$6.37
60.95
58.77
72.01
84.95
73.84
79.08
76.93
CyclMAE (ViT small Supervised Pretrained)
53,40
67.24
55.72
61.04
58.79
21.93
84.87
74.62
79.18
77.14
Table Results of the cross-domain generalization on DomainNet. All of the models are trained on Clipart, Infograph, Quickdraw domains of DomainNet and tested on the other three domains. The title of each column indicates the name of the domain used as target. All the models are pretrained for 1000 epoches before finetuned on the labeled data. Results style: best, second best.
5.2.2 EXPERIMENTS ON OPPOSITE SETTING OF DOMAINNET
We showed part of our results in the main text where we train our model on Painting, Real, and Sketch, and evaluate the generalization ability of our model on Clipart, Infograph, and Quickdraw. In this section, we train our model on Clipart, Infograph, and Quickdraw, and evaluate on Painting, Real, and Sketch. Same as we mentioned in the main text, we exactly follow the all correlated setting proposed by DARLING (Zhang et al. 2022).
The results are presented in Tab Our CycleMAE still achieves good performance. We achieve a improvement by + +1.00 % and +0.79% % on overall accuracy and +1.52% and +1.05% on average accuracy for 1% and 5% fraction setting. Specifically, for 10% fraction and 100% fraction setting, our CycleMAE improves the statc-of-the-an methods by +1.76 % and +1.81 % on average accuracy and +1.94% and +1.90% on overall accuracy.
5.3 EXPERIMENTS WITH VIT TINY BACKBONE
In this section, we use a smaller backhone, ViT_tiny, to illustrate the effectiveness of our proposed CycleMAE. We still follow the all correlated setting of DARLING (Zhang et al. 2022).
5.3. EXPERIMENTS ON DOMAINNET
In this section, we use the VIT -tiy as the backbone of our proposed CycleMAE and evaluate the performance on DomainNet with two tasks. The first task is training our model Painting, Real and Sketch, and evaluating the generalization ability of our model on Clipart, Infograph and Quick- draw (Painting, real, sketch + Clipart, Infograph, Quickdraw). The second task is training our model on Clipart, Infograph and Quickdraw, and evaluating it on Painting, Real and Sketch (Clipart, Infograph, Quickdraw Painting, real, sketch).
14

---

Published as a conference paper at ICLR 2023
Label Fraction 1%
Label Fraction S8
methed
Clipart
Infogrihh
Quickdon
Oveaal
Avg.
Clipart
Infograph
Quickdraw
Ovenli
Avg.
ERM
6.54
2.96
5,00
4.75
4,83
10.21
7.08
5.34
6.81
7.54
McCo
18.85
10.57
11.92
28.13
13.79
SimCLR
312000
23.51
15.42
14.74
34,03
17.17
10.88
20.69
BYOL
Grill
2020
6.21
3.48
4.27
4.45
4.65
9.60
5.09
6.02
6.49
6.90
AdCa
Hu ct al.
2021
16.16
12.26
5,65
9.57
11.36
30.77
18.65
7.75
15.44
19.06
MAE small He ct
2021
22.38
12.62
10.50
13:51
15.17
32.60
15.28
13.43
17.85
20.44
DARLING 7hone el 20123)
18.33
10.62
12.65
13.93
39.32
19.09
10.50
22.97
small) Yang ct
26.52
15.47d
15.47
19.15
42.31
18.87
CycleMAE (VT.tiny)
37.82
CyckMAE (VT small)
37.54
18.01
17.13
21.53
24.23
55.14
20.87
19.62
27.31
31.88
CydlMAE (VT small Supervised Pretrained)
39.02
18.63
14,50
20.70
24.08
56.27
22.22
17.97
26.96
32.15
Label Fraction 10%
Label Fraction 100%
method
Clipart
Infograph
Quickslos
Overall
Avg.
Clipart
Infograph
Quickdraw
Oiveral
Avg.
ERM
15.10
9.39
7.11
9.36
10.53
52.79
23.72
19.05
27.19
31.85
McCo
32.46
18:54
27.44
SimCLR
19:87
27.60
BYOL
Grill
2020
14.55
8.71
5.95
8.46
9.74
54.48
23.70
20.42
28.23
32.86
AdCa
Hu et al. 2021
32.25
17.96
11.56
17.53
20.59
62,84
26.69
26.26
33,80
38.60
MAE smalll l
He al
2021
51.86
24.81
23.94
29.87
33.54
59.21
28.53
23.27
32.06
37.00
DARLING Zhuna et al.
25.15
20.88
15.69
21.08
23.91
72.79
32.01
33.75
41.19
46.18
smally Yang al
70.78
38.06
39.20
83.87
CycleMAE (NT..iny)
44.99
CyckMAE (VT small)
24.82
38.42
26.32
40.61
47.20
85.39
44.31
40.03
50,16
56.58
CydleME (VI small Supervised Pretrained)
75.93
39.11
29.19
41.47
48.05
86.40
44.93
40.56
51.12
57.30
Table 8: Results of the cross-domain generalization DomainNet. All of the models trained on Painting, Real, Sketch domains of DomainNet and tested on the other three domains. The title of each column indicates the name of the domain used as target. All the models pretrained for 1000 epoches before finetuned on the labeled data. Results style: best, second best.
Painting, real, sketch Clipart, Infograph, Quickdraw. We present the results of this task in Tab. 8 From the results, our proposed CycleMAE achieves competitive performance even with the ViT_tiny. For 1% and 5% fraction setting, our CycleMAE get similar result compared with the previous state-of-the-art method DiMAE (Yang et al., (022), although ViT_tiny is smaller than the backbone used in DiMAE, Except DiMAE with ViT ViT_small, our CycleMAE improves other previous methods by +3.70% % +1.16%, +5.09% average accuracy for 1%, 5%, 10% fraction setting.
Clipart, Infograph, Quickdraw > Painting, real, sketch. The results of this task are shown in Tab. 7 For fair comparison, do not compare CycleMAE with ViT tiny with DiMAE which use the VIT small as the backbone. Compared with previous methods except DiMAF, our CycleMAE improve the average accuracy by +0.28 % +2.56% and +4.08 % for 1 %, 10% and 100% fraction setting.
5.3.2 EXPERIMENT ON PACS
We also use the PACS evaluate the effectiveness of our CycleMAE. We present our results in S- T In this setting, our CycleMAE achieves better performance than previous works on most tasks and gets significant gains over DiMAE, DARLING and other SSL methods on average aceu- racy. Compared with state-of-the-art methods, our CycleMAE improves the accuracy by +2.46% % +0.60% on average for 1%, 5% fraction setting, respectively. And for 10% and 100% fraction set- ting. compared with other otate-of-theearr the-art methods except DiMAE, get +14.56% and +10.80% performance gains on average accuracy.
5.4 VISUALIZATION
We visualize more reconstruction results of CycleMAE using ViT-base in Fig 7 We can know from the visualization that the cross-domain pairs generated by CycleMAE have good content- consistency.
15

---

Published as a conference paper at ICLR 2023
Label Finction
1%
Label Finction 5%
method
Phoso
Am.
Cartoon
Sketch
Photo
Cartoon
Sketch
MoCo Chen
20204
15.58
23.65
25.27
21.87
37.39
25.57
28.11
31.16
30.56
SinCh
20200
17.43
54.67
40.68
BYOL Grill
2020
14.53
eu AdCo
2021
Zhung
DIMAF smmllly Yang
2022
11.73
25.83
CycleMAE tiny)
CycleMAE (VT small)
26.25
42.15
CycleMAE Supervised Presrained)
38.13
method
Photo
Art.
Cartoon
Skotch
Avg.
Photo
Art.
Cartoon
Sketch
Avg.
MoCo V2
Chen
2020dl
44.19
25.85
33.53
24.97
32.14
59.86
28.58
48.89
34.79
43.03
SimCLR V2
Chen et
2020c
54.65
37.65
46.00
28.25
41.64
67.45
43.60
54,48
34.73
50.06
BYOL
Grill
1020
27.01
25.94
20.98
19.69
23.40
41.42
23.73
30.02
18.78
28.49
AdCo
Hu
46.51
30.21
31.45
22.96
32.78
58.59
29.81
50.19
30.45
42.26
MAR(VIT smmlli
(He et
202.1
35.89
25.59
33.28
32.39
31.79
36.84
25.24
32.25
34.45
32.20
DARI ING Zhaing of nl.
21122
53.37
39.91
46.41
90.17
42.47
68.66
41.53
$6.89
37.51
51.15
DIMAE (Vil'smally) Yang et al
2022
77.87
59.77
37.72
39.25
58.65
78.99
63.23
$9.44
55.89
64.39
CyckMAE (ViT_inny)
83.27
56.42
53.58
34.85
57.03
46
611.47
60.884
37.04
61.95
CycleMAE smaall)
85.94
67.95
59.34
38.25
62.87
90.72
75.34
69.33
50.24
71.41
CycleMAE sVill Supervised Pretraincd)
87.02
68.31
60.20
37,10
63.23
92.17
77.69
70.56
$1.07
72,87
Table 9: Results of the cross-domain generalization setting on PACS. Given the experiment for each target domain respectively, there is overall accuracy across domains. Thus we report the average accuracy and the accuracy for each domain. The title of each column indicates the name of the domain used as target. All the models are pretrained for 1000 epochs before finetuned on the labeled data. Results style: best, second best.
5.5 BROADER IMPACT
We propose effective generative- based unsupervised domain generalization method and get a more realistic cross-domain pairs from model outputs. However, in our experiments, there are a potential issue that we should consider to remedy in the future. The issue is that our experiments rely on many GPUs to pretrain and test, which may consume a lot of electricity. And we know that using too much electricity can cause pollution which may influence our world.
16

---

Published as a conference paper at ICLR 2023
Origin
Painting
Real
Sketch
Figure 7: Reconstruction visualization of different decoders.
17


Reviews:
[{"review_id": "rt2hJlEnGY", "reviewer": "Reviewer_BvPn", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes to use cycle consistency to help learn domain-invariant features for unsupervised domain generalization.", "summary_of_the_review": "The paper proposes to use cycle consistency to help learn the domain-invariant features and the results are promising.", "strength_and_weaknesses": "Pro:\n\n1.  Previous method DiVAE only considers the reconstruction loss ||y_m-x_m||. The proposed loss considers how to utilize the information of y_n and use cycle consistency to further help the task, i.e., adding a backward transformation process for other domain labels $n\\neq m$.\n\n2. The paper also notices that the generated images in other domains $$y_n$$ may be trivial such that all images are of the same style. To encourage diverse domain information, they propose a domain contrastive loss.\n\n3. The results are promising and the paper is well-written.\n\nCons:\n\n1.  Since the model adds a backward process compared to DiVAE, I'm wondering how much training time is increased. It would be good to have a running time comparison.\n\nComments:\n\nThe proposed domain contrastive loss encourages the outputs to have different domain styles but there is no supervised loss to encourage it to be exactly the domain style i. Is it possible that the outputs of different domain-specific decoders are out of the given domains and harm the performance? \n\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "fpTtsmPCHx", "reviewer": "Reviewer_7ZLc", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper focuses on the setting of unsupervised domain generalization, where the multiple source domains do not have label supervision and the target domain is unseen. Based on the previous work (i.e., DiMAE) that applies the masked-reconstruction-modeling as a self-supervised task, this paper further proposes two more objective functions: the cycle reconstruction loss and domain contrastive loss. The first one loss transforms the generated image back to original image domain and the second one loss performs a contrastive learning across the features from different decoders. The authors have conducted experiments on the DomainNet and PACS and achieves some improvements over the previous methods.", "summary_of_the_review": "This paper proposes to use a cycle-reconstruction loss and domain contrastive loss in the unsupervised domain generalization. Consistent performance gains are detected on two widely-used benchmarks. Currently, the reviewer’s concerns are two-fold. On the one hand, the authors should make sure that the experimental comparisons are fair and reproducible. On the other hand, the two new losses are kind of limited in the terms of novelty, which might be lower than the bar to be accepted to a venue like ICLR.", "strength_and_weaknesses": "Issues regarding the fair experimental comparisons.\n+ To the reviewer’s knowledge, some previous works such as DiMAE use a supervised pre-trained model while this paper adapts an unsupervised pre-trained model on ImageNet. To this end, could the authors explain how much will the different pre-trained models affect the final performance? Meanwhile, pre-trained models using different pretraining recipes (e.g., number of training epochs) are also different starting points for downstream tasks. Overall, the reviewer hopes the authors to make sure the experimental comparisons are fair in terms of the pre-trained model.\n***\nIssues regarding the methods.\n+ Could the authors clarify the extra training overhead they have introduced in their framework? Because more decoders and forward passes are appended upon previous approach.\n+ As claimed in the paper that, the domain contrastive loss encourages more diverse domain-specific styles to be learned in the decoder. The reviewer is interested that why such constraints should be placed in the early layer of the decoder since the higher layer closer to the pixel space might be more responsible to encoder style information? \n+ The unsupervised domain generalization is a new setting in the field. The reviewer believes that reporting the performance in the unsupervised source domain is also of significance. Could the authors provide such information?\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "B4zyon7nYMm", "reviewer": "Reviewer_QvTm", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper aims to address unsupervised domain generalization via a proposed cycle-consistent masked autoencoder, which includes a CycleGAN, MaskAutoEncoder, and contrastive learning. The authors claim that the poposed method can significantly improve state-of-the-art performances, which are demonstrated by the experimental results to some extent.", "summary_of_the_review": "Please see [Strength And Weaknesses].", "strength_and_weaknesses": "# Strength\n\n1. The motivation of introducing CycleGAN, MaskAutoEncoder, and contrastive learning sounds reasonable.\n\n2. The experimental results look solid.\n\n3. The writing is good, and the structure is easy to follow.\n\n# Weaknesses\n\n1. The novelty of this paper seems marginal. The main modules of the proposed method, including CycleGAN, MaskAutoEncoder, and contrastive learning, are all existing famous works. It seems that the authors simply combine these works into UDA.\n\n2. The visualization results in Figure 5 are unsatisfied, especially the sketch ones. It is highly suggested to release more visualization results to demonstrate the effectiveness of the reconstrcution process.\n\n3. I notice that the authors use Vit-small as the backbone network. Do all the other comparison methods use this network? If not, it is suggested to compare the backbones of different methods, including training time, model size, and inference speed. \n\n4. The proposed method needs to be pre-trained for 1000 epochs, which may easy to cause overfitting. It is suggested to add a set of experiments to fully prove the generalization of the proposed method.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Decision: Accept: poster