Semi-Supervised Single Domain Generalization with Label-Free Adversarial Data Augmentation
Authors: Ronghang Zhu
Xiang Yu
Sheng Li
Venue: Accepted by TMLR ()

Abstract: Domain generalization (DG) has attracted increasing attention recently, as it seeks to improve the generalization ability of visual recognition models to unseen target domains. DG leverages multiple source domains for model training, while single domain generalization (SDG) further restricts such setting by exploiting only a single source domain. Nevertheless, both DG and SDG assume that the source domains are fully labeled, which might not be practical in many real world scenarios. In this paper, we present a new problem, i.e., semi-supervised single domain generalization (SS-SDG), which aims to train a model with a partially labeled single source domain to generalize to multiple unseen testing domains. We propose an effective framework to address this problem. In particular, we design a label-free adversarial data augmentation strategy to diversify the source domain, and propose a novel multi-pair FixMatch loss to generalize classifiers to unseen testing domains. Extensive experiments on OfficeHome, PACS and DomainNet20 datasets show that our method surpasses the latest SDG and semi-supervised methods. Moreover, on PACS and DomainNet20, our method approaches the fully supervised ERM upper bound within $5\%$ gap, but only uses less than $8\%$ of the labels.

Published in Transactions on Machine Learning Research (11/2023)
Semi-Supervised Single Domain Generalization with Label- Free Adversarial Data Augmentation
Ronghang Zhu University of Georgia
ronghangahu@@uga.edi
Xiang Yu Amazon
yncraag@34@mmmiilccmm com
Sheng Li University of Virginia
shengli@@virginia.dd edu
Reviewed on OpenReview: https: // openreview. net/ forum? id= sUlbRfLijj
Abstract
Domain gencralization (DG) has attracted increasing attention recently, as it secks to im- prove the generalization ability of visual recognition models to unseen larget domains. DG leverages multiple source domains for model training, while single domain generalization (SDG) further restricts such setting by exploiting only single source domain. Neverthe- less, both DG and SDG assume that the source domains are fully labeled, which might not be practical in many real world scenarios. In this paper, present new problem, i.e,, semi-supervised single domain generalization (SS-DDG), which aims to train model with partially labeled single source domain to generalize to multiple unseen testing domains. We propose an effective framework to address this problem. In particular, we design label- free adversarial data augmentation strategy to diversify the source domain, and propose a novel multi-pair FixMatch loss to generalize classifiers to unseen testing domains. Exten- sive experiments OfficeHome, PACS and DomainNet20 datasets show that our method surpasses the latest SDG and semi-supervised methods. Moreover, PACS and Domain- Net20, our method approaches the fully supervised ERM upper bound within 5% gap, but only uses less than 8% of the labels.
1 Introduction
Deep neural networks (DNNs) lead large success in the past decade in many fields, e.g., object detection and classifications. Many of the applications rely on the assumption that training and testing distributions are identical or close. However, in real scenarios, data acquiring always encounters the environment variance, i.e., the lighting changes from dawn to night, or the camera moves from one place to another. The environment variance inevitably brings the domain shift. for the captured training and testing data (Recht et al., 2019: Hendrycks Dictterich, 2019). Closing this domain discrepancy has become one of the recent popular topics in the community.
Domain adaptation (DA) (Wang & Deng 2018) and domain generalization (DG) (Zhou et al. 202 Wang ct al. 2021a) are the major techniques to tackle this problem. DA methods jointly exploit the source and largel domain data for model training, in which the methods attempt. to align the feature space wetw the source and target domains. While DG methods solve a more challenging task, utilizing multiple labeled source domain data learn towards a generalized model, to predict the target domain data which is unavailable in the training process. Compared DA, DG relaxes the assumption on target domains and usually enjoys better model generalization ability. The differences between DA and DG illustrated in Figure 1

---

Published in Transactions on Machine Learning Research (11/2023)
DA Imaning Phose
SS-DA
DG
SS-DG
SDG
SS-SDG
franing Prase
Training Phase
Tranng Phase
Iranng Prase
Training Phuas
Testing Phose
Testing Phose
Tosting Phace
Testing Phose
Testing
Testing Phize
Source Domain
Target Damain
Labeled Samples
ooD
Unlabated Samples
Figure 1: Problem setting differences among domain adaptation (DA), single domain generalization (SDG), domain generalization (DG), semi-supervised domain adaptation (SS-DA), semi-supervised domain gener- alization (SS-DG), single domain generalization (SDG), and our newly introduced setting semi-supervised single domain generalization (SS-SDG) task. Different colors indicate different domains.
Though promising, DG still faces two major limitations. First, DG methods require multiple source domains for model training. In practice, it would be expensive or even infeasible collect multiple source domains' data. To address this problem, single domain generalization (SDG) (Qiao et al. 2020 has been proposed recently. Different from DG, SDG aims to train model with single source domain data and generalize to multiple unscen target domains. Second, DG methods require that the souree domains should be fully labeled, which is usually expensive and labor intense. However, in real-world application scenarios, it's not always feasible to have access to a fully labeled source domain. Using face recognition as an example, the offline face rocognition model relics on labeled data typically obtained in controlled environments or through prior registration processes. However, in dynamic and open settings like survellancee scenarios, encountering new users may not involve prior registration step, resulting in a significant portion of unlabeled data. Tn this context, the term "source" data remains consistent, representing the labeled dataset acquired during the offline phase. Conversely, the "target" data show variability across diverse scenarios, including indoor and outdoor environments. This discrepancy poses a significant challenge since the offline data are well-labeled, while real-time data, forming the "target" data, are mostly unlabeled. In essence, the practical application of SS-SDG in face recognition grapples with effectively managing both labeled offline data and predominantly unlabeled real-time data. The overarching goal is to develop a model that is both generalizable and robust for face recognition. Very recently, a solution to this challenge has emerged in the form of semi-supervised domain generalization (SS-DG) (Zhou et al., 2021b). This approach operates under the assumption that only small porlion of the samples are labeled in the source domains. Clearly, SDG and SS-SDG separately tackle each of the two limitations, i.e., single source domain and few labeled data. Different from existing problems, we desire a unified framcwork that can boost domain generalization from both of the two aspects.
Consequently, we propose a more practical yet unsolved problem, i.e., semi-supervisee single domain gener- alization (SS-SDG). In this problem, we assume that only one source domain is available for model training, and it consists of few labeled and abundant unlabeled samples. The relationships between SS-SDG and other related problems are illustrated in Figure [ While SDG is the most relevant setting to ours, most of existing SDG methods (Zhao et al. 2020 Qiao et al. 2020; Fan et al. (021) are based on adversarial data augmentation (Volpi et al.] (2018) and require the label information to generate new samples to enrich the diversity of source domain. As the result, these SDG methods would fail in our proposed SS-SDG setting which cannot provide sufficient and accurate label information for data augmentation. Our empirical results validate that existing SDG methods perform unsatisfying in our SS-SDG task.
To address the new challenging problem, propose novel label-free adversarial data augmentation frame- work to enrich the source domain diversity without label information, well to leverage such generated data in a multi-pair FixMatch way to regularize for better training. Inspired by self-supervised learning (SSL) Yang et al. 2021), we design a label-free adversarial data augnentation strategy, which is an inter- active feature extractor pre-training and adversarial sample generation approach without label information, to enhance the diversity of source domain data. Given source domain and the newly generated samples, we organize them into multiple training pairs and propose a novel multi-pair FixMatch (MPFM) loss regularize the classifier training for better generalization ability to unseen testing domains.
2

---

Published in Transactions on Machine Learning Research (11/2023)
Our contributions are thus summarized as:
We introduce a new challenging domain generalization task, namely the semi-supervised single do- main generalization (SS-SDG), addressing the scenario training on a partially labeled single source domain while generalizing multiple unscen target domains.
We propose an efficient framework, introducing a novel label-free adversarial data augmentation strategy to enrich diversity of the single source domain diversity without label information, and a multi-pair FixMatch regularization to better utilize the diversified data towards a more generalized classifier for unseen target domains.
We conduct extensive SS-SDG experiments on OfficeHome, PACS and DomainNel20,, and achieve superior performance over the state-of-the-arss, e.g., approaching supervised ERM upper bound within 5% accuracy gap by only using less than 8% labels.
2 Related Work
2.1 Domain Adaptation
The basic idea of domain adaptation (DA) 7hh et al. 2023 Shi et al. 2022 is to leverage the labeled source data and unlabeled target data to adapt the model from the source domain to target domain. Existing domain adaptation methods address the domain gap between source and target domains via distribution alignment
by statistical metrics (Gretton et al., 2006 Shen et al., 2018) or adversarial learning (Ganin & Lempitsky
2015 Zhu et al. 2021b Zhu & Li 2022b Rezayi et al. 2023). Among different DA problem settings (Zhu)
& Li 2021), Semi-supervised domain adaptation (SS-DA) (Donahue al., 2013 Saito et al., 2019) is the
most relevant DA problem to proposed SS-SDG where few target labels are available. The early SS-DA efforts (Donahue et al. 2013 Yao et al., 2015 Ao et al. 2017) involved the creation of individual models for each domain by regulating through the use of constraints. Recently, the Minimax Entropy (MME) (Saito (et al. 2019) utilizes the conditional entropy value of unlabeled larget data to play min-max game wetween feature encoder and classifier to minimize domain discrepancy and simultaneously acquire discriminative features for the task. AdaMatch (Berthelot et al., (221)) induces random logit interpolation address the differences between source and larget domains. Domain adaptation is indeed a potential solution for our proposed SS-SDG problems. However, the requirement of having access to the target domaín data hinder the application of traditional domain adaptation methods to our SS-SDG challenges.
2.2 Domain Generalization
Different from domain adaptation, domain generalization (DG) (Wang et al., 2021a Zhou et al. 2021a) takes
into account situations where the target domain is not available during the process of model learning. The goal of DG is to learn a generalized model with multiple source domains to adapt to an unseen target domain.
The DG methods can be roughly divided into four groups, i.c., domain alignment based methods Gaanin
et al.]
2016:
L.i et al.
2018b: Piratla et. al. 2020) meta-learning based methods (Li et al.
2018a Dou
et al.
2019
Du et al.
2020), data augmentation based methods (Shankar et al. 2018 Zhou et al.
2020a
Robey et al. 2021
Huang et al. 2021). ) and casual based methods (Dii et al. 2023 Mahajan et al.
2021
For domain alignment based methods, the central concept is to minimize the discrepancy between source domains for learning domain-invariant representations. The Maximum Mean Discrepancy based Adversarial Autoencoder (MMD-AAE) (Li et al. 2018b) minimizes the MMD distance between source domains in feature space, while employing adversarial learning to make the distributions of the source domains similar prior distribution. For meta-learning based methods, the main idea is to leverage the multiple source domains in a meta-train and meta-test manner and update the gradient in a more sophisticated way. The Meta-Learning Domain Generalization (MLDG) (Li et. al. (0188a) proposes a model-agnostic training procedure to improve the generalization of learned model by simulating the train/test domain shift during training by generating virtual test domains. For data augmentation basod mothods, the goal is to explore how to generate extra synthetic data based on the source domain data, and use the joint data to recover the unseen target domain distribution. The Model-Based Domain Generalization Robey al. 2021 proposes novel framework
3

---

Published in Transactions on Machine Learning Research (11/2023)
for DG, in which the goal is to impose invariance to the underlying transformations of data that reffect the variations between domains. For causal-based methods, the objective is to employ causal reasoning to establish invariant conditions for domain generalization. An example of this approach is MatchDG (Dai ct al. 2023), which introduces an object-invariant condition for domain gencralization. This is achieved by minimizing the distance between representations of the same object across different domains through causal reasoning. However, these existing domain generalization (DG) methods are not well-suited for addressing the SS-SDG problem we have proposed. The primary reason is that they depend on having access to fully labeled multi-source domains, making them unsuitable for direct application to the SS-SDG scenario.
2.3 Single Domain Generalization
Single Domain Generalization (SDG) is a more challenging setting compared to domain generalization, which only uses one source domain to learn a model and generalize to multiple unseen target domains. The early SDG method, i.e., Meta-Tearning based Adversarial Domain Augmentation (M-ADA) (Qiao et al.) 2020). jointly utilizes adversarial data augmentation (Volpi et al. 2018) relaxed by a Wasscrstein auto-encoder and meta-learning to improve the diversity of generated samples and the generalizat ion of learned model. Following this direction, numerous methods (Zhao et al. 2020 Li et al. 2021, Fan et al. 2021 Wang et al. 2021b Zhu & Li, 2022a) have been proposod for expanding the source data by generating samples that fall outside the source domain, with the goal of encompassing the range of the target domain. For examples, the Marning-to-Diversee (T.2D) (Wang et al. 2021b) proposes a style-complement module to improve the model's generalization ability. This is achieved by synthesizing samples from diverse distribution that complement the original source domain. Further, two-step wise mutual information optimization is used to learn distinctive features from diversely styled data. The Progressive Domain Expansion Network (PDEN) (Li et al.] 2021 designs two subnetworks, i.c., domain expansion subnetwork and representation learning subnetwork, to work together in a mutually beneficial way through joint learning. The former subnetwork aims to generate multiple domains progressively to simulate various photometric and geometric transformations in unseen domains. The later subnetwork tries to learn domain invariant representation that clusters each class well, resulting in a better decision boundary and improved generalization. Recently, the Center-aware Adversarial Domain Augmentation (CADA) (Chen el al., 2023) proposes a novel angular center loss to modify the source samples in such a way as to move them away from the class centers. Thus, the generated samples are diversified enough to simulate a large domain shift and imporve the genoralization of learned model. In contrast to standard single domain generalization (SDG), we introduce more practical yet challenging setting known as semi-supervised single domain generalization (SS-SDG), where the availability of labeled samples in the source domain is limited. Unlike traditional SDG methods, which heavily rely on labeled data for data augmentation, our SS-SDG setting presents unique challenge. Here, the effectiveness of these SDG methods may be compromised due to the insufficient availability of labeled information in our setting,
2.4 Semi-Supervised Learning
Semi-Supervised Learning (SSL) is a fundamental rescarch topic in computer vision and machine learning, which seeks to learn a model with few labeled and a large amount of unlabeled data. SSL methods can be
generally divided into three categories: (1) The pseudo-labeling based methods (Lee et al.) 2013 Berthelot
let al.] 2020 Xie et al.) 2020b), utilize the intermediate model to predict the pseudo ground truth label and iteratively update the model with the pseudo labeled data. For example, the MixMatch (Berthllot ct al. 2020 induces distribution alignment and augmentation anchoring to simultancously align the marginal
distribution of prediction on unlabeled data with the marginal distribution groundtruth labels and enforce the
output of multiple highly transformed versions of same data to be similar for weakky-transformest version of the data. (2) The consistency constraint (Oliver et al.) 2018) based methods (Tarvainen & Valpola, 2017 Miyato et al. 2019 Zhang & Qi 2020), leverage the consistency across the same data with multiple augmentations or perturbed models to regularize the embedding learning. For instance, the Worse-Case Perturbation (WCP) (Zhang & Qi 2020) is inspired by the concept that a robust model should not be easily impacted by minor variations and keep the output decision remain as consistent as possible for both label and unlabeled data. Therefore, two regularizations, i.c., additive and DropConnect perturbations, are proposed to perturb the model by injecting noise to model weights and changing connections in model structure.
4

---

Published in Transactions on Machine Learning Research (11/2023)
(3) The comprehensive methods (Sohn et al. 2020 Xie et al. 2020a Zhang et al. 2021 combine both the pseudo-labeling and data augmentation for more performance boost. Such as, the Flex Match (Zhang let al., 2021) proposes curriculum learning strategy to make use of unlabeled data based on the model's learning progress. The essence of this stratogy is to dynamically adjust thresholds for different classes at each iteration, allowing informative unlabeled data and their pseudo labels to be utilized. The canonical SSL, setting would be impractical, as it assumes the training distribution and testing distribution are similar. To relax the assumption, a new semi-supervised domain generalization (SS-DG) (Zhou et al.) 2021b) is proposed, i.e., StyleMatch (Zhou et al. 2021b), which tackles SS-DG by inducing style augmentation (Huang & Belongie 2017a) to extend source domain distribution in the FixMatch (Sohn et al. 2020) framework, and alleviates overfitting problem with stochastic classifier. Our proposed approach, semi-supervised single domain generalization (SS-SDG), represents a significant advancement in the field. It deals with the challenge of generalizing from a single source domain and a limited amount of labeled data to multiple target domains. In this context, traditional semi-supervisod domain generalization (SS-DG) methods, such as FixMatch, tend to perform sub-optimally due to the extreme scarcity of labeled data available in the source domain.
2.5 Data Augmentation
Data augmentation has become a widespread practice for enhancing the training of machine learning models, helping to mitigate overfitting and improve generalization. The fundamental concept behind data augmen- tation involves extending the original (data, label) pairs with new pairs, denoted as (T(data), label), where T(.) represents transformation. Existing data angmentation methods can be roughly categoried into three categories: (1) Hand-crafted transformations: the main idea is to encompass the stacking of traditional image transformations within T, such as random flips, random crops, and rotations. In addition to these conventional image transformations, there have been explorations into techniques like Mixup (Zhang et al. 2017), which involves mixing instances at the image level. However, it's important to note that these transformations may have limitations in certain applications, potentially altering the associated labels. (2) Adversarial learning based augmentation: inspired by adversarial attacks (Xu et al.) (020a), this branch involves perturbing images using sign-flipped gradients back propaprop from classifie. This method is commonly used in single domain generalization to generate 'challenging" images, making it more difficult for the classifier to make accurate judgments. However, a drawback is the increased computational cost associated with these techniques. (3) CNN based augmentation: in this category, Convolutional Neural Networks (CNNs) (Gu et al. 2018) serve as the transformation function T(.). The primary goal is to alter the styles of images, thereby enriching the diversity of the existing datasct. A notable limitation of these learnable image synthesis methods is that the newly generated images may not exhibit significant differences from images in other domains due to the lack of style information. This direction can be further divided into two subcategories. The first one is learnable based augmentation (Yue et al. 2019 Zhou et al. 2020b) which focuses on learning an image translation model or employing pre-existing style transfer models like CycleGAN (Zhu et al. 2017) or AdaIN (Huang & Belongie, (017b).. While the other one is Nen-learnable augmentation (Xu et al. 2020b) where T(-) is randomly initialized with a Gaussian distribution at each iteration. While CNN-based augmentation is a potential solution for addressing generalization tasks related to a single source domain, it's important to note that its requirement of multiple source domains makes it less applicable to the SS-SDG problem, which primarily deals with single source domain. In contrast, adver- sarial serring--bsedd augmentation has demonstrated its efficiency in tackling single domain generalization (SDG) problems. Therefore, opting for an adversarial learning-based approach represents a promising and practical solution for addressing SS-SDG challenges.
3 Our Approach
The overall proposed framework is illustrated in Fig. 2 which consists of two stages. At stage 1, we designed two phases: pretrain phase and data generation phase. In pretrain phase, we adopt contrastive learning to adapt the feature generator G to source domain by mining the similarity among samples in feature space. In data generation phase, we generate new diversified samples to complement source domain by perturbing source samples in an adversarial way with contrastive learning. The data generation phase is alternated with the pretrain phase. At stage 2, we jointly train the classifier C and the pre-trained G by adopting
5

---

Published in Transactions on Machine Learning Research (11/2023)
Stage 1-1; Pretrain (Sec. 3.2)
Stage 1-2: Data Augmentation (Sec. 3.3)
Stage 2: Multi-Pair FixMatch (Sec. 3.4) [X x.J
D
5 A-Aug
WsAus
S-Aup
S-Aug
Pretrain Timeline
epocch
epoch
G
EMA
epoch
opoch 1+1
EMA Eq
Pg
e
mamory
mamory
MPPM
Update
Update
Figure 2: The proposed framework for SS-SDG. Our framework consists of two stages. At stage 1, we jointly train a feature extractor G in a self-supervised manner by excavating samples intrinsic supervision with contrastive learning (Stage 1-1), and generate new samples to enrich the diversity of source domain in an adversarial way (Stage 1-2) without label information. At stage 2, We propose a novel multi-pair FixMatch (MPFM) loss to better utilize the rich pair information for better generalization unseen target domains.
cross-entropy loss with source labeled samples. We also propose novel loss function, Multi-Pair FixMath (MPFM) in Eq. equation 10 to propagate usefuu training signals from source unlabeled samples and generated samples. Firstly, MPFM generates pesudo-labels utilizing the classifier C's output on weakly transformed unlabeled images. These pesudo-labels will be treated as true labels for strongly transformed version of same sample to train C and G with cross-entropy loss. Here the weak transformation indicates a simple data augmentation which includes random cropping and horizontal flipping while strong Iransformation denotes extensive data augmentation like RandAugment (Cubuk al, 2020 and CTAugment (Berthelot al. 2019). The full algorithm is summarized in Algorithm
Below, introduce the framework in detail. Firstly, We start by introducing the SS-SDG problem with formal notations in Sec. 3.1. Then, we introduce the model pre-train (Figure 2 "Stage 1-1") in Sec. 3.2 including the contrastive learning loss design and the model parameter update. Further, we illustrate the new label-free adversarial data augmentation (Figure 2 "Stage 1-2") in Sec. Sec.3.3 Finally, we explain our novel multi-pair FixMatch loss (Figure 2 "Stage 2") in Sec. 3.4 and propose the overall training objective.
3.1 Preliminaries
In semi-supervised single domain generalization (SS-SDG), source domain is denoted as D, = {D', D,), where D = {(>}', ) [
is the portion of limited labeled samples and D. (< indicates abundant unlabeled samples. The multiple target domains are defined as D. (D)..., "p.....,p)) where Dj = indicates the data from the i-hh unseen target domain. The source and target domains are sampled from different distributions but share the same label space. Our goal is to learn model {G, C), that perform well on unseen multiple target domains D, by utilizing partially labeled single source domain D.. Tn particular, we aim to learn a feature extractor G : r + that maps the input sample x into an embedding space, and train a classifier, C: 9 + that conducts classification by optimizing:
1 n
min ((C((())))
Σ
(1)
where L general classification loss term. Notice that y is known only the testing phase for the target domains. G and C are learned only with the source domain data. We hereby introduce the learning process.

---

Published in Transactions on Machine Learning Research (11/2023)
3.2 Self-supervision with Contrastive Learning
Previous SDG methods (Volpi et al. 2018 Zhao et al. 2020) may encounter degraded performance in the
SS-SDG setting the source domain data with insufficient labels. Targeting on the lacking label venue, recent contrastive learning (He ct al. 2020
Chen et al. 2020 Grill et al. 2020 Zbontar et al. 2021 Zhu et al. 2021c) methods have shown promising results on unsupervised feature learning. They train the feature extractor by pulling the features from same sample under different augmented views close, and pushing the
features from different samples apart.
Following the spirit, we propose adopt contrastive learning framework, i.e., MoCo (He et al. 2020)
pre-train the feature extractor G with all source samples. As shown in Figure 2 "Stage 1-1: Pretrain", the contrastive framework consists of two feature extractors G and G, two MLP projection heads F and F, strategy
and a memory bank M used to store recent k samples' features by adopting the Iirsst-n-First-- to dynamically update it. Notice that F and F are the MLP projection heads only for the pre-train stage. They are different from our final classifier C. We apply strong transformation (i.c., RandAugment (Cubuk et al. 2020)) on source sample x twice, to obtain two strongly transformed inputs. Then by feeding these two inputs into G and G respectively, we compute the similarity between the transformed sample feature f and all features stored in memory bank and adopt the InfoNCE (Oord al. 2018 loss to optimize the framework:
exp - /T)
CeL - log
(2)
E exp (ff f((/TT
where fi represents the features of the t-th iteration inputs, and t is a temperature controls the concentration level of the distribution (Hinton et al.] 2015). To improve the consistency of features in memory bank, we employ the exponential moving average (EMA) strategy (Cai et al. 2021),
that
which could improve the generalization of the learned model, to update parameters (8*) and normalization factors (μ² and o*) in G* and F from G and F:
8r + at + (1 - a)),
n¹ +- oui + (1 - a)j;:
o* t oot + a)o,
where a is a momentum coefficient close to 1, e.g,, 0.999. The parameters 0 in G and F are learned by standard SGD optimizer. There is no back-propagation through the G" and F The G and F can be viewed as the smooth temporal ensemble of G and F along the training iterations.
3.3 Label-Free Adversarial Data Augmentation
While the self-supervised pre-train provides a good initialization, the problem of generalization to unseen target domains still exits. As we face the challenge of only using a single source domain to train and C, the thumb obstacle is the source data diversity. A highly concentrated source distribution can easily cause
model overfitting. Many existing single domain generalization (SDG) methods (Volpi et al. 2018 Zhao
ct al. 2020 Qiao et al. 2020 Fan et al.] 2021 Wang et al. 2021b) adopt the adversarial data augmentation
lashion [Goodfellow et al.] 2015) to complement the diversity of source domain. They formulate the SDG
problem into a worst-case scenario (Sinha et al. 2018):
min sup (B[C.. d(D., D.) S p)),
(4)
w
where d represents distance metric to evaluate the distribution similarity between source and target do- mains. indicates the largest domain discrepancy between D, and D, in embedding space. &e denotes model parameters optimized by cross-entropy loss Lee with label information. The worst-case scenario (Equation can reformulated into a Lagrangian optimization problem with a fixed penalty parameter 3:
min sup (Ee (1h: Dz)l 3dw (Dy. D.).
(5)
D,

---

Published in Transactions on Machine Learning Research (11/2023)
where Dg indicates the generated domain from Dj and dw denotes the Wasserstein metric (Volpi et al, 2018) applied to preserve the semantics of the generated samples. The overall loss function is formulated as:
LsDa D.) (v; D₂) 3dw D.),
(6)
The new domain Dg is generated from D3 by maximizing LSDG with label information under small number of iterations: 2,+1 t- 2; + n/ vV 2, (1:
Clearly, existing adversarial data augmentation based methods in SDG highly rely on label information used by Lee, whereas the limited labeled source data from SS-SDG setting impede the above SDG methods to generate plentiful new samples to diversify the source data distribution. By scamlessly combining the self-supervised signal introduced in Equation 2 and the worst-case scenario in Equation we formulate our label-free adversarial data augmentation
Co(0; = (o) -dd (Dg. Dz).
(7)
Following the same updating rule, as shown in Figure 2 "Stage 1-2", we imput each source sample into G and G, respectively. We compute the similarity between the one passes G and all features stored in memory bank (a different memory bank from the pre-train stage). With a small number of iterations to maximize Equation by:
J++1 + 23 + - (0,
(8)
In this way, we generate new domain D3 to diversify source domain D, without label information. The data augmentation phase is alternated with the pre-train phase along the training epochs.
3.4 Muiti-Pair FixMatch Regularization
With the augmented D, and original Dz, there are rich information amongst the source domain data now. For instance, we can pair the strongly transformed data with the original data, or strongly transformed data with the weakly transformed data. Furthermore, we have the option to pair the generated data, both strongly and weakly transformed generated data, with the original data. This multi-pair approach significantly enhances our capacity to regulate feature representation learning when compared to the traditional practice of pairing only original data with transformed data. Therefore, it could introduce more diversity into the in our
original domain. As FixMatch (Sohn et al.] 2020) applies similar strong/ weak transformations self-supervised training performance, we consider to formulate
framework, and demonstrates our multi-pair constraint into a consistent format, termed Multi-Pair FixMatch regularization.
Specifically, for rongly transformed data D;- Tr.n,(D.) and weakly transformed data D.r = Tran (Da), we generate pseudo-labels g argmax(p" for weakly transformed sample x" E DD,, where pm C(G(z"))
is the output of classifier C. FixMatch penalizes by assigning the ascudo-label g" as the true label for strongly
transformed sample x² E Dj with cross-entropy loss:
(FM(x) -I(max(p")) 2 p) logg"(("")
(9)
where p" C(G(x")) and is the threshold to decide whether (FM is applied on sample
Further, with strongly and weakly transformed Dj and DW, we adopt our proposed label-free adversarial data augmentation (Figure 2 "Stage 1-2: Data Augmentation") to generate D; and Du respectively. With the generated D, = (D; g. D.J. as shown in "Saure?""tagee 2: Multi-Pair FixMatch", we further apply strong transformation on top of generated Dÿ and Di to derive D= and Dus as the input pairs. These two types of data will be paired with D." create two additional pairs for the MPFM loss:
(MPFM( -I(max(p")) 2 "floog"""""" + "og"""""""))
(10)
where p** (a)()))) an E Das and pe = Ci (x)))) zue E Des. The objective for updating our model (G, C) stagel-2 is:
Ls2 Lee + XLFM + *LMPPM
(11)
where X,y are balancing
and empirically set as and 0.5, respectively. Leg cross-entropy loss over labeled source domain data. Our method procedure is sunmarized in Algorithm
8

---

Published in Transactions on Machine Learning Research (11/2023)
Algorithm Our proposed Algorithm.
Require: D., Dg, D Dÿ initialized G, F, C.
Ensure: Learned G and classifier C
1: for t= to Tp do
b Tp #iterations
2:
Apply Equ. 2 for Pre-train (Sec. 3.2
if Mod(t,Q)=00 then
Q iteration interval
Apply Eqn. g for Data Augmentation (Sec. 3.3 end if end for
7:
for
t= to T'y do
> Ty #iterations
8:
Group 2ÿ) , 21
9:
Apply S-Aug on zy 2g x and W-Aug on 2, 27
10: Apply overall loss Eqn. 11 train G,
11:
end for
4
Experiments
In this section, we evaluate the effectiveness of our method as the following: Firstly, we introduce the Experimental settings. Next, we compare our method to other methods mainly from single domain gener- alization and semi-supervised learning. Then, we provide an extensive ablative study investigating cach of our proposed modules. Last, we present. the latent feature space visualization.
4.1 Experimental Settings
4.1.1 Datasets
We leverage three commonly used datasets i.e, PACS, OfficeHome, and DomainNet20.
(1) PACS (Li (Li et al. 2017) is a recent challenging domain adaptation/genut /gneeralizzaionn benchmark which shows larger domain discrepancy. It consists of seven object categories from four domains, namely art paintings, cartoon, sketch, and photo. For this dataset, evaluate on two SS-SDG settings: 15 labeled samples per class (total 105 labels) and 25 labeled samples per class (total 175 labels). (2) OfficeHome (Venkateswara et al. 2017) contains four domains (art, clipart, product, and real world) with 65 classes. This is one of the canonical domain ddlaptation/generallaaa
benchmarks. We design two SS-SDG settings on this dataset: 10 labeled samples per class (total 650 labels) and 15 labeled samples per class (total 975 labels). subset of DemainNet. We self-construct the setting as picking 4 domains (clipart, painting, real, and sketch) and 20 classes (0-baseball bat, 1-binoculars, 2-bracelet, 3-diving board, 4-goatee, 5-hamburger, 6- 6-hurricane, 7-knee, 8-parachute, 9-pickup truck, 10-pillow, 11-pizza, 12-sandwich, 13-saw, 14- scorpion, 15-speedboat, 16-square, 17-swing set, 18-tent, 19-trumpet) out of the entire dataset. The reason why we create a subset of original DomainNet is that, we find that many categories are with limited samples. When we consider the scttings of 15 or 20 samples per class, not all the categorics defined in DomainNet are valid. Moreover, to indicate the same behavior trend of our trained models, we choose to trim the dataset at the same scale of the other two datasets, OfficeHome and PACS. In this way, we select domains and 20 categories to form DomainNet20. We adopt two SS-SDG settings: 15 labeled samples per class (total 300 labels) and 25 labeled samples per class (total 500 labels).
(3) 200
4.1.2 Baselines
We compare with two main streams of the statc-of-tho-art methods: (1) Single domain generalization meth- ods, namely Adversarial Data Augmentation (ADA) (Volpi et al. 2018 and Maximun-Enuropy Adversarial Data Augmentation (MEADA) (Zhao et al.] 2020 (2) Semi-supervised learning methods, namely Entropy
Minimization (ENT-MIN) (Yves Grandvalct 2004 and FixMatch (Sohn et al. 2020).. Besides, we also pro- vide two kinds of baselines: (1) Integration of Single Domain Generalization and Semi-Superoised Learning Techniques: This approach combines single domain generalization and semi-supervised learning methods,

---

Published in Transactions on Machine Learning Research (11/2023)
specifically Fix+tixMath and MEADA+ Here, we directly incorporate the semi-supervised learning method, iee. FixMatch, with single domain adaptation methods, namely ADA and MEADA. (2) Single Domain Generalization Methods with Pseudo Labels: In this strategy, denoted as ADA+PL and MEADA+PL, we employ FixMatcc to assign pseudo labels to unlabeled source data. Subsequently, ADA and MEADA are applied to the entire dataset, which includes both labeled and pseudo-labeled source data. Furthermore, we include another bascline, namely Empirical Risk Minimization (ERM) (Koltchinskii 2011). and explore the potential upper bounds achievable when full labeled source domain data is available, both with ERM (supervised) and ADA (supervised).
4.1.3 Evaluation Metrics:
For each of the benchmarks, amongst all the domains defined, we iteratively take one domain as the source domain and test all the rest domains. The mean average precision (mAP) is reported by averaging over 3 random splits of all the classes' average precision.
4.1.4 Implementation Details:
For our framework: at stagel in Sec. 3.2 and 3.3 We adopt an ImageNet-pretraineed ResNet18 as the feature extractor G and 2-layer MLP head (hidden layer 512-d, with ReLU) as projection head F. We set memory bank size to 1600 and batch size to 32 in whole training process. At the model pre-train stage in Sec. 3.2 we use SGD optimizer with lcarning rate 0.0005, weight decay 0.0005 and momentum 0.9, and train for 1500 iterations. At data augmentation stage in Sec. 333 we adopt SGD optimizer with learning rate 50.0 and 15 iterations to maximize Equation 77 The data augmentation involved only once at the 200-th iteration during model pre-train. At "Stage 2n in Sec. 3.4 we adopt SGD with learning rate 0.001, weight decay 0.0005, momentum 0.9, batch size 32 and train for 8500 iterations.
For baselines: the ADA and MEADA perform adversarial data augmentat ion with the labeled source data. ADA-FixMatch and +iiMatth jointly apply adversarial data augmentation on labeled source data and utilize FixMatch on unlabeld source data. ADA+PL and MEADA+ firstly employ FixMatch to train a model and assign pseudo-labels to the unlabeled source data, and then apply adversarial data augmentation to augment the entire source domain, which includes both the source data with ground truth labels and the source data with pseudo labels.
4.2
Performance Evaluation
4.2.1 Results on PACS
Quantitative comparison is shown in Table [ We group the methods on top three rows compared to ours on the last row. On the first row, notice that ERM (supervised) and ADA (supervised) use all the dataset labels for supervised learning hence serves as the upper bound, where it is neither the 15 label per class setting nor in the 25 per class setting. We equip ERM with our semi-supervised domain generalization setting for the fair comparison. On the second row, there are two semi-supervised learning methods. One is canonical, the entropy minimization method, the other is recent top method FixMatch. On the third row, we present two cutting-edge domain generalization methods: Adversarial Data Augmentation (ADA) and Maximun-Entropp Adversarial Data Augmentation (MEADA). In the fourth row, we introduce ADA and MEADA as two potential frameworks and combine them with FixMatch, resulting in ADA+FixMatch and M pseudo-labels for unlabeled source data. Subsequently, we apply ADA and MEADA to the source domain, considering both labeled and pseudo-labeled source data in ADA+PL and MEADA+PL, respectively. On 15 instance per class setting, we observe that our method consistently outperforms all the compared methods with significant margin. For example, we get 2.43% better than FixMatch and 14.48% better than MEADA in terms of 'Agg'. - By checking the 25 labels per class setting, we see 2.4% and 15.92% performance gains on "Painting" compared with FixMatch and MEADA, respectively. Further, surprisingly our "Avg." number is approaching the upper bound ERM (supervised) which utilizes all the label information from the datasct within less than 4% gap, whereas our method only utilizes less than 5% (15 per class) or 8% (25 per class) of all the labels. Furthermore, we have observed that simply combining single domain generalization met
Moving to the fifth row, we utilize FixMatch on the source domain to assign
10

---

Published in Transactions on Machine Learning Research (11/2023)
Table 1: (%) on PACS (averaged over 3 random splits). Named in column source domain for training. Rest domains are the testing unseen domains and precision for each testing domain is averaged and reported. (A: Art painting, C: Cartoon, P: Photo, S: Sketch)
labels:
105 (15 per class)
labels: 175 (25 per class)
Method
A
C
P
Avg.
P
S
Avg.
ERM (supervised)
70.9
76.5
43.3
53.1
60.7
70.9
76.5
43.3
53.1
60.7
ADA (supervised)
72.43
71.97
44.63
45.73
58. 70
72.43
71.97
44.63
45.73
58.70
ERM
49.45
50.84
30.36
25.25
38.96
52.51
54.05
30.76
23.61
40.23
ENT-MIN
54.15
55.61
38.08
24.95
43.20
56.10
59.65
36.99
26.92
44.91
FixMatch
57.67
69.13
45.46
42.17
53.61
63.70
68.27
45.79
42.86
53, 16
ADA
47.87
50.71
30.31
30.46
40.08
51.71
53.42
30.33
26.74 74
40.55
MEADA
48.79
52.81
34.23
30.42
41.56
51.97
54.54
32.27
28.19
41.74
ADA-FixMatch
60.02
69.44
45.36
39.24
53.51
59.73
68.97
44.31
40.58
53.40
+
59.99
69.65
46.41
40.55
54.15
61.12
70.10
47.41
40.81
54.86
ADA+PL
59.31
65.69
43.23
39.31
51.89
59.63
66.57
44.43
40.21
52.71
MEADA+PL
62.17
61.92
41.66
37.16
50.72
59.88
65.92
44.59
41.44
52.96
Ours
60.26
69.91
47.30
46.70
56.04
65.58
70.11
48.19
47.95
57.96
Table 2: mA)(%) on OfficeHome (averaged over 3 random splits). Named in column is source domain for training. Rest domains are testing unseen domains and precision for each testing domain is averaged and reported. (A: Art, C: Clipart, P: Product, R: Real world)
labels: 650 (10 per class)
labels: 975 (15 per class)
Method
A
Avg.
Avg.
ERM (supervised)
51.27
49.18
44.12
56.86
50.36
51.27
49.18
44.12
56.86
50.36
ADA (supervised)
53.24
50.02
45.76
57.36
51.60
53.24
50.02
45.76
57.36
51.60
ERM
41.09
35.57
36.09
44.15
39.23
44.94
38.99
37.84
47.85
42.40
ENT-MIN
40.74
37.10
38.69
46.98
40.88
45.43
40.93
39.96
51.09
44.35
FixMatch
40.52
36.53
39.19
47.73
40.99
44.70
41.72
40.21
52.04
44.67
ADA
40.75
35.63
35.83
43.76
38.99
44.67
38.97
37.16
47.43
42.05
MEADA
40.74
35.80
35.86
44.15
39.14
44.97
39.41
37.14
47.71
42.31
-axMMhhathhh
41.41
38.82
40.53
47.62
41.26
41.56
51.48
43.73
MEADA FixMacha
41.79
38.72
40.19
47.73
42.11
44.70
41.72
53.04
45.34
ADA+PL
38.57
37.00
36.35
45.13
39.26
42.68
39.05
38.77
48.84
42.34
MEADA+PL
39.17
37.36
37.61
45.43
39.89
42.13
40.30
39.42
48.01
42.47
Ours
43.55
40.65
40.54
49.13
43.47
47.67
43.39
42.49
53.46
46.75
with semi-supervised learning methods, such ADA+FixMatch and MEADA-FixMalie does not yield comparable results to approach. Additionally, the baselines, ADA+PL and MEADA+PL, encounter challenges due to the unsatisfied quality of the pscudo-labels, resulting in the performance that also falls short of achieving results comparable to our method.
4.2.2 Results on OfficeHome
Evaluation on OfficeHlome is shown in Table 2 Similarly as PACS, we organize the experiment in the same layout, where the same five groups of the compared methods are listed on the top three rows with EMR (supervised) and ADA (supervised) the upper bound. In this evaluation, again we find that our method consistently and significantly outperforms all the compared methods on both of the two settings in Table 2 On 10 instance per class setting, we observe that our method consistently outperforms all the compared methods with significant margin, i.e., 2.8% better than second best on "Art" as source domain and 3.55% better than sccond best on "Clipart". The same trend observed on 15 instances per class sctting on the right column. Besides, as more instances per class is used in training, we see that the performance on the right column is generally higher than the left column setting expected. Another interesting point is while
11

---

Published in Transactions on Machine Learning Research (11/2023)
Table 3: m%) on DomainNet20 (averaged over 3 random splits). Named in column is source domain for training. Rest domains are the testing unseen domains and precision for each testing domain is averaged and reported. (C: Clipart, P: Painting, R: Real, S: Sketch)
labels: 300 (15 per class)
labels:
500 (25 per cluss)
Mothod
P
Avg.
P
R
S
Avg.
ERM (supervised)
59.57
64.74
55.57
55.82
58.93
59.57
64.74
55.57
55.82
58.93
ADA (supervised)
60.22
65.69
56.17
56.07
59.54
60.22
65.69
56.17
56.07
59.54
ERM
44.87
62.27
39.38
42.34
44.72
48.81
54.21
44.56
47.47
48.76
ENT-MIN
45.22
53.78
46.84
42.36
47.05
50.93
53.14
49.59
49.73
50.83
FixMatch
51.45
55.2
50.17
49.56
51.61
54.23
57.42
50.35
53.26
53.7 72
ADA
45.28
52.11
39.69
43.10
45.05
48.93
53.68
44.82
48.74 78.74
49.04
MEADA
46.48
52.55
40.93
43.37
45.83
49.74
53.50
45.08
49.50
49.46
ADA+FixMatch
51.47
52.25
47.91
50.58
50.55
54.25
54.85
51.04
52.61
53.19
+
52.13
49.74
49.88
48.10
49.96
54.28
51.90
53.15
53.12
53.11
ADA+PL
43.79
51.06
39.27
45.24
44.84
48.94
53.07
45.60
49.85
49.37
MEADA+PL
46.35
53.03
39.88
43.57
45.71
49.11
52.98
45.78
51.02
49.72
Ours
53.31
57.52
52.61
52.91
54.09
56.21
59.35
53.38
56.18
56.28
our performance to upper bound gap is less than 7%, our setting only uses less than 18% labels of OfficeHome (10 instances per class) and 26.71% (15 instances per class). Furthermore, we still find that our label-free adversarial data augmentation based framework consistently and significantly outperforms ADA, MEADA, and their variant combinations, with more than 10.0% advantage under different settings.
4.2.3 Results on Dem2
The original DomainNet (Peng et al. 2019 is relatively a large scale domain adaptation benchmark with 6 domains and overall 345 categories. Since our setting is supervuper,, where we only utilize a few labeled data, e.g, we constrain only 15 or 25 samples per class presenting their labels. In this way, for those categorics that are with less than 15 images, we discard them from the training data. Mcanwhile, to provide a similar-scale evaluation to other benchmarks such as OfficeHome and PACS, we pick 4 out of domains and 20 out of the 345 categories which contain much more samples in each category form our training and testing data for demonstration. In Table 3] method's performance across the 300-label and 500- label settings show clear advantage over the other methods. Compared to the most competitive opponent FixMatch, our method surpasses by 2.47% on 300-label " "Avg." and 2.56% on 500-label "Avg.". Compared to the representative SDG methods, i.e., MEADA, we see 8.26% performance gain on 300-label "Avg. and 6.82% performance gain 500-label " "Avg.". In this dataset, we observe that the ERM baseline performs at the same level as those single domain generalization methods, partially suggesting that this dataset is more challenging as label information other than domain discrepancy becomes more critical, where the self- supervised methods can benefit more. Notice that our method "Avg" presents close performance to the upper bound ERM (supervised) and ADA (supervised) with less than 5% accuracy gap, but using only 7.7% (15 per class) or 12.96% (25 per class) of the overall labels, which further indicates the effectiveness of our self-supervised adversarial data augmentation design.
4.3 Ablation Study
We ablate proposed core components in Table 4 We clearly decompose the loss terms and abbreviate the simple baseline (ERM) "BL" and the strong baseline FixMatch "FM". LeL refers to pre- training stage 1-1 and data augmentation stage 1-2. LMPPM refers to that, in stage 1-2, we further conduct LFADA on weakly and strongly transformed data to generate data for the multi-pair loss. Compared to the naive ERM "BL", the strong baseline "FM" achieves much better performance. Our intention was not to show the progressive improvement on accuracy from "BL" to "FM" and then to "Ours". Indoed, we aim demonstrate that proposed CMPFM and LeL can orthogonally improve the performance of baselines, e.g., comparing "BI." to "BL+S1", and "PM" to "FM+S1", where clear boost of 1. 1.8% and 0.8% is observed,
12

---

Published in Transactions on Machine Learning Research (11/2023)
Table 4: Ablation study on PACS with 15 per class setting. Named in column is source domain. Rest domains are the testing unseen domains, (A: Art painting, C: Cartoon, P: Photo, St Sketch)
PACS Lie LPM DCL CMPFM
A
C
P
S
Avg.
49.45
50.84
30.36
25.25
38.96
BL+S1 v
50.60
52.40
32.75
27.78
40.88
FM V
57.67
69.13
45.46
42.17
53.61
FM+S1 V
/
59.39
69.61
45.67
43.02
54.42
Ours / /
/
/
60.26
46.70/66.044
Accuracy (55
Accuran auuurayy (16)
Accuracy
60-
12
20
2.05
10.5
135
augmentations 'Slage
augmentt "Stage
(a)
(b)
(c)
(d)
Figure 3: (a) Accuracy versus different number of augmentations in "Stage 1 (b) Accuracy versus when augment in "Stage 1 (c) Accuracy versus different values of X. (d) Accuracy versus different. values of y. Here, the Sketch, comes from PACS dataset, is used as source domain
respectively. Furthermore, by comparing "Ours" to "FM+S1" and "FM', achieve another 1.6% and 2.4% accuracy gain. Each of the incremental combination demonstrates the effectiveness of the components, such as the label-free adversarial data augmentation and the Multi-Pair FixMatch, suggesting the LFADA is advantageous to handle semi-supervisod single domain generalization (SS-SDG) problems.
4.4 Number of Augmentations in "Stage 1"
We study the effect of the hyper-parameter: the number of augmentations in "Stage 1", on PACS dalaset where Sketch is adopted as source domain. We plot the accuracy curve under different augmentation times. As shown in Fig. 3 (a), we find that the accuracy reaches the best value when augmenting twice and the accuracy is slowly getting worse with further increased the number of augmentation.
4.5 When to Do Augmentation in "Stage 1"
We explore the influence of when to augment in "Stage 1" on PACS dataset where Sketch is used source domain. The experimental results are reported in Fig. 3 (b). Here, the maximum iteration in "Stage 1" is set to 1500. It can be observed that the performance of our method is improving with augmentation being executed later in "Stage 1". Because a more well pre-trained model "Stage 1" benefits more on our data augmentation process. After 800 iterations, the choice of different iterations for data augmentation is very close, demonstrating that the pre-trained model in "Stage 1" after 800 itcrations alrcady converged.
4.6 Hyper-parameters Sensitivity Study
We discuss the sensitivity of hyper-parameters À and in Eq. equation 11 by evaluating them on PACS dataset where Sketch is selected as source domain. The value of hyper-parameter À is selected from (0.1, 0.2, 0.4, 0.8, 1.0, 1.2, 1.5, 1.8, 2.0) and the value of hyper-parameter ? fixed to 0.5. Fig. 3 (c) shows that our method relatively stable in the range [0.1, 2.0] and gets better performance in the range [0.25, 0.75). The value of hyprr-parameter y is selected from (0.1, 0.2, 0.4, 0.5, 0.8, 1.0, 1.2, 1.5, 1.8, 2.0) and the value
13

---

Published in Transactions on Machine Learning Research (11/2023)
(a) ERM
(b) ADA
(c) MEADA
(d) CADA-SDG (Ours)
Figure 4: The t-SNE visualization of feature space with baselines, i.e. ERM, ADA, and MEADA, and our method. Same color indicates the same category. Circle indicates source domain. Star indicates unseen target domains. Best viewod in color and zoomed in.
of À is fixed to 1.0. As shown in Fig. 3 (d), the classification accuracy is relatively stable in the range [0.75, 1.5]. When is out of the range [0.75, 1.5], the performance is slightly degraded.
4.7 Visualization
To give intuitive understanding of proposed method, visualize feature space extracted by models trained with our method and the baseline methods, i.e., ERM, ADA, and MEADA, PACS dataset where Sketch is the target domain. As shown in Figure 4 Tt appears that our model yields clear better separation of different categories. Meanwhile, we observe that for different shapes in the same color, with our method, the data points tend to cluster closer, while other methods leave those same classes but different domain data points separated. The closer clustering indicates better aligument of source domain to target domains. Our method presents clear lower domain gap than baselines, i.e., ERM, ADA, and MEADA, demonstrating that our method indeed continuously generalizes towards the unseen target domains.
5 Conclusion
In this work, we propose to solve a new problem under realistic sctting, namely the scmi-supervised single domain generalization, where number of domains for generalization single and with only very limited label information of training data. We leverage expertise from self-supervised learning and propose multi-pair FixMatch loss to mitigate the lack of label issue. Further, we newly introduce label-free adversarial data augmentation to enrich the source domain distribution under insufficient label information scenario. Our augmentation is significantly different from traditional adversarial ones that heavily rely on label information. Extensive study across three challenging domain generalization benchmarks demonstrates our method's advantage, not only surpassing state-of-the-art methods, but also approaching the fully supervised upper bounds.
The work of R. Zhu and S. Li is in part supported by the the U.S. Army Research Office Award under Grant Number W911N9-21-1-01099 and the National Science Foundation under Grant IIS-2316306.
Acknowledgement
References
Shuang Ao, Xiang Li, and Charles Ling, Fast generalized distillation for somi-supervised domain adaptation. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 31, 2017.
David Berthelot, Nicholas Carlini, Ekin D Cubuk, Alex Kurakin, Kihyuk Sohn, Han Zhang, and Colin Raffel. Remixmatch: Semi-supervised learning with distribution matching and augmentation anchoring.
Tn International Conference on Learning Representations, 2019.
14

---

Published in Transactions on Machine Learning Research (11/2023)
David Berthelot, Nicholas Carlini, Ekin D Cubuk, Alex Kurakin, Kihyuk Sohn, Han Zhang, and Colin Raffel. Remixmatch: Semi-supervised learning with distribution alignment and augmentation anchoring. In International Conference on Learning Representations, 2020.
David Berthelot, Rebecca Roelofs, Kihyuk Sohn, Nicholas Carlini, and Alex Kurakin. Adamatch: A unified approach to semi-supervised learning and domain adaptation. arXiv preprinl arXiv:2166. 00060.473, 2021.
Zhaowei Cai, Avinash Ravichaudran, Subhransu Maji, Charless Fowlkes, Zhuowen Tu, and Stefano Soatto. Exponential moving average normalization for self- supervisd and semi-supervised learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 194-203, 2021.
Tianle Chen, Mahsa Baktashmotlagh, Zijian Wang, and Mathieu Salzmann. Center-aware adversarial aug- mentation for single domain generalization. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pp. 4157-4165, 2023.
Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. simple framework for contrastive learning of visual representations. In International Conference Machine Learning, pp. 1597-1607, 2020.
Ekin D Cubuk, Barret Zoph, Jonathon Shlens, and Quoc V Le. Randaugment: Practical automated data augmentation with a reduced search space. In Proceedings of the IEEE/CVF Conference on Compuler Vision and Pattern Recognition Workshops, pp. 702-703, 2020.
Rui Dai, Yonggang Zhang, Zhen Fang, Bo Han, and Xinmei Tian. Moderately distributional exploration for domain generalization. arXiv preprint arXic:2304.13976 2023.
Jeff Donahue, Judy Hoffman, Erik Rodner, Kate Sacnko, and Trevor Darrell. Semi-supervised domain adaplation with instance constraints. In Proceedings of the IEEE conference computer vision and pattern recognition, pp. 668 675, 2013.
Qi Dou, Daniel Coelho Castro, Konstantinos Kamnitsas, and Ben Glocker. Domain generalization via model-agnostic learning of semantic features. In Advances in Neurol Information Processing Systems, volume 32, pp. 6450-6461, 2019.
Yingjun Du, Jun Xu, Huan Xiong, Qiang Qiu, Xiantong Zhen, Cees GM Snoek, and Ling Shao. Learning to learn with variational information bottleneck for domain generalization. In Proceedings of the European conference on computer vision, pp. 200-216. Springer, 2020.
Xinjic Fan, Qifei Wang, Junjie Ke, Feng Yang, Boqing Gong, and Mingyuan Zhou. Adversarially adaptive normalization for single domain generalization. In Proceedings of the IEFE/CVP Conference on Compuler Vision and Pattern Recognétion, pp. 8208 8217, 2021.
Yaroslav Ganin and Victor Lempitsky. Unsupervised domain adaptation by backpropagation. In Interna- tional conference machine learning, pp. 1180 1189. PMLR, 2015.
Yaroslav Ganin, Evgeniya Ustinova, Hana Ajakan, Pascal Germain, Hugo Larochelle, François Laviolette, Mario Marchand, and Victor Lempitsky. Domain-adversnn training of neural networks. The journal of machine learning research, 17(1):2096 2030, 2016.
lan J Goodfellow, Jonathon Shlens, and Christian Szegedy. Explaining and harnessing adversarial examples.
In International Conference on Learning Representations, 2015.
Arthur Gretton, Karsten Borgwardl, Malte Rasch, Bernhard Schölkopf, and Alex Smola. A kernel method for the wwo-sample-problem. Advances in newral information processing systems, 19, 2006.
Jean-Bastien Grill, Florian Strub, Florent Altché, Corentin Tallec, Pierre H Richemond, Elena Buchatskaya,
Carl Docrsch, Bernardo Avila Pires, Zhaohan Daniel Guo, Mohammad Gheshlaghi Azar, et al. Bootstrap your own latent: A new approach to self-supervised learning. In Advances in neural information processing systems, 2020.
15

---

Published in Transactions on Machine Learning Research (11/2023)
Jiuxiang Gu, Zhenhua Wang, Jason Kuen, Lianyang Ma, Amir Shahroudy, Bing Shuai, Ting Liu, Xingx- ing Wang, Gang Wang, Jianfei Cai, et al. Recent advances in convolutional neural networks. Pattern recognition, 77:354 377, 2018.
Kaiming Hc, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. Momentum contrast for unsupervised visual representation learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 9729 9738, 2020.
Dan Hendryeks and Thomas Dietterich. Benchmarking neural network robustness common corruptions and perturbations. In International Conference Learning Representations, 2019.
Gooffrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the knowledge in a neural network. arXiv preprint arXiv:1503.0255 2015.
Jiaxing Huang, Dayan Guan, Aoran Xiao, and Shijian Lu. Fsdr: Frequency space domain randomization for domain generalization. In Proceedings of the LEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 6891-6902, 2021.
Xun Huang and Serge Belongie. Arbitrary style transfer in real-time with adaptive instance normalization. In Proceedings of the IEEE/CVE international conference computer vision, pp. 1501-1510, 2017a.
Xun Huang and Serge Belongie. Arbitrary style transfer in real-time with adaptive instance normalization. In Proceedings of the IEEE international conference on computer vision, pp. 1501 1510, 2017b.
Vladimir Koltchinskii. Oracle Inequalities in Empirical Risk Minimization and Sparse Recovery Problems: Ecole d'Eté de Probabililés de Saint-Flon XXXVIII-2008, volume 2033. Springer Science Business Media, 2011.
Dong-Hyun Lee et al. Pseudo-label: The simple and efficient semi-supervised learning method for deep neural networks. In International Conference Machine Learning Workshop, volune 3, pp. 896, 2013.
Da Li, Yongxin Yang, Yi-Zhe Song, and Timothy M Hospedales. Deeper, broader and artier domain gener- alization. In Proceedings of the IEEE/CVF international conference on compuler vision, pp. 5542 5550, 2017.
Da Li, Yongxin Yang, Yi-Zhe Song, and Timothy M Hospedales. Learning to generalize: Meta-learning for domain generalization. In Thirty- Second AAAI Conference on Artificial Intelligence, 2018a.
Haoliang Li, Sinno Jialin Pan, Shiqi Wang, and Alex C Kot. Domain generalization with adversarial feature learning. In Proceedings of the IEEE/CVE Conference on Computer Vision and Pattern Recognition, pp. 540005449,, 2018b.
Lei Li, Ke Gao, Juan Cao, Ziyao Huang, Yepeng Weng, Xiaoyue Mi, Zhengze Yu, Xiaoya Li, and Boyang Xia. Progressive domain expansion network for single domain generalization. In Proceedings of the IEEE/CVF Conference on Computer Vision and Patlern Recognition, pp. 224 233, 2021.
Divyat Mahajan, Shruti Tople, and Amit Sharma. Domain generalization using causal matching. In Inter- national Conference on Machine Learning, pp. 7313-7324. PMLR, 2021.
Takeru Miyato, Shin-ichi Maeda, Masanori Koyama, and Shin Ishii Virtual adversarial training; regular- ixation method for supervised and semi-supervised learning. IEEE Iransactions pallern analysis and machine intelligence, 41(8):1979 1993, 2019.
Avital Oliver, Augustus Odena, Colin Raffel, Ekin D Cubuk, and Ian Goodfellow. Realistic evaluation of deep semi-supervised learning algorithms. In Advances in neural information processing systems, 2018.
Aaron van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predictive coding. arXiv preprint Xrx 100. 107.00374, 2018.
16

---

Published in Transactions on Machine Learning Research (11/2023)
Xingchao Peng, Qinxum Bai, Xide Xia, Zijun Huang, Kate Saenko, and Bo Wang. Moment matching for multi-source domain adaptation. In Proceedings of the IEFE/CVF internalional conference on compuler vision, pp. 1406 1415, 2019.
Vihari Piratla, Praneeth Netrapalli, and Sunita Sarawagi. Efficient domain generalization via common- specific low-rank decomposition. In International Conference on Machine Learning, pp. 7728 7738. PMLR, 2020.
Fengchun Qiao, Long Zhao, and Xi Peng. Learning to learn single domain generalization. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognation, pp. 12556-12565, 2020.
Benjamin Recht, Rebecca Roelofs, Ludwig Schmidt,, and Vaishaal Shankar. Do imagenet classifiers generalize to imagenet? In International Conference on Machine Learning, pp. 5389-5400, PMLR, 2019.
Saed Rezayi, Handong Zhao, Ronghang Zhu, and Sheng Li. Xdc: Adaptive cross domain short text clustering. In Proceedings of the 2023 SIAM International Conference on Data Mining (SDM), pp.
SIAM,
2023.
Alexander Robey, George J Pappas, and Hamed Hassani. Model-based domain generalization. Advances in Newral Information Processing Systems, 34:20210 20229, 2021.
Kuniaki Saito, Donghyun Kim, Stan Sclaroff, Trevor Darrell, and Kate Saenko. Semi-supervised domain adaptation vía minimax entropy. In Proceedings of the IEEE/CVF international conference on computer vision, pp. 8050 8058, 2019.
Shiv Shankar, Vihari Piratla, Soumen Chakrabarti, Siddhartha Chaudhuri, Precthi Jyothi, and Sunita Sarawagi. Generalizing across domains via cross-gradient training, In International Conference on Learn- ing Representations, 2018.
Jian Shen, Yanru Qu, Weinan Zhang, and Yong Yu. Wasserstein distance guided representation learning for domain adaptation. Proceedings of the AAAI Conference on Artificial Intelligence, volume 32, 2018.
Weili Shi, Ronghang Zhu, and Sheng Li. Pairwise adversarial training for unsupervised class-imbalanced domain adaptation. In Proceedings of the 28th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, pp. 1598-1606, 2022.
Aman Sinha, Hongseok Namkoong, Riccardo Volpi, and John Duchi. Certifying some distributional ro- bustness with principled adversarial training. In International Conference on Learning Representations, 2018.
Kihyuk Sohn, David Berthelot, Nicholas Carlini, Zizhao Zhang, Han Zhang, Colin A Raffel, Ekin Dogus Cubuk, Alexey Kurakin, and Chun-Liang Li. Fixmatch: Simplifying semi-supervised learning with con- sistency and confidence. 2020.
Antti Tarvainen and Harri Valpola. Mean teachers are better role models: Weight-averaged consistency tar- gets improve semi-supervised deep learning results. In Advances in newral information processing systems, volume 30, 2017.
Hemanth Venkateswara, Jose Eusebio, Shayok Chakraborty, and Sethuraman Panchanathan, Deep hashing network for unsupervised domain adaptation. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 5018-5027, 2017.
Riccardo Volpi, Hongseok Namkoong, Ozan Sener, John C Duchi, Vittorio Murino, and Silvio Savarese. Generalizing to unseen domains via adversarial data augmentation. In Advances in neural information processing systems, 2018
Jindong Wang, Cuiling Lan, Chang Liu, Yidong Ouyang, Wenjun Zeng, and Tao Qin. Generalizing to unseen domains: A survey domain generalization. Xiv preprint orxim:2103.0m 033.0397,, 2021a.
17

---

Published in Transactions on Machine Learning Research (11/2023)
Mei Wang and Weihong Deng. Deep visual domain adaptation: A survey. Newrocomputing. 312:135-153, 2018.
Zijian Wang, Yadan Luo, Ruihong Qiu, Zi Huang, and Mahsa Baktashmotlagh. Learning diversify for single domain generalization. In Proceedings of the IEEE/CVF international conference on computer vision, pp. 834 843, 2021b.
Qizhe Xie, Zihang Dai, Eduard Hovy, Minh-Thang Luong, and Quoe V Le. Unsupervised data, augmentation for consistency training. In Advances in neural information processing systems, 2020a.
Qizhe Xic, Minh-Thang Luong, Eduard Hovy, and Quoc V Lc. Self-training with noisy student improves imagenet classification. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 10687-10698, 2020b.
Han Xu, Yao Ma, Hao-Chen Liu, Debayan Deb, Hui Liu, Ji-Liang Tang, and Anil K Jain. Adversarial attcks and defenses in images, graphs and text: review. International Journal of Automation and Computing, 17:151-178, 2020a.
Zhenlin Xu, Deyi Liu, Junlin Yang, Colin Raffel, and Mare Niethammer. Robust and generalizable visual representation learning via random convolutions. arXiv preprint orXic:2007.13003 2020b.
Xiangli Yang, Zixing Song, Irwin King, and ZZenllin Xu. A survey on deep semi-supervised learning. arXiv preprint Xiv:22103. 00550, 2021.
Ting Yao, Yingwei Pan, Chong- Wah Ngo, Houqiang Li, and Tao Mei. Semi-supervised domain adaptation with subspace learning for visual recognition. In Proceedings of the IEEE conference Computer Vision and Pattern Recognition, pp. 2142-2150, 2015.
Xiangyu Yue, Yang Zhang, Sicheng Zhao, Alberto Sangiovanni Vincentelli, Kurt Keutzer, and Boqing Gong, Domain randomization and pyramid consistency: Simulation-to-real generalization without accessing tar- get domain data. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 2100 2110, 2019.
Sangiovamni-Vnnenttelll
Yoshua Bengio Yves Grandvalet. Semi-supervised learning by entropy minimization. In Advances in newral information processing systems, 2004.
Jure Zbontar, Li Jing, Ishan Misra, Yann LeCun, and Stéphane Deny. Barlow twins: Self-supervised learning via redundancy reduction. In International Conference on Mochine Learning, 2021.
Bowen Zhang, Yidong Wang, Wenxin Hou, Hao Wu, Jindong Wang, Manabu Okumura, and Takahiro Shinozaki. Flexmatch: Boosting semi-supervised learning with curriculum pseudo labeling, In Advances in neural information processing systems, 2021.
Hongyi Zhang, Moustapha Cisse, Yann N Dauphin, and David Lopez-Paz. mixup: Beyond empirical risk minimization. XrXiv preprint arXiv:1710.0944 2017.
Liheng Zhang and Guo-Jun Qi. Wep: Worst-case perturbations for semi-supervised deep learning, In Proceedings of the IEEE/CVE Conference on Computer Vision and Pattern Recognition, pp. 3912-3921, 2020.
Long Zhao, Ting Liu, Xi Peng, and Dimitris Metaxas. Maximum-entropy adversarial data augmentation for improved generalization and robustness. In Advances in neural information processing systems, 2020.
Kaiyang Zhou, Yongxin Yang, Timothy Hospedales, and Tao Xiang. Learning to generate novel domains for domain generalization. In Proceedings of the European conference on computer vision, pp. 561 578. Springer, 2020a.
Kaiyang Zhou, Yongxin Yang, Timothy Hospedales, and Tao Xiang. Learning to generate novel domains for domain generalization. In Computer Etsion-ECCh 2020: 16th European Conference, Glasgow, UK August 23-28, 28, 2020, Proceedings, Part XVI 16, pp. 561-578. Springer, 2020b.
18

---

Published in Transactions on Machine Learning Research (11/2023)
Kaiyang Zhou, Ziwei Liu, Yu Qiao, Tao Xiang, and Chen Change Loy. Domain generalization: A survey. arXiv preprint or Xri:221033. 02503, 2021a.
Kaiyang Zhou, Chen Change Loy, and Ziwei Liu. Semi-supervised domain generalization with stochastic stylematch. arXiv preprint (:Xi:2116.000990 2021b.
Jun-Yan Zhu, Taesung Park, Phillip Isola, and Alexei A Efros. Unpaired image-to-inage translation using cycle-consistent adversarial networks. Proceedings of the IEEE international conference on computer vision, pp. 2223-22222222 2017.
Ronghang Zhu and Sheng Li. Self-supervised universal domain adaptation with adaptive memory separation. In 2021 IEEE International Conference on Dala Mining (ICDM), pp. 1547-1552 IEEE, 2021.
Ronghang Zhu and Sheng Li. Crossmatch: Cross-classifier consistency regularization for open-set single domain generalization. In International Conference Learning Represenlations, 2022a..
Ronghang Zhu and Sheng Li. Self-supervision based semantic alignment for unsupervised domain adaptation. In Proceedings of the 2022 SIAM International Conference on Data Mining (SDM), pp. 1 SIAM, 2022b,
Ronghang Zhu, Xiaodong Jiang, Jiasen Lu, and Sheng Li. Cross-domain graph convolutions for adversarial unsupervised domain adaptation. IEEE Transactions on Neural Networks and Learning Systems, 2021a.
Ronghang Zhu, Xiaodong Jiang, Jiasen Lu, and Sheng Li. Transferable feature learning on graphs across visual domains. In 2021 IEEE International Conference on Multimedia and Expo (ICME), pp. 1-6. IEEE, 2021b.
Ronghang Zhu, Dongliang Guo, Daiqing Qi, Zhixuan Chu, Xiang Yu, and Sheng Li. Trustworthy represen- tation learning across domains. ar preprint arXio:2308.12815, 2023.
Rui Zhu, Bingehen Zhao, Jingen Liu, Zhenglong Sun, and Chang Wen Chen. Improving contrastive learn- ing by visualizing feature transformation. In Proceedings of the IEEE/CVF international conference on computer vision, pp.
2021c
19

---

Published in Transactions on Machine Learning Research (11/2023)
Table 5: (%) on PACS (averaged over 3 random splits). Named in column is source domain for training, Rest domains are the testing unseen domains and precision for each testing domain is averaged and reported. (A: Art painting, C: Cartoon, P: Photo, S: Sketch)
labels: 105 (15 per class)
labels:
175
(25 per class)
Method
A
C
P
Avg.
A
C
P
Avg.
ERM (supervised)
0.26
0.33
0.36
0.42
0.34
0.23
0.34
0.26
0.31
0.28
ADA (supervised)
0.31
0.34
0.39
0..4
0.35
0.23
0.23
0.31
0.24
0.25
ERM
0.46
0.54
0.49
0.45
0.47
0.42
0.38
0.56
0.49
0.46
ENT-MIN
0.36
0.35
0.46
0.39
0.38
0.33
0.48
0.46
0.45
0.43
FixMatch
0.31
0.38
0.33
0.35
0.36
0.33
0.29
0.28
0.34
0.31
ADA
0.41
0.44
0.39
0.40
0.42
0.36
0.32
0.46
0.42
0.41
MEADA
0.36
0.41
0.49
0.35
0.41
0.32
0.39
0.36
0.39
0.38
0.45
0.39
0.35
0.38
0.32
0.41
0.36
0.29
0.36
FEADA+FixMatch
0.35
0.39
0.33
0.41
0.37
0.37
0.29
0.36
0.39
0.37
ADA+PL
0.36
0.38
0.36
0.41
0.38
0.36
0.28
0.36
0.35
0.35
MEADA+PL
0.31
0.33
0.38
0.37
0.35
0.33
0.31
0.33
0.32
0.32
Ours
0.31
0.35
0.34
0.37
0.35
0.31
0.30
0.27
0.28
0.29
Table 6: std(%) on OfficeHome (averaged over 3 random splits). Named column is source domain for training. Rest domains are testing unseen domains and precision for each testing domain is averaged and reported. (A: Art, C: Clipart, P: Product, R: Real_worlee
labels: 650 (10 class)
labels: 975 (15 class)
Method
A
C
P
R
Avg.
A
C
P
R
Avg-
ERM (supervised)
0.30
0.35
0.27
0.32
0.32
0.24
0.26
0.27
0.27
0.25
ADA (supervised)
0.38
0.36
0.34
0.34
0.36
0.27
0.24
0.25
0.28
0.26
ERM
0.35
0.43
0.44
0.38
0.41
0.38
0.34
0.36
0.39
0.37
ENT-MIN
0.37
0.43
0.41
0.35
0.39
0.33
0.37
0.40
0.33
0.35
FixMatch
0.38
0.39
0.32
0.31
0.35
0.29
0.21
0.28
0.25
0.26
ADA
0.37
0.43
0.39
0.38
0.39
0.35
0.27
0.38
0.36
0.34
MEADA
0.41
0.43
0.38
0.37
0.40
0.35
0.39
0.31
0.37
0.36
ADA+ FixMaach
0.41
0.35
0.37
0.35
0.38
0.31
0.27
0.33
0.29
0.30
MEADA+FixMatch
0.34
0.40
0.38
0.31
0.36
0.27
0.28
0.29
0.35
0.29
ADAI | PL
0.37
0.41
0.39
0.44
0.41
0.37
0.39
0.34
0.35
0.36
MEADA+PL
0.38
0.44
0.37
0.41
0.40
0.35
0.38
0.32
0.33
0.35
Ours
0.33
0.34
0.29
0.31
0.32
0.25
0.24
0.26
0.25
0.25
A Appendix
A.1
Standard Deviation of Results
The standard deviation of the experimental results PACS, OfficeHome, and Domain are show in Tables 5
respectively.
A.2 Visualization of Generated Data
We visualize the strongly transformed generated data, e.g., Dao, used in Eq.(10). As shown in Figure 5 (a), besides the overlap of generated and original data, there is a distinct portion non-overlapping generated data represented by the green dots in the figure. Furthermore, the visualization demonstrates that the original and generated samples are notably distant from each other, as indicated by the color connections. This observation effectively confirms the enrichment of data diversity through our approach.
20

---

Published in Transactions on Machine Learning Research (11/2023)
Table 7: std( on DomainNet20 (averaged over 3 random splits). Named column source domain for training. Rest domains are the testing unseen domains and precision for each testing domain averaged and reported. (C: Clipart, P: Painting, R: Real, S: Sketch)
labels: 300 (15 per class)
labels: 500 (25 per class)
Method
C
P
It
Avg.
P
R
Avg.
ERM (supervised)
0.51
0.48
0.53
0.57
0.52
0.43
0.44
0.48
0.47
0..45
ADA (supervised)
0.58
0.46
0.56
0.54
0.53
0.47
0.46
0.51
0.41
0..6
ERM
0.61
0.55
0.59
0.61
0.59
0.58
0.48
0.53
0.54
0.52
ENT-MIN
0.49
0.63
0.31
0.57
0.55
0.42
0.55
0.39
0.49
0.46
FixMatch
0.38
0.43
0.32
0.49
0.45
0.35
0.39
0.28
0.33
0.34
ADA
0.63
0.51
0.55
0.58
0.57
0.51
0.43
0.49
0.48
MEADA
0.59
0.57
0.61
0.58
0.48
0.50
0.42
0.47
0.47
0.45
0.51
0.48
0..7
0.58
0.48
0.45
0.41
0.39
0.43
MEADA+FixMatch
0.39
0.61
0.53
0.47
0.51
0.32
0.45
0.49
0.43
0.42
ADA+PL
0.57
0.58
0.49
0.41
0.51
0.48
0.42
0.43
0.44
0.44
MEADA+PL
0.60
0.54
0.57
0.55
0.56
0.38
0.38
0.45
0.47
0.42
Ours
0.45
0.41
0.44
0.34
0.41
0.32
0.37
0.29
0.28
0.32
Source Domain: Clipart (ResNet50)
mAPQN)
EBM
ENT
Pictaatn
MRAAA
ADAVFFaManD
NEADAFRASEE
ADAME
Olre
Source
Generated
(b)
Figure 5: (a) Visualization of features of source and generated samples ("Giraffe" category in Cartoon domain of PACS). (b) mAP (%) on DomainNet20 with Clipart as the source domain. ResNet50 is selected as the feature extractor.
A.3 Results on Dem2 with ResNet50
As illustrated Figure 5 (b), we chose the Clipart domain as our source domain and employed ResNet50 the feature extractor during training. The remaining domains, namely Painting, Real, and Sketch, were designated unseen testing domains. Our results demonstrate that method achieves comparable per- formance to two upper bounds, namely ERM (supervised) and ADA (supervised). Additionally, it holds a significant advantage over the best bascline, FixMatch, with an improvement of more than 2%. This outcome strongly validates the effectiveness of our approach.
A.4 Ablation Study on D and Dws
We conducted deeper investigation into the dataset D and Dws, which is applied with strong transfor- mation, as described in Equation (10). Specifically, we compared our proposed method with a variant that did not apply strong transformations to the combined dataset D, = Table clearly demonstrate that. applying strong transformations to D, consistently leads to performance improvements.
The results, as displayed in
21

---

Published in Transactions on Machine Learning Research (11/2023)
Table 8: Ablation study on PACS with 15 per class setting. Named in column is source domain. Rest domains are the lesting unsee domains. (A: Art painting, C: Carloon, P: Photo, S: Sketch)
UsCSg Dg and D2
P
S
Avg.
Ours
59.76 68.45 46.71
45.98 55.23
Ours
60.26)69.91 47. 07.33)46.777 70 56.04
22


Reviews:
[{"review_id": "bxPAgrxAkz", "reviewer": "Reviewer_gbFc", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "- Strengths\n1. The paper is well written and presented. In particular, the relationship between the proposed setting and prior settings is clearly explained and presented, e.g., Figure 1.\n2. The paper presented a new setting SS-SDG, where the single source domain contains unlabeled data and limited labeled data, which is expected to generalize to multiple unknown target domains. The setting seems to be more difficult than previous SDG setting. \n3. The proposed method achieved good experimental performance when compared with the baselines. The authors also included an ablation study to examine the effectiveness of the components within the proposed method, which is quite important for checking the proposed method in-depth. \n\n- Weaknesses\n1. The proposed method is quite artificial and heuristics, more like a combination of several existing techniques, i.e., contrastive learning, adversarial data augmentation and FixMatch regularization, without any theoretical motivation/guarantee. \n2. After reading the paper, I feel not so convinced by why the proposed SS-SDG setting is important. It would be more clear if additional explanations are given on this point, e.g., motivations of the setting, some real-world examples. \n3. I have a concern of how to ensure the fairness in experiments when comparing the proposed method with baselines. It seems to me that the proposed method contains many key techniques, e.g., adversarial data augmentation, contrastive learning. I'm wondering if the baselines also had such techniques in the empirical comparison, e.g., whether the baselines were conducted on top of the same degree of data augmentation as that in the proposed method.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper studied a new setting, semi-supervised single domain generalization (SS-SDG), where the model is trained from one partially labeled source domain and generalizes to multiple unknown target domains. An effective framework was proposed for this new setting that consists of three key components: 1) pre-train the model based on contrastive learning; 2) diversify the source domain data by adversarial data augmentation; 3) apply the multi-pair FixMatch regularization in training. The proposed framework outperforms baselines in experiments on three datasets: OfficeHome, PACS and DomainNet20.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "1. I think it is better to add more explanations on why the proposed SS-SDG setting is important to be studied (i.e., weakness 2).\n2. I suggest to add more explanations on how to ensure fairness comparison with baselines (i.e., weakness 3). \n3. Table 1 seems to be unnecessary given the Figure 1. It may be better to keep Figure 1 only.", "broader_impact_concerns": "None", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "o1IhBSMMtH", "reviewer": "Reviewer_5cbU", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strength**\n\n1. The paper studies a novel problem: semi-supervised (SS) single-source domain generalization (SDG).\n\n2. The problem is well-motivated (good introduction section). The paper provides excellent figures and tables in the introduction section to compare related paradigms.\n\n3. The paper clearly identifies the deficiency of existing solutions and proposes an algorithm that closely addresses the deficiency. (That is, the proposed solution is well-motivated and solid.)\n\n4. The experiments contain multiple datasets and ablation studies.\n\n**Weakness**\n\n1. I hope the authors provide more discussions on the SS-SDG problem and outline potential solutions instead of sticking to only the data augmentation-based approaches. Concretely, as one of the first papers in SS-SDG, I encourage the authors to discuss the challenges and opportunities of SS-SDG broadly, which could strengthen the paper's impact. For instance, sect. 2.3 discusses multiple ways to SDG. Can other methods beyond data augmentation be applied or extended to SS-SDG? If not, what is the critical challenge, and how may we resolve them?\n\n2. Returning to the proposed approach, while I buy the motivation of the proposed problem and the deficiency of existing SDG solutions, the proposed solution looks not exciting. Specifically, the proposed solution seems like a straightforward combination of self-supervised data augmentation and existing SSL algorithms (with a small extension to incorporate multiple pairs). This further leads to several doubts. \n\n- First, while existing SDG methods use label-conditioned data augmentation, I suppose in the data augmentation community, there have been \"learnable data augmentation methods\" without the need to provide image labels as input. The current manuscript lacks a discussion on existing data augmentation methods (not necessarily proposed for SDG). I think the authors should provide a detailed survey and see if some existing methods are worth considering as baselines.\n\n- Second, given that the main difference between SDG and SS-SDG is the scarce labeled data, I think a more direct solution is to first apply SSL methods in the source domain to recover the missing labels and then apply standard SDG approaches (maybe with some treatments to overcome noisy labels). Looking at Table 2 and Table 4, SSL methods alone, like FixMatch, have largely improved ERM. Applying SDG methods on top of the true and pseudo-labels should further improve the accuracy. This baseline approach should be compared.\n\n3. I have doubts about whether a simple adversarial data augmentation can discover the variations among domains. I understand this may not be the core problem the manuscript needs to address. However, I am not fully convinced that adversarial data augmentation can create faithful augmented data reflecting actual domain variations. \n\n4. According to Algorithm 1, the authors seem to generate a fixed set of $D_g$ at the end of Stage 1 and use it for Stage 2. However, since $G$ is still updated in Stage 2, I wondered whether $D_g$ should be regenerated or dynamically updated.\n\n5. The authors should provide a stronger upper bound in Tables 2-4 to showcase the challenge of this research problem. Concretely, the current upper bound \"ERM (sup.)\" is simply a supervised model learned with the full source labeled data and without any treatment for domain generalization. In my opinion, the authors should provide an upper bound using SDG methods with full source labeled data. This will show the need to perform domain generalization. Otherwise, the current upper bound seems to suggest that if one can recover the missing labels in the source data, there is no need to perform domain generalization.\n\n6. The paper only considers a single and relatively shallow model backbone ResNet 18. Stronger backbones like ResNet 50 or ViT should be considered.\n\n7. The idea of \"multi-pair\" is not clearly motivated in the introduction. I suggest adding some descriptions to the introduction. Also, in Eq (10), there seem to be other ways to construct the pairs, for example, using s or w on $D_g$ rather than ss or ws. Also, the authors may consider pairing weakly augmented and strongly augmented $D_g$. It may be worth comparing them in Table 5.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper studies a novel problem: semi-supervised (SS) single-source domain generalization (SDG). The key challenge vs. existing SDG works is the scarce labeled source data, making it hard to apply label-conditioned data augmentation to simulate diverse source data for domain generalization. The authors thus propose an algorithm combining *semi-supervised data augmentation* and *semi-supervised learning* to train the model. Experimental results on multiple multi-domain datasets show better accuracy vs. baseline approaches.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "**Major**\n\n1. Please see the above weaknesses. I expect the authors to address them in the rebuttal or revised version.\n\n2. Some technical details could be more precise or better motivated.\n\n- Why is ss or ws needed in Eq (10)? Why do the authors only apply it to generated data rather than original data?\n\n- The term \"augmented\" or \"augmentation\" is a bit overly used. For example, at the beginning of section 3.4. $D_g$ means *augmented* data, and the authors then mentioned strong *augmentation* and weak *augmentation*. I suggest that the authors differentiate these terms in the paper clearly. \n\n3. In Figure 1, the test data of DG and SS-GD should contain multiple test domains like SDG and SS-SDG. In essence, DG trains a model that is expected to be generalizable to various test domains.\n\n4. Please clearly describe how existing SDG methods are applied in the experiments. I suppose the authors only use the scarce labeled data to perform data augmentation. Nevertheless, a detailed experimental setup is recommended. \n\n5. Not all contrastive learning approaches need a memory bank. Thus, the authors should modify section 3.2 to indicate which particular contrastive learning method was used clearly.\n\n**Minor**\n\n1. The paper has several typos and grammatical errors. For example:\n\n- feature extractor pre-train --> feature extractor pre-training\n\n- indicates the i-th unseen target domain data --> indicates data from the i-th unseen target domain. \n\n- Here weak augmentation indicates augmentation --> please modify this sentence.\n\n- Firstly, MPFM generates pesudo-labels utilizing the classifier F’s output --> I suppose this should be \"C\" not \"F\".\n\n- Eq (6) --> I suppose it should be $G_g$ not $D_s$ in the CE loss.\n\n2. The captions of the figures and tables should be self-contended. Thus, please expand the captions to mention the notations used in the figures and tables clearly (specifically, Figure 1 and Table 2). For instance, ERM (sup.) should be clarified.", "broader_impact_concerns": "No concerns.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "5bsDb2wsIw", "reviewer": "Reviewer_Ycg7", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "### Summary \n\nThis paper considered domain generalization problems under the limited observed labelled dataset and unsupervised dataset. The problem setting sounds interesting and compelling. Simultaneously, this reviewer currently has several concerns, where the current recommendation is **borderline positive**.\n\n### Would some individuals in TMLR's audience be interested in the findings of this paper?\n\nSure. This paper considers a relative novel setting in out-of-distribution generation. \n\n\n### Are the claims made in the submission supported by accurate, convincing and clear evidence?\nIn general, the proposed method seems a bit complicated by building up different components. Upon reflection, most building components are believed to be reasonable. Simultaneously, it would be great to have discussion on the following points\n\n\n- Why do we consider Wasserstein distance based adversarial samples? It could be possible to use other forms of attack?\n- In Equation (9), how to set the threshold? Is this threshold important in your experiments?\n- In Equation (10), is this related to prediction entropy regularization? Or to minimize the cross entropy loss?\n- Could you explain why using the data memory in eq (2)? I could not find the rationale. \n- It is suggested to report the variance of the empirical results.\n- It would be better to provide the code and additional visualizations of the proposed methods.  For example, what did the adversarial samples look like? The importance of data-set memory. Since this paper has no source code, appendix, I would strongly recommend a revision by providing details of the proposed framework. \n\n\nSide note: In general, I appreciate authors’ efforts by making a clear presentation on complex modules. However, this reviewer believes there are better ways to clearly present the method/paper rather than simply say we follow paper [xxx]. I would provide several examples.\n\n- Related work (Sec 2.1-2.3) This paper simply presents different aspects such as domain adaptation, generalization, and semi-supervised. However, this reviewer feels quite redundancy in reading this, because this does not provide any insightful discussions about the ***relation** with your current paper.  It is just a simple combination of different recent papers….\n- The comparison in Fig 1 and Tab 1 seem a bit redundant. I do think it is not hard to grasp the novelty in your setting. It would be much better to highlight why your setting is difficult and current approaches **does not work** well. E.g, several empirical results.\n- The third concern is related to motivation. I know the proposed setting is generally uncommon. However, I could not feel a strong motivation of why it is important. Based on the experiments on the standard dataset, I feel it would be a bit unclear for me to understand the importance of your problem.\n- Better explanations through better paper structures. I would strongly suggest a better illustration on the paper, for example why it is important to use EMA in your scenario? It is better to justify rather than saying based on paper [XXX].\n\n\n\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper considered out-of-distribution (or domain generalization) in the context of semi-supervised learning. Specifically, this paper considered two stages approaches. In the first stage, it contains (1) pre-training of the deep network through contrastive learning; (2) data-augmentation based adversarial sample generation; (3) confidence pseudo-label based approach. \n\n", "claims_and_evidence_yn": "No", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "In general, I would think a significant revision on the paper's structure and presentation could improve the paper. Thus authors are suggested to check my detailed comments in the review. \n\n*Actually, the claim and evidence are partially done. Since there is no such an option, I would currently recommend No. (This does not imply no claim or evidence). *", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "nMLZ7GhhPu", "reviewer": "Reviewer_j6st", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n1) the problem setting is novel to current DG community.\n2) The authors further suggest a self-supervised model pretraining method employing contrastive learning, aimed at enhancing the quality of representations. Additionally, they introduce an adversarial data augmentation strategy designed to increase the diversity within the source domain.\n3) The authors adeptly adapt the FixMatch method into a Multi-Pair FixMatch format, specifically tailored to address the SS-SDG problem. The empirical results underscore the efficacy of the proposed framework.\n4) SOTA experimental performance!\n\nWeaknesses:\n1) The thechniques used is not novel. It seems the work is a combination. But it is ok to me.\n2) Do you think semi-supervised source domain is really necessary? I mean it will impact the methods and experiments essentially?\n3) Missing references: please cite the lasest paper related to DG, such as Moderately Distributional Exploration for Domain Generalization, Domain Generalization using Causal Matching", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper introduces a novel task: semi-supervised single domain generalization, specifically tailored to situations that involve a single source domain with only partial labeling, yet need to generalize to multiple unseen target domains. The authors put forth a label-free adversarial data augmentation strategy, designed to amplify the diversity of a single source domain in the absence of label information. Additionally, they propose a Multi-pair FixMatch regularization, which capitalizes on the diversified data to further refine the generalized classifier for unseen target domains. Experimental findings substantiate the superior performance of the proposed method when compared to SOTA baselines.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Please address the issues in weaknesses.", "broader_impact_concerns": "My concern is that the appearance of fundation models, is it necessary to consider DG again?", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The submission studied a new problem setting called semi-supervised single domain generalization which is conceptually interesting and practically important (by the way, Figure 1 is especially nice for understanding the new setting). A learning method was then proposed to solve the above problem as a combination of three previously existing techniques (contrastive learning, adversarial data augmentation, and FixMatch regularization). The author rebuttals successfully addressed most of the concerns from the reviewers. Three reviewers (including an emergency reviewer) recommended acceptance while one reviewer was not responsive and did not submit his/her official recommendation. Thus, I think we should accept the submission for publication at TMLR.
Decision: Accept (poster)