Beyond Loss Functions: Exploring Data-Centric Approaches with Diffusion Model for Domain Generalization
Authors: Sobhan Hemati
Mahdi Beitollahi
Amir Hossein Estiri
Bassel Al Omari
Soufiane Lamghari
Yasser H. Khalil
Xi Chen
Guojun Zhang
Venue: Accepted by TMLR ()

Abstract: There has been a huge effort to tackle the Domain Generalization (DG) problem with a focus on developing new loss functions. Inspired by the image generation capabilities of the diffusion models, we pose a pivotal question: Can diffusion models function as data augmentation tools to address DG from a data-centric perspective, rather than relying on the loss functions? Our findings reveal that trivial cross-domain data augmentation (CDGA) along with the vanilla ERM using readily available diffusion models without additional finetuning outperforms state-of-the-art (SOTA) training algorithms. 

This paper delves into the exploration of why and how this rudimentary data generation can outperform complicated DG algorithms. With the help of domain shift quantification tools, We empirically show that CDGA reduces the domain shift between domains. We empirically reveal connections between the loss landscape, adversarial robustness, and data generation, illustrating that CDGA reduces loss sharpness and improves robustness against adversarial shifts in data. Additionally, we discuss our intuitions that CDGA along with ERM can be considered as a way to replace the pointwise kernel estimates in ERM with new density estimates in the \textit{vicinity of domain pairs} which can diminish the true data estimation error of ERM under domain shift scenario. These insights advocate for further investigation into the potential of data-centric approaches in DG.

Under review as submission to TMLR
Beyond Loss Functions: Exploring Data-Centric Approaches with Diffusion Model for Domain Generalization
Anonymous authors Paper under double-blind review
Abstract
There has been a huge effort to tackle the Domain Generalization (DG) problem with a locus on developing new loss functions. Inspired by the image generation capabilities of the diffusion models, we pose a pivotal question: Can diffusion models function as data augmentation tools to address DG from a data-centric perspective, rather than relying the loss functions? Our findings reveal that trivial cross-domain data augmentation (CDGA) along with the vanilla ERM using readily available diffusion models without additional finetuning outperforms state-of-the-art (SOTA) training algorithms.
This paper delves into the exploration of why and how this rudimentary data generation can outperform complicated DG algorithms. With the help of domain shift quantification tools, We empirically show that CDGA reduces the domain shift between domains. We empirically reveal connections between the loss landscape, adversarial robustness, and data generation, illustrating that CDGA reduces loss sharpness and improves robustness against adversarial shifts in data. Additionally, considered as way to replace the pointwise kernel estimates in ERM with new density estimates in the vicinity of domain pairs which can diminish the true data estimation error of ERM under domain shift scenario. These insights ad advocate for further investigation into the potential of data-centric approaches in DG.
discuss intuitions that CDGA along with ERM can be
1 Introduction
Out-of-distribution (OOD) generalization stands as crucial capability for deep learning models in real-world scenarios. The prevalent setting for investigating OOD generalization is termed domain generalization (DG) Blanchard et al. (011), involving multiple source domains to generalize to an unseen target domain. In DG problems, there is a shift between the training domains and the larget domain which makes the models trained using Empirical Risk Minimization (FRM) Vapnik (1999b) struggle to maintain their performance the target domain.
To enhance OOD generalization within the DG framework, researchers have proposed innovative loss functions- achieved by introducing regularizers to TR facilitate the learning of domain- invariant mechanisms across domains. Nevertheless, none of these approaches consistently outperforms others across all datasets, as illustrated by results from the DomainBed benchmark (Gulrajaní Lopez-Pat Paz 2020 This observation suggests that a singular regularizer capable of capturing all invariances might not exist. We posit that the absence of such a universal regularizer arises from the diverse shifts present each datasot, encompassing correlation shift, diversity shift, label shift, etc. Consequently, a rigid, data-independent regularizer may fall short in eliminating all types of spurious correlations and shifts. Additionally, the incorporation of sub-optimal regularizers can exacorbate optimization challenges, introducing excessive risk [Sener& & Koltun 2022), additional hyperparameters, and computational bottlenecks in ERM.
Rather than relying solely on traditional loss functions, recent advancements in generative foundation models open up new avenues for addressing the DG problem from a data-centric standpoint. Specifically, the capability of Denoising Diffusion Models (Ho et al. 2020 [Song et al.) 2020 Rombach al. 2022) in generating high-fidelity synthetic images offers innovative approach for advanced data augmentation,

---

Under review as submission to TMLR
10
ERM CDGA
State-of toate-ofthhe-at
8.5%
Improvements over ERM (%)
8
6
5.7%
5.1%
4.3%
4
Hutch- inson
2.4%
2
0.8%
Fishr
CORAL
PACS
OfficeHome
VLCS
Net
Dataset
Figure Improvements of CDGA + ERM and state-of-the-art (SOTA) DG training methods over ERM across four dataset using DomainBed benchmark. See Section for experimental details. CDGA + ERM outperforms SOTA algorithms across all datasets, highlighting the absence of singular algorithm achieving best, results across dalasets.
enabling the creation of domain-invariant images to enhance OOD generalization. To examine this hypothesis, we employ a straightforward Cross Domain Generative Augmentation (CDGA) method. CDGA, synthetic images are generated conditioned on images text descriptions from all possible combinations of the training domains using pre-trained latent diffusion model (T.D)) (Rombach et al. 2022). In Figure
show
that applying vanilla ERM to combined generated and real images outperforms the previous state-of-the-art algorithrns across all dalasels.
This paper delves into an exploration of the reasons and mechanisms behind the superior performance of CDGA's simplistic data generation strategy compared to complex DG training algorithms. Our investigation involves quantifying and visualizing domain shifts across domains of generated syuthetic images, validating that cross-de cross-domant domain data generation mitigates the gap between domains. We also discuss our intuitions that CDGA be seen way to replace pointwise kernel estimates in ERM with density estimates in the proximity of domain pairs. This modification to ERM can the inherent data estimation error in the presence of domain shift, subsequently enhancing its out-of-distribution (OOD) performance. Furthermore, our empirical results establish connections between the loss landscape, adversarial robustness, and data generation, revealing that cross-domain data generation lessens loss sharpnoss and improves robustness against adversarial shifts data. To the best of our knowledge, we are the first to utilize latent diffusion models as a data-centric approach for DG.
Our primary contributions as follows:
Demoustrating superior performance, our study reveals that combining CDGA with vanilla ERM outperforms srate-of-the-ate DG training algorithms. We validate this across four datasets using three model selection strategies.
Employyng various metrics such as transfer measure, diversity shift, and near-duplicate quantification, we empirically demonstrate that CDGA reduces distribution shift among domains, attributing to its superior performance.
Providing possible intuitions that CDGA can be seen as way replace pointwise kernel estimates in ERM with new density estimates in the proximity of domain pairs similar to the Vicinal Risk Minimization principle (VRM) (Chapelle al. 2000) within DG setup.
2

---

Under review as submission to TMLR
4. Through empirical analysis, calculate the loss landscape sharpness of CDGA during training with FRM,, showcasing its reduced sharpness compared to vanilla ERM. Additionally, we demonstrate the robustness of CDGA + ERM against two adversarial shift attacks.
5.
Our extensive ablation studies compare single-domain and cross-lomain data generation, underscoring the necessity of cross-domain generation for substantial improvements. Furthermore, our ablation on the size of generated data highlights its potential as a method for mitigating class imbalance challenges.
2 Problem Settings and Related Work
We denote our prediction model f and its parameters as 0. In DG, the goal is to learn a shared model from source (train) population domains (environments) {EI,...,En, Enh, to generalize to an unseen target domain T. For a given domain ε, the classification loss is:
(0) = (( [[[ 0),
(1)
where each x and y are the input data point and its corresponding label and {ff (). y) is the cross entropy loss between [xx;;) () and y-
ERM The standard baseline for training deep learning models is ERM Vapnik 1999b which minimizes the average of losses over the entire available training domains i.e., {S...... Sp)
S. Σ (f(())0),
min Σ S
(2)
where (rt, vit ~ & is i.i.d. sample from the training set S of domain i with |S;] samples, ot is the k-th data point in S; and vi is its associated label. However, in the case of domain shift between domains, the pointwise kernel estimation of true data distribution proposed by ERM in Eq. 2 becomes less accurate which results in lack of OOD generalization of ERM.
Improving the Loss Function:
In the pursuit of improving OOD performance within ERM, diverse
avenues have been explored. Notable efforts include robust optimization (Sagawa et al. 2020
Hu et al. 2018
invariant representation learning on the feature level (Sun & Saenko 2016 Ganin et al, 2016 Li al. 2018
Tzeng et al. 2014), and classifier head adjustments (Arjovsky et al. 2019 Additionally, ad aancementss in loss functions, such as those discussed by Krueger et al. (2021). reveal their efficacy in enhancing domain generalization. Further insights stem from investigations into loss gradients/Hessians /Hessias (Parascandolo et al., 2020; Shahtalebi et al., 2021 Koyama & Yamaguchi 2020 Shi et al. 2021 Hemati et al. 2023), showcasing the multifaccted approaches taken to fortify ERM against domain shifts.
Data Augmentation for DG: Various strategies enhance OOD performance in Empirical Risk Minimization (ERM). Classic data augmentation, as explored by Gulrajani & Lopez-Paz (2020), demonstrates improved results under the DomainBed evaluation protocol. [sse ct al. (2021) introduces Select Data Augmentation, method that identifies transformations detrimentally affecting validation accuracy. Mixup (Zhang al. 2017) generates mixed data points and soft labels through linear convex combinations from two classes, while MixStyle (Zhou et al. 2021) combines per-sample feature statistics (mean and variance) across domains. Somavarapu et al. (2020) introduces a stylization transformation based on in-domain data. To the best of our knowledge, we are the first to utilize latent diffusion models as a data-centric approach for DG.
Denoising diffusion models and their applications. Recent advances in diffusion-based generative models (Ho et al. 2020, Song et al. 2020 Rombach et al.. 2022 Zhang & Agrawala, 2023) demonstrate their capability to achieve SOTA image quality. Additionally, works such as Unclip (Ramesh et all 2022 have successfully integrated Foundation models like CLIP (Radford et al. 2021p with stable diffusion, introducing new generative functionalities such as image-lo-image, text-to-imaae, image variation, and image mixing lo diffusion-baase models. The application of diffusion models in representation learning has also been explored, as exemplified by StableRep proposed by Tian et al. (2023). In a setup akin to SimCLR (Chen et al. 2020), they demonstrated that synthetic images generated by stable diffusion models can enhance self-supervised learning.
3

---

Under review as submission to TMLR
3 Cross Domain Generative Augmentation
In this section, we provide a detailed description of CDGA. CDGA utilizes LDM to perform transformation denoted by M(-). This transformation takes two arguments: data point in one domain and guidance attribute in another domain from the same class. Formally,
=M guide"),
(3)
where a is synthetic image transformed from domains i and j, generated from the k-th sample in S.. The attribute guide' serves as guidance towards another domain, Sj, within the same class. The workflow is illustrated in Figure 2
Input
Domain 1
"Painting"
Domain 1,2
Domain 2
LDM
"Cartoo""
Domain 1,3
Domain 3
LDM
Figure 2: Illustaation of CDGA. For each input image of domain, we generate a new image using the image the description of another domain.
ERM
ERM+CDGA
ERM+CDGA*
P
A
P.P
A.A
P.P
A,A
PA
A,P
PA
A,P
Test
P.S
A,S
Figure 3: Illustaation of the implementation structure of ERM, CDGA, and CDGA* PACS dataset when using P and A domains training and S as target domain.
In CDGA, each data point in domain S; undergoes transformation to all n domains, including its own domain. This augmentation increases the number of samples for domain S, from |S,] to (b x n + +1 x [Sy], where n is the number of training domaïns, |S,] is the number of data points S,, and b is the generation batch size. Furthermore, we introduce CDGA", where we assume access to a guidance attribute of the target domain. In this scenario, the size of domain S, increases from |S;| to (b (n (1++))+ 1) x [S For inplementing CDGA, offline augmentation where we first generate images between each pair of training domains and then start the training process. As an example, the folder structure of our implementation for the PACS dataset. when using P and A domains as train domains and S domain for test domain is illustrated in Figure 3 For all the methods, we set generation batch size b = unless stated otherwise.
CDGA with Prompt Guidance (CDGA-PG): CDGA-PG, given the k-th image S,, i.c., 2t, the guidance attribute guide domain description text prompt that represents the same class in Sy. Having the image and the prompt guidance, the LDM to generate b synthetic images which expect to
4

---

Under review as submission to TMLR
interpolate domains i and for the same class. For each image in S₁, perform these image translations for all the training domains j. Vj E n1. We also consider the scenario where we can utilize the target. domain description, ie., guide' as the guidance.
CDGA with Image Guidance (CDGA-IG): For scenarios where a text prompt description of domains is not available, CDGA-IG is used where the guidance an image from Sj instead of a text description. More precisely, in CDGA-IG we attempt mix two images from two different domains which is also known as the image mixer in the literature.
4 CDGA Outperforms SOTA
In this section, we compare CDGA + ERM with SOTA DG training methods, demonstrating its superior performance. We assess CDGA and CDGA* on for datasets, namely VLCS (Fang et al. 2013). PACS Li et al. 2017), OfficeHome /Venkateswara et al. 2017), and DomainNet (Peng et al. 2019), using the DomainBBd benchmark (Gulrajani Lopez-Pase Paz, 2020). This benchmark has gained popularity as a fair and standard evaluation platform for domain generalization algorithms. The evaluation process involves comparing DG algorithms across 20 hyperparameter choices and 3 trials, utilizing three distinct model selection techniques. To demonstrate CDGA's effectiveness, we present its evaluation results using the DomainBed benchmark in Tables The tables follow format of presenting the first and second results. For brevity, we report only the top five performing algorithms for each model selection, with full results available in the appendix. Examining Tables 14 CDGA" consistently achieves SOTA performance across all datasets and model selection techniques. Specifically, we applied prompt guidance for PACS, OfficeHome, and NemainNet, while using image guidance (i.e., image mixer) for VLCS. The code implementation for deploying CDGA-generan data within the DomainBed scheme is detailed in Appendix H
Table 1: DomainBed benchmark for training-domain validation set model selection method.
Algorithm
PACS
OfficHHome
DomainNet
Avg
ERM
85.5
66.5 0.3
40.9
61.3
CORAL
86.2 + 0.3
68.7 + 0.3
41.5 + 0.1
65.5
SagNet
86.3 0.2
68.1 0.1
40.3 0.1
61.9
Fish
85.5 + 0.3
68.6 + 0.4
42.7 + 0.2
65.6
Fishr
85.5
67.8
41.7
65.0
HGP
84.7 + 0.0
68.2 à 0.0
41.1 + 0.0
64.7
ERM CDGA-PG
88.5 # 0.5
68.2 # 0.6
43.7 #0.1
66.6
ERM EDGA--PG
89.5
70.8
44.8 40.0
68.4
Table DomainBed benchmark for leave-one-domain-out cross-validation model selection.
Algorithm
PACS
OfficeHome
DomainNet
Avg
ERM
83.0 + 0.7
65.7 # 0.5
40.6 + 0.2
63.1
CORAL
82.6 0.5
68.5 0.2
41.1 0.1
64.1
SagNet
82.3 4 0.1
67.6
: 0.3
40.2
+ 0.2
63.4
MLDG
82.9 1.7
66.1
= u.5
41.0
= 0.2
63.3
HGP
82.2 0.0
67.5 : 0.0
41.1 + 0.0
63.6
Hutchinson
84.8 0.0
68.5 : 0.0
41.4 # 0.0
ERM + CDGA-PG
86.8 0.4
68.7 + 0.4
43.6 20.1
66.2
FRM + CDGA*-P
88.4 0.5
70.2 0.4
44.8 10.0
67.8
5 CDGA Reduces Domain Shift
In this section, we empirically confirm that CDGA reduces domain shift. To validate the efficacy of CDGA in mitigating domain shift, employ five domain shift quantification techniques from the literature the

---

Under review as submission to TMLR
Table 3: DomainBed benchmark test-domain validation set (oracle)model selection method.
Algorithm
PACS
OfficeHome
DomainNet
Avg
FRM
86.7 + 0.3
66.4 0.5
41.3 0.1
64.8
Mixup
86.8
68.0
39.6 0.1
64.8
MLDG
86.8 + 0.4
66.6 0.3
41.6 0.1
65.0
CORAL
87.1
68.4 0.2
41.8
65.8
SagNet
86.4 + 0.4
67.5 0.2
40.8 + 0.2
64.9
Fish
85.8
66.0 2.9
43.4 +
65.1
Fishr
86.9 + 0.2
68.2 0.2
41.8 + 0.2
65.6
Hutchinson
86.3
68.4 = 0.0
41.9
65.5
ERM CDGA-PG
89.6 + 0.3
68.8 0.3
44.4 #0.1
67.2
ERM + CDGA--PG
90.4 0.3
70.2 + 0.2
44.8 40.0
68.5
Table 4: DomainBed benchmark VLCS dataset.
Training domain
Leave-one domain
Method
Oracle
ERM
77.5 0.4
77.2
77.6
CORAL
78.8 = 0.6
78.7 = 0.4
77.7 = 0.2
SagNet
77.8 = 0.5
77.5 = 0.3
77.6 = 0.1
Fishr
77.8 0.1
78.2
78.2
HGP
77.6 0.0
76.7 0.0
77.3 0.0
Hutchinson
76.8 # 0.0
79.3 0.0.
77.9 = 0.9
ERM + CDGA-IG
78.9 + 0.3
77.9 0.5 79.5 + 0.1
PACS dataset. Specifically, we utilizee t-SNE visualization of feature embeddings, near-duplicate analysis Oquab et al. 2023), transferability (Zhang al. 2021 Hemati et al. 2023), and diversity shift metrics (Ye et al.
2022 to quantify the shift between domains.
5.1
Domain shift Visualization
To visualize domain shifts in CDGA-based data for the class "dog" across all domains (P, A, C, and S), we generate synthetic images for A + A, A P, A + C, and A S. Subsequently, utilize the pretrained CLIP ViT-B/32 3/2 image encoder [Radford et al. 2021 to extract features from both real and synthetic images. These features are then projected onto swo-dimensional space using t-SNE and presented in Figure ! Notably, the cross-domain synthetic images effectively interpolate between different domains, addressing the desired distribution shift. In Figure examining domains (in red) and (in pink) reveals a significant distribution shift in their two-dimensional representations, despite all images belonging to the dog class. However, A S synthetic images seamlessly bridge the gap between A and S representations. Refer to Figure 15 in the appendix for t-SNE plots of other classes.
5.2 Transferability Measurement
Transferability is another recent approach proposed by Zhang et al. (2021) to quantify the domain shift. The transferability measure is upper bound of the difference between the source and the target domain excess risks. Subsequently, Hemati et al. (2023) showed that upper bound for transferability measure is ] H Hsl1z + o(82) where 5 is constant, H. and H5 are target and source classifier hcad's Hessians. Following these findings, to quantily the dynamics of domain shift, we monitor classifier heads Hessian distances between all possible domain pairs through the training steps for ERM and CDGA, where we set domains P, A, and C train (source) and domain as target. Figure shows the difference between the classifier head's Hessians given data from domains A and S during the steps. We see that CDGA and CDGA* lead to lower classifier head's Hessian difference and subsequently smaller transferability and domain shift.

---

Under review as submission to TMLR
C
A
A A A A
A-A A+C A A C S
Figure 4: The t-SNE plot of features extracted from the original PACS dataset and generated images by CDGA from A domain. This figure shows that CDGA can fill the gap between the original domains. Check Section for details.
18
ERM
ERM + CDGA
H Hslla)
ERM + CDGA*
10
6
1000
2000
3000
4000
Step
Figure 5: Classifier head Hessian difference between domains P & S during training.
compared to ERM. This pattern consistently exists for other domain pairs where the full results are presented in Figure 14 the appendix.
5.3 Diversity Shift
2022) proposed numerical method to measure diversity shift which is equivalent to total variation (Zhang et al. 2021 to quantify domain shift. Diversity shift is usually due to the novel domain-specific features in the data. We employ the proposed algorithm by Ye et al. (2022) to quantify and compare diversity shift between training domains and the target domain in a leave-one domain out scheme for PACS real data, CDGA-PACA, and CDGA datasets. Figure X shows both CDGA and CDGA* reduce the diversity shift between training domains and the target domain.
Ye et al.
5.4 Near-duplicate Analysis
We employ near-duplicate image delection on images generated using CDGA to quantify the similarity between the generated and original images in each domain. Following the self-supervised image retrieval technique outlined (Oquab al. 2023). we utilize the protrained CLIP VIT-B/32 image encoder (Radford et al. 2021) extract embeddings and calculate cosine similarity between original and generated images. For each original image, if at least. image in a generated domain exhibits cosine similarity above 0.95,
7

---

Under review as submission to TMLR
Original Domains
0.4%
0.0%
0.0%
0.0%
0.0%
0.3%
0.0%
0.0%
7.6%
6.6%
12.8%
7.1%
1.5%
0.5%
2.9%
3.8%
C-P
C-AA
C-C
C-+5
Generated Domains
Figure 6: Heat map of the percentage of near-duplicates of each original domain in the generated domains.
This table shows that using target-domain description results in more near-duplicate images.
Original images used for augmentation
Generated near-duplicates
Original image
COGA
Cartoon
Cartoon -> Sketch
CDGA
Sketch
Cartoon
Cartoon Cartoon
Figure Examples of near-duplicates (right-most column) found for the dog image in Sketch domain (left-most. column) that are generated using CDGA from the original images (middle column).
we categorize the original domain as having a near-duplicate. Figure 0 provides a summarized view this experiment for the case of generated images from domain C, while the complete results are available Figure 12 in the appendix. In Figure 6 we report, for each original domain, the percentage of near-duplicates relative to the original domain size. Clearly, generating synthetic images within the manifold between training domains allows us to obtain examples that are near-duplicates of the target domain. Figure 7 showcases some of the near-duplicates identified for sample image in the S domain using this technique. Additional examples can be found in Figure 13 in the appendix.
6 Intuitive Discussion: CDGA with ERM an approximate Extention of Vicinal Risk Minimization Principle to DG Setup
In this section, attempt to provide intuitive justification for the reasons behind the success of CDGA. Our justification relies on the Vicinal Risk Minimization principle (VRM) (Chapelle et al., 2000) initially introduced by Vapnik (1999a). The motivation behind VRM is to improve true data distribution estimation in ERM. First note that we can rewrite ERM (Vapnik
1999a) loss in Eq. as
15
e((x;; 2))0))))(())..)))))))) 21)rx drddy,
min
8

---

Under review as submission to TMLR
1.0
Raw Data
CDGA
Diversity Shift
0.8
CDGA*
0.6
0.4
0.2
0.0
S
A
C
Domain Figure 8: Diversity shift as a measure of iid-ness of PACS (raw data) against CDGA, and CDGA* augmented datasets. Each column is the target domain and the rest of the domains are training domains. CDGA and CDGA* reduce the diversity shift.
where 8( at) is the Dirac delta distribution for the k-th data point in S.. To unify this formulation with the loss on true distribution data we can rewrite Eq. 2 as
min Unif())( (n (x;
(4)
where Unif(n) the discrete uniform distribution to and the sample distribution & is defined as
[S.1
(x) 1
fa; of (kr.
55)
S
To improve the true dala estimation error of ERM, this is necessary to improve the sample distribution & To this end, the VRM principle suggests replacing the point-wise estimate in ERM i.e., 6(x; xf.)) in eq, 3 with some better kernel estimate of the density in the vicinity of the data point & within S; which we call K [[ x An example for K (xx;zf) can Gaussian kernel functions that act as smooth 8(x; x{) functions. In this case, the VRM sample distribution represented by & is written as
18
(x)
Sa Σ
K(x; (()
and the overall loss is obtained by replacing (xr in Equation by (x).. In practice, VRM is implemented by employing data augmentation along with ERM. VRM suggests that data augmentation ERM cstimation error by adding additional synthetic examples from the vicinity distribution around each observation within each domain in the data. We argue in the DG setting, if we only employ classic data augmentation the estimation error of VRM is still high. This is because, in the DG setting, the estimation error of truc data distribution by ERM is mainly caused by the distribution shift between domains which cannot be fully addressed by simple data augmentation techniques within each domain.
improve
Extending the VRM (Chapelle al. 2000) principle to the DG setup, possible intuitive justification on the success of CDGA is that employing CDGA replaces the pointwise estimates in ERM with new density estimates that can be in the vicinity of domain pairs that the distribution shift between domains is further reduced. We believe this step can potentially reduce the estimation error induced by ERM under the domain shift scenario. To see the difference between classic augmentation and CDGA more clearly, first, we need to define the projection operator:
Definition 6.1 (domain projection operator). Given a metric defined on supp(E.) U supp(E,), the domain projection operator P-) that projects data point x &, (i.e., I E supp(E,)) onto domain j is defined as
Pr)n(()) argmin d(z,x). supp(E,)

---

Under review as submission to TMLR
The projection operator gives the mathematical definition for what we mean by "finding the closest point another domain." For example, if x is a cat image of painting style (&.), then P=-)(æ) would mean the most similar image to æ in the sketch domain Ej. The projection operator relies on the metric d, which measures the similarity of two samples. This similarity can be implemented, c.g., by computing the Lg distance in some latent. feature embedding space.
Define the smoothed sample distribution of & projected to domain j by Eng:
(x) - S
K(t; Ras
(7)
where and j are domain counters and k is the data point index, K(x; Pj (arE) represents the vicinity of k-th data point in & projected onto & which is label invariant. Similar to VRM, the expression for the final loss function can be obtained by replacing (x) with E, (1)( in equation
In practice, thanks to image manipulation of diffusion models, such projection to other domains and sampling from the vicinity of projected data points has become feasible (approximately). To this end, in CDGA, we employ LDM, denoted by M(-) which lakes two arguments, one is a data point a domain and the second argument guidance attribute in another domain from the same class i.c,, Ty = =Mar guide') ~ P(-) where P()) defined P(.) (=))) Effectively, we are drawing an augmented sample near the
KG:P, (y(()))
projected sample in the target domain H and the kernel estimation relies on the LDM we choose. In this case the proposed proposed loss realized by CDGA along with ERM has the following form.
min Uait(a) Bn)-Uni((n) R (m.)) (x; 0).
(8)
Since our kernel relies on the LDM we use, the estimation error comparison between equation Sannd equation 5 seems infeasible. More theoretical exploration along this direction is left as future work.
7 Ablation Studies
7.1 Mitigating Class Imbalance
CDGA can also be utilized to mitigate the class imbalance problem in datasets where the number instances in each class of each domain not equal. In such scenarios, one can use different b for cach class of the data such that after generating samples, the number of instances in each class of generated domains becomes equal. We test the effectiveness of CDGA method in balancing the OfficeHome dataset (which is highly imbalanced) through the DomainBed benchmark. More specifically, for every class and domain Sj. we find the number of samples n(Sy.c) c) and then we find m = max.,j n(Sj.e) which is 100 for OfficeHome. Then for every domain Sj and class c we set = m(Sj.c fewer data points and subsequently balances the dataset. The results of this experiment are presented in Table 5 Clearly, by choosing b in a way that the dataset is more balanced, the OOD generalization has been further improved.
m which leads to larger batch size for domains and classes with
Table 5: OOD accuracy of models with and without balanced generation in OfficeHome dataset.
Training domain
Leave-one domain-out
Method
Oracle
ERM
66.5
65.7
66.4
ERM + CDGA (b = 1)
68.2 + 0.6
68.7 0.4
68.6 0.3
ERM CDGA (b .ट..
69.9 1 0.2
69.7 1 0.4
70.0 1 0.7
We implicitly assume that P1- is singleton for any Otherwise, can extend the definition E.g by averaging over all "clasest" projections.
10

---

Under review as submission to TMLR
7.2 Single Domain Generative Augmentation (SDGA) vs CDGA
To show the advantage of employing cross-domain data to mitigale domain shift, We also explore the SDGA method, where, unlike CDGA, the image from S; is augmented only from the guidance of the same domain i.e., M(guide'), where guidance can either be an image (SDGA-IG) or a prompt (SDGA-PG). In SDGA-PG, for each image in S; ie., z we create prompt guidance guide' that can contain label and/or domain information from Sj and feed guide' to the LDM. In SDGA-IG, for each image from Si, L.e,, oy we construct guidance that contains both z and label information. To compare variations of CDGA and SDGA, we evaluate them on the PACS dataset using the DomainBed benchmark with twenty different hyperparameters and one trial. The results of this experiment are presented in Table G Clearly, CDGA consistently outperforms all variations of SDGA. As some suggestions for use cases, we believe as long as the objective is maximum OOD performance, either textual or visual descriptions of different domains are accessible, and there is no computational bottleneck, the CDGA is a better choice compared with SDGA. On the other hand, if we do not have access to the domain descriptions, or we aim to achieve OOD improvement as fast as possible with fewer training examples, SDGA can be better option.
Table 6: OOD accuracy of models trained with variations of CDGA and SDGA on PACS dataset using Domainbed benchmark.
Training domain
Leave-one domain-out
Method
Oracle
ERM
85.5
83.0
86.7
SDGA-PG (label)
86. I +
83.7 +:
87.3
SDGA-PG (label domaii)
85.9 4: 1.0
84.6 &
87.5 1.1
+ SDGA-IG (label)
87.5 + 0.6
86.5 + 1.1
89.5 0.3
+ CDGA-PG (canny edge)
86.8 4: 0.9
79.2 4: 3.6
87.8 0.3
CDGA-PG
88.5 4: 0.5
86.8 +: 0.4
89.6 0.3
CDGA
89.5 + 0.3
88.4 0.5
90.4 + 0.3
7.3 Scaling Law of Data Size
In this experiment, we aim to measure the effect of generation batch size b which determines the final augmented dataset size on the OOD generalization of models trained with CDGA. To this end, on the PACS dataset, using DomainBed benchmark with 20 choices of hyperparameters and trial, we apply CDGA-PG with b equal 1, 2, 3, and 4. We conduct this experiment for two models: ResNet-18 model pretrained on ImageNet and ResNet-18 model with random initialization. As we from Table for both pretrained and random initializations, increasing b, i.c., increasing the data size, further improves OOD generalization. However, for the pretrained model, the performance gets saturated which can be due to the capacity of the model.
8 Does superior performance of the CDGA method comes from cross-domain transfer or the improvement is just because of extra synthetic images from pre-trained diffusion model
A critical question that may be raised whether the benefit of the CDGA method comes from cross-domain transfer and not from generating extra synthetic data from a pre-trained diffusion model. We do believe the superior OOD performance of CDGA due cross-domain transfer which removes the distribution shift between domains and not just because we are using model that is pro-trained and generates synthetic data. Here we provide our arguments to support this claim.
The reason behind beter OOD generalization is synthetic samples between domains (cross domain augmentation) and not just additional samples from pre-trained diffusion
11

---

Under review as submission to TMLR
model. By looking more closely at Table. 6 observe that although in both SDGA and CDGA techniques the same pre-trained diffusion model has been used, the OOD generalization improvement for CDGA is much higher than the improvement by SDGA. This observation suggests that the superior OOD generalization not just due to incorporating additional syntethic samples. Instead, this improvement is due employing additional syntethic samples that reduce the distribution shift across domains. The claim that CDGA actually roduces the distribution shift more than SDGA is experimentally supported in Figure where for example A - C' synthetic samples from CDGA are filling the gap between domains A and C while for SDGA, only A + A samples are generated which are not able to reduce the domain shift between domain A and C. More examples on this can be found in Figure 15 In other words, while both SDGA and CDGA employ the same pre-trained model, CDGA is able to generate samples between domains and reduce the distribution shift better than SDGA and as a result achieve better OOD generalization.
We are already empoying pre-trained ResNet for the domainbed experiments and fails to provide good OOD generalization. Consider the experimental results from the domainbed benchmark where a large imageNet pro-trained model (ResNet) used for all algorithms Tables 1-4 in our paper. As you can see, even with large pre-trained model, the model fails to achieve good OOD generalization while employing CDGA further boosts the performance.
These arguments suggests that merely employing pre-trained model, no matter this is for training data Iugmentation, does not necessarily improve the OOD generalization. In fact, the contributing factor is cross domain ability of CDGA which reduces the domain shift between domains.
Table 7: Effect of generation batch size on CDGA for PACS dataset with diffcrent initialization.
Training domain
Leave-one -omain-out
Initialization
b
Oracle
66.1
61.6
65.0
2
70.2
69.2
70.3
Random
3
71.8
71.8
74.5
73.9 4
73.2
74.6
1
87.0
86.4
89.0
2
87.4
86.2
89.0
Pre-trained
3
88.4
89.1
88.9
4
88.4
88.2
89.2
9 Conclusions
In this paper, we showed that simple cross domain generative augmentation (i.e., CDGA) alongside ERM surpasses SOTA DG algorithms the standard DomainBed benchmark. Additionally, empirical results show by employing various distribution shift quantification methods, we observe significant reduction distribution shift between training domains after applying CDGA. Furthermore, we conduct comprehensive ablation studies, particularly focusing adversarial robustness, loss landscape analysis, and data scaling laws. Notably, the use of CDGA enhances adversarial robustness and reduces the sharpness of the loss landscape, both contributing to improved model generalization. Finally, intuitively, we establish that CDGA with ERM approximately could be considered as an extension of the VRM principle to the DG setup. Our work provides a novel data-centric point of view for domain generalization, the cra when AI Generated Content. (AIGC) becomes more and more popular.
References
Martin Arjovsky, Léon Bottou, Ishaan Gulrajani, and David Lopez-Pazz Invariant risk minimization. arXiv preprind ar XrXii:1907.0288 2019.
12

---

Under review as submission to TMLR
Gilles Blanchard, Gyemin Lee, and Clayton Scott. Generalizing from several related classification tasks to new unlabeled sample. Advances in neural information processing systems, 24:2178-2183 2011.
Olivier Chapelle, Jason Weston, Léon Bottou, and Vladimir Vapnik. Vicinal risk minimization. Advances in neural information processing systems, 13, 2000.
Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive lcarning of visual representations. In International conference machine learning, pp. 1597-1607. PMLR, 2020.
Chen Fang, Ye Xu, and Daniel N Rockmore. Unbiased metric learning: On the utilization of multiple datasets and web images for softening bias. Tn Proceedings of the IEEF International Conference on Computer Vision, pp. 1657 1664, 2013.
Pierre Forct, Ariel Kleiner, Hossein Mobahi, and Behnam Neyshabur. Sharpncss-aware minimization for efficiently improving generalization. arXiv preprint arXie:201000141 2020.
Yaroslav Ganin, Evgeniya Ustinova, Hana Ajakan, Pascal Germain, Hugo Larochelle, François Laviolette, Mario Marchand, and Victor Lempitsky. Domain-adversarial training of neural networks. The journal of machine learning research, 17(1):2096-2) 202220000000,,,, 2016.
Tan Goodfellow, Jonathon Shlens, and Christian Szegody. Explaining and harnessing adversarial examples. In Yoshua Bengio and Yann LeCun (eds.), 3rd International Conference on Learning Representations, ICLR 2015, Son Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings, 2015. URL http:
//arxiv. wwtp://///abb....eeeeo
Ishaan Gulrajani and David Lopez-Paz. In search of lost domain generalization. arXio preprint arXiv:2007.01433 2020.
Sobhan Hemati, Guojun Zhang, Amir Estiri, and Xi Chen. Understanding hessian alignment for domain generalization. Proceedings of the IEEE/CVP International Conference Computer Vision (ICCV), 2023.
Jonathan Ho, Ajay Jain, and Picter Abbecl. Denoising diffusion probabilistic models. Advonces in newral information processing systems, 33:6840 6851, 2020.
Weihua Hu, Gang Niu, Issei Sato, and Masashi Sugiyama. Does distributionally robust supervised learning give robust classifiers? In International Conference on Machine Learning, pp. 2029-2037. PMLR, 2018.
Maximilian llse, Jakub M Tomezak, and Patrick Forré. Selecting data augmentation for simulating interven- tions. In International Conference on Machine Learning, pp. 4555-4562. PMLR, 2021.
Nitish Shirish Keskar, Dheevatsa Mudigere, Jorge Nocedal, Mikhail Smelyanskiy, and Ping Tak Peter Tang. On large-batcl training for deep learning: Generalization gap and sharp minima. arXiv preprint 1rXiu:1609, 2016.
Masanori Koyama and Shoichiro Yamaguchi. When is invariance useful in an aat-of-distribution generalization problem? arXiv preprint arXix:2008.0 01888 2020.
David Krueger, Ethan Caballero, Joern-Henree Jacobsen, Amy Zhang, Jonathan Binas, Dinghuai Zhang, Remi Le Priol, and Aaron Courville. Out-of-distributio generalization via risk extrapolation (rex). In International Conference on Machine Learning, pp. 5815 5826. PMLR, 2021.
Da Li, Yongxin Yang. Yi-Zhe Song, and Timothy M Hospedales. Deeper, broader and artier domain generalization. In Proceedings of the IEEE international conference on computer vision, pp. 3542-5550 2017.
Haoliang Li, Sinno Jialin Pan, Shiqi Wang, and Alex C Kot. Domain generalization with adversarial feature learning. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 5400-5409, 2018.
13

---

Under review as submission to TMLR
Aleksander Madry, Aleksandar Makelov, Ludwig Schmidt, Dimitris Tsipras, and Adrian Vladu. Towards deep learning models resistant to adversarial attacks. In 6th International Conference on Learning Representations, ICLR 2018, Vancouver, BC, Canada, April 30 May 3, 2018, Conference Track Proceedings. OpenReview.net, net, 2018. URL httpps://openreviet net/fforumiid-x32lBBB..
Maxime Oquab, Timothée Darcet, Théo Moutakanni, Huy Vo, Mare Szalraniec, Vasil Khalidov, Pierre Fernandez, Daniel Haziza, Francisco Massa, Alaaeldin El-Nouby, et al. Dinov2: Learning robust visual features without supervision. arXiv preprint or orXiv:2304.071 071193, 2023.
Giambattista Parascandolo, Alexander Neitz, Antonio Orvieto, Luigi Gresele, and Bernhard Schölkopf. Learning explanations that are hard vary. arXiv preprint arXiv:2009.008: 000000 2020.
Xingchao Peng, Qinxun Bai, Xide Xia, Zijun Huang, Kate Saenko, and Bo Wang. Moment matching for multi-source domain adaptation. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 1406-1415, 2019.
Justin Pinkney. Image mixer.
2023.
Alec Radford, Jong Wook Kim, Chris Hallacy, Adilya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pp. 8748 8763. PMLR, 2021.
Alexandre Rame, Corentin Dancette, and Matthieu Cord. Fishr: Invariant gradient variances for out-of- distribution generalization. In International Conference on Machine Learning, pp. 18347- PMLR, 2022.
Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. Hierarchical text-conditional image generation with clip latents. arXiv preprint arXiv:2204.661 2022.
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. High-resolttion image synthesis with latent diffusson models. In Proceedings of the IEEE/CVF conference on computer vision and pattern recogmition, pp. 10684-10695, 2022.
Shiori Sagawa, Pang Wei Koh, Tatsunori Hashimoto, and Percy Liang. Distributionally robust neural networks for group shifts: On the importance of regularization for worst-case generalization. In International Conference on Learning Representations, 2020.
Ozan Sener and Vladlen Koltun. Domain generalization without excess empirical risk. Advances in Neural Information Processing Systems, 35:13380-13391, 2022.
Soroosh Shahtalebi, Jean- Cean-Christophe Gagnon-Audel, Touraj Laleh, Mojtaba Faramarzi, Kartik Ahuja, and Irina Rish. mand-mask: An enhanced gradient masking strategy for the discovery of invariances in domain generalization. aix preprint ar Xix::1100 02266, 2021.
Yuge Shi, Jeffrey Seely, Philip HS Torr, N Siddharth, Awni Hanmun, Nicolas Usunier, and Gabriel Synnaeve. Gradient matching for domain generalization. arXiv preprint arXiv:2104.09977 2021.
Connor Shorten and Taghi M. Khoshgoftaar. A survey image data augmentation for deep learning. Journal of Big Dala 6(1), July 2019. doi: 10. 10.1186/540037--099-0177 URL http:ttt///.ooogg//////// 10866/
840537-019-0197-0
Nathan Somavarapu, Chih-Yao Ma, and 7solt Kira. Frustratingly simple domain generalization via image stylization. arviv preprint arvv::2066. 11207, 2020.
Jiaming Song, Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models. ar preprint arXiv:2010.0225 2020.
Baochen Sun and Kate Saenko. Deep CORAL: Correlation alignment for deep domain adaptation. In European conference on computer vision, pp. 443 -400. Springer, 2016.
14

---

Under review as submission to TMLR
Yonglong Tian, Lijie Fan, Phillip Isola, Huiwen Chang, and Dilip Krishnan. Stablerep: Synthetic images from text-to-image models make strong visual representation learners. arXiv preprint arXiv:2306.000 000.6. 2023.
Eric Tzeng, Judy Hoffman, Ning Zhang, Kate Saenko, and Trevor Darrell. Deep domain confusion: Maximizing for domain invariance. arXic preprint or Xix: 2014.
Vladimir Vapnik. The nature of slatistical learning theory. Springer science & business media, 1999a.
Vladimir N Vapnik. An overview of statistical learning theory. IEEE transoctions neural networks, 10(5): 988-999, 1999b.
Hemanth Venkateswara, Jose Eusebio, Shayok Chakraborty, and Sethuraman Panchanathan, Deep hashing network for unsupervised domain adaptation. In Proceedings of the IEEE conference on computer vision and pottern recognition, pp. 5018-5021 2017.
Nanyang Ye, Kaican Li, Haoyue Bai, Rumpeng Yu, Lanqing Hong, Fengwei Zhou, Zhenguo Li, and Jun Zhu. Ood-bench: Quantifying and understanding two dimensions of out-of-distribution generalization, Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 7947-7958, 2022.
Guojun Zhang, Han Zhao, Yaoliang Yu, and Pascal Poupart. Quantifying and improving transferability in domain generalization. Advances in Neural Information Processing Systems, 34:10957 10970, 2021.
Hongyi Zhang, Moustapha Cisse, Yamn N Dauphin, and David Lopez-Paz. mixup: Beyond empirical risk minimization. orXiv preprinl arXiv:1710.09412, 2017.
Lvmin Zhang and Mancesh Agrawala. Adding conditional control to text-to-image diffusion models. arXiv preprint Xr1:::330..0544, 2023.
Kaiyang Zhou, Yongxin Yang, Yu Qiao, and Tao Xiang. Domain generalization with mixstyle. arXiv preprinl arXiv:2104.02008 2021.
15


Reviews:
[{"review_id": "rWkIk4iry4", "reviewer": "Reviewer_Azv7", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strengths:**\n- The proposed cross-domain data augmentation (CDGA) for dataset augmentation is a novel and interesting approach. The domain shift analysis of CDGA is particularly insightful, as it effectively demonstrates the advantages of the algorithm over conventional ERM methods.\n- The evaluation is thorough, utilizing four different benchmark datasets and comprehensive comparisons with state-of-the-art (SOTA) algorithms.\n- The ablation study involving single-domain generative augmentation (SDGA) offers valuable insights and opens up interesting avenues for future research. These results, in particular, could benefit from further theoretical exploration.\n- Overall, the paper is well-written and clear in its presentation.\n\n**Weakness:**\n- While the main contributions of the paper are well-justified, the inclusion of the loss landscape analysis and adversarial robustness study feels somewhat tangential to the core findings. These sections could be relocated to the appendix for better focus.\n- A deeper analysis of the ablation studies would have been more beneficial, offering additional insights into the functioning of the proposed augmentation technique.\n- Finally, incorporating statistical significance tests would further strengthen the results. For example, in the ablation studies, was the performance difference between CDGA and SDGA statistically significant? Similarly, in the DomainBed benchmarks, did ERM + CDGA significantly outperform the MixUp method? Statistical tests would provide clearer evidence of the method's advantages.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper introduces a data-centric approach that integrates cross-domain data augmentation (CDGA) with the empirical risk minimization (ERM) algorithm, achieving improved state-of-the-art (SOTA) performance in terms of accuracy on both training and test datasets. CDGA leverages latent diffusion models to perform transformations that combine an input from one domain with a guidance attribute from a different domain, but within the same class. The guidance attribute can either be a prompt or an image, with the two variants including prompt-based guidance (CDGA-PG), which utilizes class labels or descriptions, and image-based guidance (CDGA-IG).\n\nThe authors demonstrate several key properties of the CDGA + ERM framework: (1) it generalizes the vicinal risk minimization (VRM) principle to the domain generalization setting, (2) t-SNE analysis shows that it effectively reduces domain shift, (3) it promotes the formation of flatter local minima during training, and (4) it enhances adversarial robustness. The training and test data are sourced from DomainBed, including datasets such as PACS, Office-Home, DomainNet, and VLCS. The CDGA approach consistently outperformed baseline methods, particularly outperforming the standard ERM algorithm.\n\nAdditionally, ablation studies were conducted to assess the performance of CDGA against single-domain generative augmentation (SDGA) and with varying data sizes. The results indicate that CDGA shows significant potential compared to its SDGA counterparts.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "It would be beneficial to move some of the results currently in the appendix to the main paper, while shifting the sections on loss function landscape analysis and adversarial robustness to the appendix. For example, specific examples of the prompts or images used as guidance attributes could be included in the main paper to enhance clarity. As mentioned in the weaknesses, adding a section on the statistical significance of the performance differences between the algorithms would also be valuable. Additionally, the section on mitigating class imbalance could be moved to the main paper. Lastly, offering deeper insights into the comparison between SDGA and CDGA would be helpful, as the results suggest that SDGA performs only slightly worse than CDGA. Highlighting scenarios in which CDGA is preferable over SDGA would provide useful context for readers.", "broader_impact_concerns": "The work in its current form does not exhibit any broader impact concerns.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "uJGD6O4SPc", "reviewer": "Reviewer_xJm6", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "I am not very familiar with the fields of domain generalization and diffusions models, so my review should definitely be taken with a grain of salt when it comes to these aspects. \n\n**Strengths**:\n1. The paper is clearly structured and mostly well-written. \n2. The idea of using diffusion models to generate cross-domain images to address domain generalization is novel, unless there are any prior works that I have overlooked. \n\n**Weakness**:\n1. There are some concerns about the comparison between the proposed method and SOTA methods. See requested changes. \n2. The theoretical analysis contains many parts that are ambiguous / not rigorous. I would suggest authors make the statements about connection to VRM more rigorous and scale back claims where necessary. For further details, please see requested changes 3~6.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper employs diffusion models to address the problem of domain generalization. In the proposed CDGA method, cross-domain synthetic images are generated using diffusion models and serve as additional training data to vanilla ERM. This method is formulated as a type of cross-domain VRM (Vicinal Risk Minimization) in the analysis. In the experiments, authors show the CDGA variants outperform several SOTA domain generalization algorithms on the benchmarking platform DomainBed. Authors also show through several measurements that the cross-domain images reduce the differences between domains.", "claims_and_evidence_yn": "No", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "**Critical**:\n1. In the experiment results on domain generalization performance (Table 1~4), the best-performing method is often CDGA*, which assumes access to guidance attributes from the target domain. Do other baseline (SOTA DG) methods considered in these results also have access to information from target domain? My main concern is whether the comparison is fair in terms of access to the target domain. \n2. How does the number of training samples in the proposed CDGA method compare to that of the other SOTA methods? \n3. What is the distance measurement $d(\\cdot, \\cdot)$ in the domain projection operator when instantiated with a latent diffusion model? I did not find further details of this in section 5. \n4. What are the assumptions needed for the kernel function, in theoretical claims about VRM? For example, does the kernel need to satisfy positivity, and/ or certain smoothness assumption, and/or translation-invariance? I didn’t see these conditions specified. \n5. Closely following point 4, I am confused about the kernel induced from CDGA. Without any theoretical evidence that CDGA in fact induces a kernel K which satisfies the assumptions needed in VRM (as specified in the expression above equation (8) on page 7), I don’t think that the claims of theoretical equivalency between CDGA and ERM(VRM) with a different kernel is well-supported. Due to the concerns with this part, I am leaving the answer to \"Claims And Evidence\" as \"No\" for now. \n6. In the expression above equation (8) on page 7, it states that $x$ is drawn from $K (\\cdot; P_{i\\rightarrow j}(x_k^i))$. However, $K (\\cdot; P_{i\\rightarrow j}(x_k^i))$ is a function, not a data distribution. If the authors meant to define a probability distribution from the kernel function (e.g. by normalizing), they should make it mathematically correct. \n7. In Section 8, regarding the results on adversarial robustness, I am interested to know how the baseline methods (considered in Table 1~4) perform in terms of adversarial robustness.   \n\n**Nice to have**:\n1. Why is it that for other datasets than VLCS, the guidance attributes are text prompts, but for VLCS the attributes are images?", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "SGQFAClzHT", "reviewer": "Reviewer_tdhq", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths: \n- The proposed approach is effective. There are significant improvements on the DomainBed benchmark against other state-of-art methods for cross-domain generalization. \n\n- The authors attempt to explain the effectiveness of their method by the vicinal risk minimization principle by Chapelle et al 2000. \n\n- There is a fairly comprehensive set of experiments to measure how the CDGA method reduces domain shift across datasets. \n\n- The authors also provide how the models trained with CDGA reduces loss landscape sharpness and improves adversarial robustness \n\n\nWeaknesses: \n- I have reservations with the use of strong models like Stable Diffusion trained on a lot of image data for data transformation across domains. The main problem is we cannot tell whether the benefit comes from generalizing from cross-domain data, or just by having a strong diffusion model that can generate extra examples for learning in the target domain. For example in Figure 5, the generated dog images do not look like the same type of dogs as the images used as guides. It is difficult to interpret the model as performing cartoon -> cartoon or cartoon -> sketch transformation as the source and target have little in common apart from being images of dogs. A control experiment can be done by generating images in the target domain (sketch, cartoon) by providing the word prompt 'dog' only and see how the accuracy improves by training on these generated data, without using any guides from other domains. In this way we can more accurately measure the effect of the source of improvement. \n\n- Some of the information presented in the appendix should be included in the main paper. For example, the first page of Appendix A includes important information about the method and should be included in the main paper. Also, Figure 10 comparing CDGA and SDGA is very informative and should be included too. \n\n- Section 3 is not clearly written. Shouldn't the number of examples for domain S_i depend on the number of examples of other domain S_j as the extra examples are translated from other domains? To clarify the authors can just write down the ERM equation like Equation 2. Also the definition of CDGA* is unclear. Is it equivalent to CDGA-PG or CDGA-IG?", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The authors propose a method for learning good cross-domain classifiers by using diffusion models to adapt data across domains. With the use of strong diffusion models like Stable Diffusion they are able to achieve state-of-art results on benchmarks like DomainBed. They also attempt to explain why their method works with the vicinal risk minimization principle and provide detailed empirical analysis on their method reduces domain shifts.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "- Improve the clarity by rewriting Section 3 and moving some of the important information from the Appendix to the main paper \n\n- Explain how we can ensure the benefit of the CDGA method comes from cross-domain transfer and not from generating extra synthetic data from trained diffusion models like Stable Diffusion.", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The recommendation is based on the reviewers' comments, the action editor's evaluation, and the authors’ response. 

This paper studies the use of pretrained diffusion models as a data augmentation generator to facilitate domain generalization. All reviewers find the studied setting novel and the results provide new insights. The authors’ rebuttal has successfully addressed the major concerns of reviewers. Therefore, I recommend acceptance of this submission. 

However, the authors are suggested to add more clarity on explaining the source of performance gain when using stable diffusion models, given that these models are trained on large-scale image data, and it is not clear whether the considered tasks are considered "out-of-domain" relative to the pertaining data. Arguing ResNet pretrained on ImageNet fail in domain transfer does not fully justify this point, because the pertaining data of stable diffusion are much more diverse and more in volume compared to ImageNet.

I also expect the authors to include the new results and suggested changes during the rebuttal phase to the final version.
Decision: Accept (poster)