Unknown Domain Inconsistency Minimization for Domain Generalization
Authors: Seungjae Shin
HeeSun Bae
Byeonghu Na
Yoon-Yeong Kim
Il-chul Moon
Venue: ICLR 2024 poster (2024)
Keywords: Robustness
Domain generalization
Sharpness-Aware Minimization
Loss Sharpness
Inconsistency
Primary area: transfer learning, meta learning, and lifelong learning
TL;DR: This paper introduces an objective rooted in both parameter and data perturbed regions for domain generalization, termed Unknown Domain Inconsistency Minimization (UDIM).

Abstract: The objective of domain generalization (DG) is to enhance the transferability of the model learned from a source domain to unobserved domains. To prevent overfitting to a specific domain, Sharpness-Aware Minimization (SAM) reduces source domain’s loss sharpness. Although SAM variants have delivered significant improvements in DG, we highlight that there’s still potential for improvement in generalizing to unknown domains through the exploration on data space. This paper introduces an objective rooted in both parameter and data perturbed regions for domain generalization, coined Unknown Domain Inconsistency Minimization (UDIM). UDIM reduces the loss landscape inconsistency between source domain and unknown domains. As unknown domains are inaccessible, these domains are empirically crafted by perturbing instances from the source domain dataset. In particular, by aligning the loss landscape acquired in the source domain to the loss landscape of perturbed domains, we expect to achieve generalization grounded on these flat minima for the unknown domains. Theoretically, we validate that merging SAM optimization with the UDIM objective establishes an upper bound for the true objective of the DG task. In an empirical aspect, UDIM consistently outperforms SAM variants across multiple DG benchmark datasets. Notably, UDIM shows statistically significant improvements in scenarios with more restrictive domain information, underscoring UDIM’s generalization capability in unseen domains.

Published as a conference paper at ICLR 2024
UNKNOWN DOMAIN INCONSISTENCY MINIMIZATION FOR DOMAIN GENERALIZATION
Seungjae Shin"!, HeeSun Bae"!, Byeonghu Na', Yoon-Yeong Kim² & II-Chul Moon 'Department of Industrial and Systems Engineering. KAIST *Department of Statistics, University of Scoul, 'summary.ai (tmdwo09 091 910, cat2507, xp03052, cmmon) @kai Staaass...... ac.aar.kk kr, yykimeuos.acm kr
ABSTRACT
The objective of domain generalization (DG) is to enhance the transferability the model learned from source domain unobserved domains. To prevent over- fitting to a specific domain, Sharpness-AWee Aware Minimization (SAM) reduces source domain's loss sharpness. Although SAM variants have delivered significant im- provements in DG, we highlight that there's still potential for improvement in generalizing to unknown domains through the exploration on data space. This pa- per introduces an objective rooted in both parameter and data perturbed regions for domain generalization, coined Unknown Domain Inconsistency Minimization (UDIM). UDIM reduces the loss landscape inconsistency between source domain and unknown domains. As unknown domains are inaccessible, these domains are empirically crafted by perturbing instances from the source domain dataset. In particular, by aligning the loss landscape acquired in the source domain to the loss landscape of perturbed domains, these flat minima for the unknown domains. Theoretically, we validate that merg- ing SAM optimization with the UDIM objective establishes an upper bound for the true objective of the DG task. In an empirical aspect, UDIM consistently out- performs SAM variants across multiple DG benchmark datasets. Notably, UDIM shows statistically significant improvements in scenarios with more restrictive do- main information, underscoring UDIM's generalization capability in unseen do- mains. Our code is available at https: //tt:://///tbb/t////// com Sww.m//SShin-AIOIT
expect to achieve generalization grounded on
INTRODUCTION
Domain Generalization (DG) Zhou et al. 2022: Wang et al, 2022) focuses on domain shift that arises when training and testing occur across distinct domains, I.e. domain of real pictures in train- ing, and a separate domain of cartoon images testing. The objective of DG is to train model on given source domain dataset, and generalizes well on other unobserved domains. To address the domain discrepancy between the source domain and other domains, various methods have been proposed: 1) alignment- methods (Li ct al. 202 lat Wald et al. 2021); 2) sugmentation-basee methods (Qiao et al.) 2020a] Zhou et al. 2021 ) and 3) regularization based methods (Arjovsky) al., Kruggrr al. 2222 Rame et al. 2022). While these methodologies have demonstrated promising results, they often underperform in settings where given domain information is particu- larly limited (Wang et al. (202 lb Qiao et al. (2020b). Also, most methods lack theoretical guarantees on the minimization of the target risk at the distribution level.
In contrast to the aforementioned methods, Sharpness-aware optimizers, which flatten the loss land- scape over perturbed parameter region, demonstrate promising performances in DG tasks (Zhang al. 2023b; Wang al., 2023). By optimizing the perturbed local parameter regions, these ap- proaches relieve the model overfitting to specific domain, thereby enhancing the adaptability of the model across various domains. Also, this concept has a solid theoretical foundation based on the parameter space analysis with PAC-Bayes theories (McAllester 1999 Dziugaite & Roy 2017).
While the perturbation methods based on the parameter space have shown promising improvements in DG tasks, this paper theoretically claims that perturbation rooted in the data space is essential for
"Equal contribution

---

Published as a conference paper at ICLR 2024
robust generalization to unobserved domains. Accordingly, this paper introduces objective that leverages both parameter and data perturbed regions for domain generalization. In implementation, our objective minimizes the loss landscape discrepancy between source domain and unknown domains, where unknown domains are emulated by perturbing instances from the source domain datasets. Recognizing the loss landscape discrepancy domains, we name our objective as Unknown Domain Inconsistency Minimization (UDIM).
Inconsistency score across different
Introduction of UDIM on the DG framework has two contributions. First, we theoretically prove that the integration of sharpness-aware optimization and UDIM objective becomes the upper bound of population risk for all feasible domains, without introducing unoptimizable terms. Second, we reformulate the UDIM objective into a practically implementable term. This is accomplished from deriving the worst-case perturbations for both parameter space and data space, each in closed-form expression. Our experiments on various DG benchmark datasets illustrate that UDIM consistently improves the generalization ability of parameter-region based methods. Moreover, we found that these improvements become more significant as domain information becomes more limited.
2 PRELIMINARY
2.1 PROBLEM DEFINITION OF DOMAIN GENERALIZATION
This paper investigates the task of domain generalization in the context of multi-class classification Arjovsky et al. 2019 Sagawa et al. 2019 Nam et al. 2021). We define an input as E R and its associated class label as y E [I,.. C). Let Le represent distribution
e-th domain. & is a set of indices for all domains, and 2 = (eeheE denotes the set of distributions for all domains, where every domain shares the same class set. For instance, let's hypothesize that video streams from autonomous cars are being collected, and the data collection at days and nights will constitute two distinct domains. A sampled dataset from the e-th domain is denoted by De = {(x,, y Th where ( y:) D. and ne is the number of data instances of e-th domain.
Throughout this paper, let A E θ represents a parameter of trained model fo. where θ is a set of model parameters. Using De. we define loss function as LD, (0) = mhe E(.. ((fo(zz)) y:), where we sometimes denote ((fo( y:) as ((r,, The population risk for domain e is given by Cg. (8) (a [e(fo(=), [(,.))=,_(((f((r)), v)]- Then, the population risk over all domains is defined as (0) = E.EE C2,) (0), where p(e) represents the occurrence probability of domain e.
In essence, the primary goal of training a model, fo, is to minimize the population risk, ( In practical scenarios, we only have access to datasets derived from a subset of all domains. We call these accessible domains and datasets as source domains and source domain datasets, respectively; denoted as 25 = {2.).es and Ds {Da)ses where S is the set of indexes for source domains. As 2s * 2, Ds deviates from the distribution D under the sampling bias of Ds. As a conse- quence, a model parameter 03 argmin, (0), which is trained exclusively on Ds. might not be optimal for Lg((), Accordingly, domain generalization emerges as pivotal task to optimize lr argmin (f) by only utilizing the source domain dataset, Ds--
2.2 VARIANTS OF SHARPNESS- AWARE MINIMIZATION FOR DOMAIN GENERALIZATION
Recently, new research area has emerged by considering optimization over the parameter space (Foret et al. 2020: K Kwnn et al. 2021). Several studies have focused on the problem of I overfitted to training dataset (Wang et al. 2023 Zhang et al., 2023b a). These studies confirmed that opti- mization on the perturbed parameter region improves the generalization performance of the model. To construct a model that can adapt to unknown domains, it imperative that an optimized param- eter point is not overfitted to the source domain datasets. Accordingly, there were some studies to utilize the parameter perturbation in order to avoid such overfitting, as elaborated below (Table
Among variations in Table D Sharpness- Aware Minimization (SAM) (Foret et al. 2020 is the most basic form of the parameter perturbation, which regularizes the local region of 8 to be the flat minima on the loss curvature as ming max Sl LD, (0 + <) + ||0|2- Here, is the perturbation vector to 6; and p is the maximum size of the perturbation vector. Subsequently. methodologies for regularizing stronger sharpness were introduced (Zhang et al. 2023b Wang al. 2023 Zhang al. 2023a
2

---

Published as a conference paper at ICLR 2024
These approaches exhibited incremental improvements in domain generalization tasks. Check Ap- pendix A for more explanations.
Table I: Objectives of SAM variants for DG
We motivated by this char- acteristic to cxtend the parameter perturbation towards data pertur- bation, which could be effective for the exploration of unknown domains. None of SAM variants proposed data-based perturbation for unknown domain discovery. Fundamentally, SAM variants predominantly focus on identify- ing the flat minima for the given souree domain dataset, Ds. In Section 3.1 we highlight that finding flat minima in the source domain cannot theoretically ensure generalization to unknown tar- get domains. Consequently, we demonstrate that generalization should be obtained from unobserved domains, rather than solely the source domain. Since SAM imposes the perturbation radius of only 0, we hypothesize D2 could be perturbed by additional mechanism to generalize D2 toward Φ.
Method
Objective
SAM
Furet al.
2020
Cp, +.+)
Thang
2023b
Co. (9)+
Hise
SAKI
Wang et al.
2023
(n, + Ca, 1 WVCn (B)/ IV2n, (0) aVCn, (9))
FAM
Zhang al.
2023a
(p.
(Cp, Co. (0)
I[VC, (0 +cl
Hass
While SAM minimizes the loss over p-ball region of 0, its actual implementation minimizes the maximally perturbed loss w.r.t 0 + e"; where the maximal perturbation, E", is approximated in a closed-fomm solution via Taylor expansion as follows:
VoLD, 00)
max (p,(0+e) ~ LD, + where e" ss ps signnnnn VoLn, (0) |<
(I)
The existence of this closed-fore form solution of c* simplifies the learning procedure of SAM by avoid- ing min-max game and its subsequent problems, such as oscillation of parameters (Chu al. 2019).
This closed-form solution can also be applied to the perturbation on the data space.
METHOD
3.1 MOTIVATION: BEYOND THE SOURCE DOMAIN-BASE FLATNESS
Based on the context of domain generalization, Theorem 3.1 derives the relationship between the SAM loss on the source domain dataset, denoted as max] CD, (0 + c). and the generalization loss on an arbitrary unknown domain, Lg (0), for model parameter E as follows:
Theorem 3.1. (Rangwani al.) (2022) For 0 E O and arbitrary domain Pe D, with probability at least 1 - 8 over realized dataset D2 from D, with [D,] = n,, the following holds under some technical conditions on L2 (0), where hg R.+ R4 is strictly increasing function.
hol |0013
L9. (0) max LD, (0 (0++)++ ((2 +D 1elllsS
Theorem 3.1 provides upper bound for Lg. (#). The second term of this upper bound, represented as Di 2, Te). corresponds to the distribution discrepancy between Dg and P.. Notably, D' denotes the discrepancy based f-divergence. When e * D. is inaccessible in the context of domain generalization. Accordingly, the SAM optimization on D, leaves an unoptimizable term in its upper bound, posing challenges for the domain generalization.
In a setting that only D. and model parameter 0 are accessible, generating unseen domain data be- comes infeasible. Nontheless, by perturbing D, towards the direction that is most sensitive given 8, we can emulate the worst-case scenario for an unobserved domain (Sagawa et al. 2019). While parameters trained via the SAM optimizer may exhibit flat regions based on the source domain dataset, D., there is no theoretic study on the flat minima under unobserved domains. By identi- fying the worst-case scenario that maximizes the loss landscape difference between domains, our methodology seeks the generalization across the unknown domains.
3.2 UDIM UNKNOWN DOMAIN INCONSISTENCY MINIMIZATION
This section proposes an objective based on both parameter and data perturbed regions for domain generalization, coined Unknown Domain Inconsistency Minimization (UDIM). UDIM minimizes
3

---

Published as a conference paper at ICLR 2024
AP
Lose mille
Unitave
12 Tasta spoce
Paruretee apoc
Parmeder quane
Pantretee spoch
Toca sgoce
(a) Changes in the loss landscape across domains based on the (b) Changes in domain wiie inconsistency perturbed parameter space initial state (left), post-SAM op- sharpness based on data space of before timization (center), and subsequent UDIM application (right). (left) and after (right) applying UDIM.
Figure 1: Illustration of our model, UDIM, based parameter space (a) and data space (b). (a) We define flatness within a perturbed region by minimizing the inconsistency loss relative to the unknown domains, around the flat region derived from the source domain. (b) Furthermore, by reducing the domain-wise inconsistency within the input perturbed regions, where Pa denotes per- turbation length, our method can also be interpreted an data space perspective of SAM.
the loss landscape discrepancy between the source domain and unknown domains, where unknown domains are empirically realized by perturbing instances from the source domain dataset, D..
Let θe {e I1e 0ll I p}. which is p-ball perturbed region of a specific parameter point When training parameter 0 on an arbitrary domain dataset D, with the SAM optimizer, some regions within Oo., are expected to be optimized as flat regions for source domain. Following the notation of Parascandolo et al. (020b),, define No.0 {0' Og, [D_ LD. (D.( S 7).
which the region Gg.p where the loss value of 0 deviates by more than y. Given small enough values of LD. (0) and y, No could be recognized as a flat minima for e-th domain.
We aim to utilize the flat minima of obtained through training on D3 using the SAM optimizer, where represent s-th domain our source domain. The goal is regularize the loss and its corresponding landscape of unknown domains, so that the flat minima of the unknown domain aligns with that of the source domain. By optimizing the domain of worst-case deviation in the loss landscape from D., we facilitate regularization across a range of intermediary domains. Eq. formalizes our motivation, which is cross-domain inconsistency score:
(0)
= max max CEÊ EN (D) LD.
(3)
In the above equation, the inner maximization seeks the worst-case parameter point 0' that amplifies the nomain-wise loss disparity, while the outer maximization identifies e-th domain that maximizes maxg' ENTU [D, - Lp. )) Parascandolo et al. (2020a) utilizes an equation similar to Eq. 3 and this equation is also employed to indicate with a loss surface that is invariant to the en- vironment changes. Our methodology differs from Parascandolo et al. (2020a), which simply uses inconsistency metric, in that we employ it directly the objective aim optimize. Let's assume that 9 exhibits sufficiently low loss along with flat loss landscape of D.. Further, if we can identify 0 with low value of I (0), then this 0 would also demonstrate consistently good gener- alization for unknown domains. This motivation leads to the UDIM's objective, which specifically targets the reduction of the cross-domain inconsistency score across unobserved domains.
Objective Formulation of UDIM While we define the cross-domain inconsistency as IZ (0). the final formulation of UDIM the optimization of & regarding both source and unknown domain losses from the flat-minima perspective. Eq. is the parameter optimization of our proposal:
min max LD, (0 + c) A1 max max Se
(D, LD, (6' >2110112)
(4)
EEE 8' ENN
The objective of UDIM consists of three components. The first term is a SAM loss on Ds, guiding toward flat minima in the context of D.. Concurrently, as 0 provides flattened local landscape, the second term weighted with >₁ is a region-based loss disparity between the worst-case domain e and the source domain. Subsequently, the algorithm seeks to diminish the region- based loss disparity between the worst-case domain and the source domain. Figure (a) illustrates the update of the loss landscape in the parameter-space for each domain, based on the optimization of Eq. As SAM
4

---

Published as a conference paper at ICLR 2024
optimization on D2 progresses, N7, is expected to be broadened. However, SAM optimization does not imply the minimization of I (0) (center). Given that the optimization of Iz(() is conducted on Nggh we expect the formation of flat minima spanning all domains in the optimal state (rightmost).
The optimization of Eq. can also be interpreted as minimizing another form of sharpness in the data space. Let's suppose all other domains lie within a finite perturbation of the source domain's dataset. In the context of the data space the D,, the optimiza- tion can be seen as identifying the worst-case perturbed dataset, De. by evaluating maxg'e [C, LD, (6) )I- If we view the perturbation of the D.. not discrete choices but a con- tinuum within the data-space; our optimization can be viewed as minimizing the sharpness of domain-win inconsistency the data space. Figure [ (b) illustrates this interpretation. After such opti- mization, the resulting parameter can offer the consistent general- ization over domains located within the perturbed data space.
1011
5431
SNDV
GA3
1000
2001
2003
1300
Itematioon
Figure 2: Inconsistency score of each method on PACS training dataset (X-axis: training iteration). Y-axis is depicted in log-scale.
At this juncture, crucial consideration is whether the SAM opti- mization on D3 focuses on minimizing the second term, (0), not. We illustrate the limited capability of SAM variants in minimizing the second term, by an ana- lytical illustration of Figure (a); and by an empirical demonstration of Figure Therefore, UDIM covers the optimization area that SAM does not operate.
Theoretical Analysis of UDIM Given the definition Op - (0' 1110" - 0112 S ph. we introduce
Og.p' = arg maxe. C D. which the largest p'-ball region around 0 in N
2,9 Theorem 3.2
introduces the generalization bound of Eq. 4 which is the objective of UDIM. Theorem 3.2 states that Eq. 4 can become the upper bound of Lg (0), which is the population risk over all domains.
Theorem 3.2. For A E θ and arbitrary domain e E E, with probability at least - 5 over realized dataset D. from De, the following holds under technical conditions on Lg. (0) and LD. (0), where h: 12, + R2 is strictly increasing function. (Proof in Appendix B.1 B.I
1013 p²
(0) max LD. (0 (0+ |
max max DCD (8') CD. (0') )1 + h EEE 8'ENTI0
(5)
[
In such a case, Theorem 3.2 retains the same form of weight decay term as presented in Theorem 3.1 Unlike Theorem 3.1 which contains terms that are inherently unoptimizable, objective is capable of minimizing every term encompassed in the upper bound of Theorem 3.2 because we do not have an inaccessible term, (2.))....
3.3
IMPLEMENTATION OF UDIM
This section reformulates the second term of Eq. into an implementable form. We first formalize the perturbation of D2 to emulate the worst-case domain for max,' ENT [CD Cp, )
Inconsistency-Aweer Domain Perturbation D. For clarification, explain the perturbation process based on an arbitrary input instance from D.. Given that the magnitude of the perturbation vector for input x is constrained to Pa. the perturbed input ā can be expressed follows:
i = r
max llp d'ENT.
argmax
(1(x ((1+ 00) E(x, = x
argmax
max {(1 + (c,
<xille>lla5p d' d'EN"
(6)
I argmax (1(x + c,, & 61)) <x:1l==lla5p=
(7)
1st Taylor
Assuming N 5 as flat minima of 0, Exx is almost invariant for 8' E N..P. We assume that the 3.8
invariant value of E(xx, 0') does significantly influence the direction of z's perturbation because it becomes almost constant in No.6 Therefore, we cancel out this term in Eq. 6
For the sake of simplicity, we do not inject the notation of o in Oo,p'
5

---

Published as a conference paper at ICLR 2024
Additionally, we cannot specify the regional shape of N2, it is infeasible search maximal point 6' No.9 As a consequence, we utilize @p.p' which is the largest p'-ball region within No.9 to approximately search the maximal point 0', should be noted that Ø C N. S 9 where we assume that o gradually approaches p during the SAM optimization. Through the first- order Taylor expansion for the maximum point within O'' we can design the aforementioned perturbation loss as Eq. 7 Consequently, the perturbation is carried out in direction that maximizes the loss and p' weighted gradient norm of the original input z.
Inconsistency Minimization on 0 After the perturbation on x E D., we get an inconsistency- aware perturbed dataset, D&; which approximates the worst-case of unobserved domains, Accord- ingly, we can formulate the optimization of IZ (0) based on 0 as ming max
(Lp.(0) ( (0))
8' 8'ENI66
For the above min-max optimization, we approximate the search for the maximum parameter in a closed-form using second order
similar to the approach of SAM in Eq. T
max of D'ENTY
( (0)) (=.())))<p,(00 (0) LD. P'l (0)l +
max E'NN 2
H D.
(8)
= (0) (0)) + ( (VooD, + + max p /x,-
(9)
Full derivation and approximation procedure of Eq, and Eq, 9 are in Appendix B.2 HD, in Eq.
denotes a Hessian matrix of the perturbed dataset, D.. Also, X, in Eq. 9 denotes i-th eigenvalue of
H Finally, Eq. [ becomes the objective with three components: 1) the loss difference between
D. and D.. 2) the gradient norm of D. and 3) the maximum eigenvalue ratio between D, and D.. Note that X />P- is minimized when Hö, becomes equivalent to HD,,
While Eq. is differentiable with respect to l and can thus be utilized as a tractable objective, com- puting the Hessian matrix for an over- wee--prameterized @ is computationally demanding. line with Rame et al. (2022), we replace the objective with the Hessian matrix (Eq. (9) with an optimization based on gradient variance. Accordingly, Eq. represenns the gradient variance-based objective as an optimization with respect to 0 as follows: (See detailed derivations in Appendix B.3)
min Vol,, (2) + Var G (.)) Var(GD,) 8
(10)
Here, is per-sample gradient for i-th sample; and Gp = {g. 1 is set of per-sample gradient for I; E D. Accordingly, the variance of Gp. which we denote as Var(Gn), is calculated as
Var (ar(Gn)= = 1 EPa (g: g) Matching the gradient variances between two distinct datasets encapsulates specific form of loss matching between them (Rame et al. 2022). This allows to unify loss matching and Hessian matching under a single optimization using Var ().
Summary Our methodology applies the perturbation technique to both input and parameter A. Par- ticularly, the perturbation on inputs and parameters is necessary to minimize I? (0) under unknown domains, which is the unique contribution of this work. Ablation study in Section 4.3 supports that UDIM, which is the combination of the Fq. and Eq. TO, yields the best performance compared to various perturbations and optimizations. Algorithm of UDIM is in Appendix C
A single iteration of UDIM optimization with SAM loss can be described as the following procedure:
1. Construction of D. via Inconsistency-Awar Domain Perturbation on D2 Vz, (f(zz 0.) + Pl Vo, f(x, 80)l2) (f(x, 0.) + V {(x, 11)
D. =((ē,, 9:) (xr, E D.} where 2; = =; + Pe
(11)
2. SAM loss and Inconsistency Minimization on the current parameter 0,
0.41 0, Vo.
max Cp, (0: + +) VV (o.)lll ||Var Var Var(GD,))ll
100112
||c|
(12)
We empirically found that extending it into second-order does affect the resulting performance.

---

Published as a conference paper at ICLR 2024
Table 2: Test accuracy for CIFAR-10-C Each level states the severity of corruption. Bold is the best case of each column or improved performances with the respective sharpness-base based optimizers.
Method
levell
level2
level3
level4
level5
Avg
ERM
75.9 0.5
72.9= 0.4
70.0 0.4
65.9 0.4
59.9 0.5
68.9
IRM (Arjovsky et al. 2019
37.6= 27
36.0= 28
34.6: 26
32
30.8 1.9
34.3
GroupDRO SSgawa al.
2019
76,0= 0.1
7.
69.8
63
59.5# 0.5
68.7
OrgMixup Zhang et al.
2018
77.1=
74.2= u.1
71
61.22 67,4
70.3
Mixup Yan el 2020
76.3=
73.2+
70.2+
6.
60.1
69.2
CutMix
al. 2209
77.9
74.2+
70.8
66.3:
60.0# 0.4
69.8
MTL Blanchard al.
2021
75.6= 0.5
02.7=204
69.9
65.9
60.2 0.3
68.9
LOO
MMD Li et al. 2018b
0.
73.2=:02
70.0
63.7:
59.6: 0.5
69.0
DG
CORAL Sun & Sacnko
2016
76.0= 0.4
72.9+
69.9 0m
65.8
59.62 0.1
68.8
Based
SagNet (Na el al.
202
76.6=
73.6
0.
66.4:1
60.1 0.4
69.5
ARM ZZhang al.
2021
75.7=
72.9=
69.9
65.9
59.8
68.8
DANN Gaamn
2016
75.4=0.4
72.6=0.3
69.7+02
65.6400
59.64022
68.6
CDANN (Li al. 2018c
75.3=02
72.3:002
69.4+02
65.3201
59,4
68.3
VREx Krueger el al.
(2021)
76.0= 0.2
73.0=
70.0
66.0
60.0+ 0.2
69.0
RSC
Huang et al. 2020
76.1=04
7.5
70.1
66.22
60.1 0.5
69.1
Fishr Kame et al.
2022
76.3= 0.3
73.4
70.4+ 0.5
66.3+
60.1 1.1
69.3
SDG Based
M-ADA QQao et al. 2020a
77.2=022
74.2+0.1
71.2+00
67.1
61.1
70.2
LTD (Wang et al.
2021a
0.2
73.0
70.610.0
67.2402
61.7.002
69.6
SAM Foret et al. UDIM SAM
2020
79.0=0a
76.0
72.9
68.7202
62.5
71.8
80.3
77.7=
75.1
71.5
66.2
74.2
SAM Based
SAGM Wang et al.
2023
79.0=0.1
76.2:00.0
73. 73.2:102
69.0.103
62,7:10.
72.0
UDIM w/ SAGM
80.1
77.5=001
74.8
71,2+02
65.9
73.9
GAM ZZhang et
2023b)
79.5=o.1
76.8
74.0+02
69.9201
64.1
72.8
UDIM w/ GAM
81.4=0.1
78.9
76.3
72.8
67.4=10.1
75.3
4 EXPERIMENT
4.1
IMPLEMENTATION
Datasets and Implementation Details We validate the efficacy of our method, UDIM, via experi- ments across multiple datasets for domain generalization. First, we conducted evaluation on CIFAR- synthetic dataset that emulates various domains by applying
10-C (Hendrycks & Dietterich] 2019), 20119) several synthetic corruptions to CIFAR-10 (Krizhevsky et al., 2009). Furthermore, we extend our evaluation to real-world datasets with multiple domains, namely PACS (Li et al. 2017). Office- Home (Venkateswara et al., 2017), and DomainNet (Peng et al.) 2019). Since UDIM can operate regardless of the number of source domains, we evaluate UDIM under both scenarios real-world datasets: !) when multiple domains are presented in the source (Leave-One-Oul Domain General- ization, LOODG); and 2) when single domain serves the source (Single Source Domain Gen- eralization, SDG). Unless specified, we report the mean and standard deviation of accuracies from three replications. Appendix D.IixxDD.ppp provides information on the dataset and implementations.
Implementation of UDIM To leverage a parameter 0 exhibitng flat minima on D. during the min- imization of 1 (0), perform a warm-up training 0 with the SAM loss on D.. While keeping the total number of iterations consistent with other methods, we allocate initial iterations for warm- up based on the SAM loss. As result, we expect to optimize Eq. with sufficiently wide region of N7. for sufficiently low 7-
The gradient variance, Var( (G in Eq. 10 necessitates the costly computation of per-sample gradients with respect to We utilize BackPACK (Dangel al. 2020). which provides the faster computation of per-sample gradients. Also, we compute the gradient variance only for the classifier parameters, which is an efficient practice to improve the performance with low computational costs (Shin al. 2023). Appendix D.I specifies additional hyperparameter settings of UDIM.
Baselines for Comparison Since our approach, UDIM, is tested under both LOODG and SDG scenarios, employed methods tailored for each scenario baselines. These include strategies for robust optimization (Arjovsky et al. 2019) Sagawa et al., 2019) and augmentations for novel
domain discovery (Zhang al. 2018 Yun al. 2019 Nam al. 2021). Appendix enumer--
7

---

Published as a conference paper at ICLR 2024
Table 3: Test accuracy for PACS. Each column represents test domain for LOODG, and train domain for SDG. denotes that the performances are from its original paper. Bold indicates the best case of each column or improved performances when combined with the sharpness-basee optimizers.
Leave-One-Out Source Domain Generalization
Single Source Domain Gemeralization
Method
Art
Cartoon
Photo
Sketch
Avg
Art
Cartoon
Photo
Sketch
Avg
amonz LOODG methods)
$8.4"
78.7"
97.0"
85.5"
46.9.z7
57.2a44
RIDG et 202 1b
86.3111
81.D11.
97.4107
77.5
85.5
76.2211
80.0
48.5.78
54,8124
64.9
ITTA Chen
87.9114
78.6127
96.2101
80.7
85.8
78.4.15
79.8
56.5.17
60.7.u0
68.8
M-ADA
85.5=07
80,7+15
97.2+05
78.4
85.4
78.0=1
79.5e11
47.1
LTD
83.7
79.9
96,9203
83.3
86.4
76.8
82.5
56.2-25
53.6214
67.3
SAM
86 8in8
79.6
96.8in
20.2
85.9
77.7a11
80.5ans
46,7a11
54.2
64.8
UDIM SAM
88.5
97.3
82.7
81.5
85.3ee
67.4
64.6
SAGM
85.3
80.9111
97.1
19.8
85.3
78.9.117
79.8.10
44.7.18
55.6.111
64.8
UDIM w/ SAGM
88,9292
86,2
97.4
79.5
88.0
81,6
84.8
68.1288
63.3:09
74.5
85.5
81.1a1a
4:02
1.00
$6.0
19.1
79.7
46.3.06
16.6:11
UDIM w/ GAM
87.1=00
86,3
97.2
81.8
88.1
82,4
84.2
68.8-88
64.0:a..
74.9
Toll
UDB vy 8AM
6122
A
5003
Henorko
(a) Sensitivity analyses of UDIM on p², Au. and Pa-
(b) Acc over iterations
Figure 3: (a) Sensitivity analyses of UDIM. (b) test accuracy plot of UDIM and sharpness-based approaches based on training iterations. Shaded regions represent standard deviation.
ates baselines for comparisons. For methods that leverage relationships between source domains, straightforward application is not feasible in SDG scenarios. In such cases, we treated each hatch as if it came from different domain to measure experimental performance. ERM means the base model trained with standard classification loss. We also utilize the sharpness-baset approaches as the baselines. Note that the first term of Eq. 4 (SAM loss on D₂) can be substituted with objectives for similar flatness outcomes, such as SAGM Wang et al. 2023) and GAM (Zhang et al., 2023b).
4.2 CLASSIFICATION ACCURACIES ON VARIOUS BENCHMARK DATASETS
To assess the effectiveness of each method under unknown domains, we present the accuracy re- sults on unknown target domains. Table 2 shows the results on CIFAR-IO-C. The sharpness-based methods exhibit excellent performances compared to the other lines of methods. This underscores the importance of training that avoids overfitting to specific domain. By adding UDIM (specifi- cally, the inconsistency term of Eq. 4 to these SAM-based approaches, we consistently observed improved performances compared to the same approaches without UDIM.
Table shows the results on the PACS dataset. Similar to the results in Table 2 UDIM consistently outperforms the existing baselines across each scenario. This improvement is particularly amplified in the single source scenario. Unlike the Leave-One-Out scenario, the single source scenario is more challenging requires generalizing to unknown domains using information from single domain. These results emphasize the robust generalization capability of UDIM under unknown domains.
This section aims examine the efficacy of UDIM by analyzing the sensitivity of each hyper- parameter used in UDIM's s implementation. Additionally, We perform an ablation study by compos- ing each part of the UDIM's objective in Eq. T2 Unless specified, each experiment is carried out in the Single source domain generalization scenario using the PACS dataset.
4.3 SENSITIVITY ANALYSES AND ABLATION STUDY
SeanoOge
Petturh
PerturkOp
Sensitivity Analyses Figure (a) shows the sensitivity
Thot Noreany (X0)
of p. A and Pz. which are main hyper-parameters of UDIM, under the feasible range of value lists. As a de- fault setting of UDIM, we set p =0.05, >1= and p==1.
12
In implementation, we multiply Pz. by the unnormalized gradient of æ. Therefore, the values presented in the ab-
PUD UIDM
60
EIS SAM CObe
Figure 4: Ablation study of UDIM
8

---

Published as a conference paper at ICLR 2024
lation study have larger scale. Also, we compare the performances between SAM and UDIM w/ SAM to compare the effectiveness of UDIM objective. Each figure demonstrates that UDIM's per- formance remains robust and favorable, invariant to the changes in each hypee-parameter. Figure (b) presents the test accuracies over training iterations while varying the ceappeess-basee approaches used alongside UDIM. Regardless of which method is used in conjunction, additional performance improvements over the iterations are observed compared to the original SAM variants.
Ablation Studies Figure 4 presents the ablation results of UDIM, which were carried out by re- placing a subpart of the UDIM's objective with alternative candidates and subsequently assessing the performances. We conduct various ablations: 'SourceOpt' represents the optimization method for D., 'Pertur'' indicates the perturbation method utilized to emulate unknown domains, and 'Per- turbOpt' indicates the optimization for the perturbed dataset D. Appendix D.3 enumerates each ablation candidate and its implementation. UDIM, depicted by the red bar, consistently outperforms in all ablations, suggesting the effectiveness of our derived objective formulation.
lowe: Min loss: 0.43
Min loss: 0.19
Min loss: 0.24
Min loss 0.51 Digen:
Finst Sigen:
First Siger:
Firtt E6g em: 0.4865
0.
6 02
-0.02
-0.02
-2.02
02
0.00
0.0Z
0.02
0.02
0.02
2.02
3.52
0.02
0.02
Max losa: 1.89
Max losa: 1.82
Mux losa: 1.96
Max luss: 1.15
Min
loss: 0:86
2.0
20
2.0
5
05
0.5
0.3
0.3
0.3
0.3
0.0
0.0
0.0
0.0
0.3
-0.3
0.3 -2.3
0.3
0.3
(a) SAM
(b) SAGM
(c) GAM
(d) UDIM
Figure 5: Sharpness plots for models trained using various methods: the upper plot shows sharpness on the perturbed parameter space, while the lower plot displays sharpness on the perturbed data space. The colormap of each row is normalized into the same scale for fair comparison.
4.4 SHARPNESS ANALYSES
Figure Melaims that UDIM would reduce the suggested sharpness in both parameter space and data space. To support this claim with experiments, F enumerates the sharpness plots for models trained with sharpness- bbareness-based methods and those trained with UDIM. These figures are obtained by training models in the single source domain setting of the PACS dataset, and each plot is drawn utilizing target domain datasets, which are not utilized for training.
The top row of Figure 5 represents the measurement of sharpness in the parameter space by in- troducing random perturbation to the parameter 0. Additionally, to examine the sharpness in the perturbed data space of unobserved domains, the bottom side of Figure Jilllustatts sharpness based on the imput-perturbed region of the target domain datasets. Current SAM variants struggle to main- tain sufficient flatness within both the perturbed parameter space and data space. On the other hand, UDIM effectively preserves flatness in the perturbed parameter space and the data space of unknown domains. Within the region, the model trained using UDIM also exhibits a lower loss value com- pared other methods. Through preserving flatness in each space, confirm that the optimization with UDIM, both in parameter and data space, has practical effectiveness.
5 CONCLUSION
We introduce UDIM, a novel approach to minimize the discrepancy in the loss landscape between the source domain and unobserved domains. Combined with SAM variants, UDIM consistently improves generalization performance on unobserved domains. This performance is achieved by perturbing both domain and parameter spaces, where UDIM optimization is conducted based on the iterative update between the dataset and the parameter. Experimental results demonstrate accuracy gains, up to 9.9% in some settings, by adopting UDIM in current sharpness-base -hase approaches.
9

---

Published as a conference paper at ICLR 2024
ACKNOWLEDGMENTS
This research was supported by AI Technology Development for Commonsense Extraction, Reason- ing, and Inference from Heterogeneous Data([ITP) funded by the Ministry of Science and ICT(2022-- 0-00077).
REFERENCES
Martin Arjovsky, Léon Bottou, Ishaan Guirajani, and David Lopez-Paz. Invariant risk minimization. arXiv preprint arXiv:1907.0289) 2019.
Gilles Blanchard, Aniket Anand Deshmukh, Ürun Dogan, Gyemin Lee, and Clayton Scott. Domain generalization by marginal transfer learning. The Journal of Machine Learning Research, 22(1):
46-100, 2021.
Liang Chen, Yong Zhang, Yibing Song. Ying Shan, and Lingqiao Liu. Improved test-time adaptation for domain generalization. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 24172 24182, 2023a.
Liang Chen, Yong Zhang, Yibing Song, Anton van den Hengel, and Lingqiao Liu. Domain gener- alization via rationale invariance. Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 1751-1760, 2023b.
Casey Chu, Kentaro Minami, and Kenji Fukumizu. Smoothness and stability in gans. In Interna- tional Conference on Learning Representations, 2019.
Gabriela Csurka. Domain adaptation for visual applications: A comprehensive survey, arXiv preprint arXiv:1702.05377 2017.
Felix Dangel, Frederik Kunstner, and Philipp Hennig. BackPACK: Packing more into backprop. In International Conference on Learning Representations, 2020. URL, /tl.ittt/////eeeevviieee
net htte//orummidd3311F2222
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hi- erarchical image database. In 2009 IEEE conference on computer vision and pattern recognition, pp. 248-255. leee, 2009.
Gintare Karolina Dziugaite and Daniel M Roy. Computing nonvacuous generalization bounds for deep (stochastic) neural networks with many more parameters than training data. arXiv preprint arXiv:1703.1100: 2017.
Dan Feldman. Introduction core-sets: an updated survey, arXiv preprint arXiv:2011. 2020.
Pierre Foret, Ariel Kleiner, Hossein Mobahi, and Behnam Neyshabur. Sharpness-aware minimiza- tion for efficiently improving generalization. In International Conference on Learning Represen- tations, 2020.
Yaroslav Ganin, Evgeniya Ustinova, Hana Ajakan, Pascal Germain, Hugo Larochelle, François Laviolette, Mario Marchand, and Victor Lempitsky. Domain-adversarial training of neural net- works. The journal of machine learning research, 17(1):2096-200 2016.
Ian J Goodfellow, Jonathon Shlens, and Christian Szegedy. Explaining and harnessing adversarial examples. arXiv preprint 1rXiv:1412.6572, 2014.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In Proceedings of the IEEE conference computer vision and pattern recognition, pp- 770-778, 2016.
Dan Hendrycks and Thomas Dictterich. Benchmarking neural network robustness to common cor- ruptions and perturbations. arXiv preprint arXiv: /903. 2019.
Zeyi Huang, Haohan Wang, Eric P Xing, and Dong Huang. Self-challenging improves cross-domain generalization. In Computer EssionEECCy 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part II 16, pp. 124-140. Springer, 2020.
10

---

Published as a conference paper at ICLR 2024
JoonHo Jang, Byeonghu Na, Dong Hyeok Shin, Mingi Ji. Kyungwoo Song, and Il chul Moon. Unknown-aware domain adversarial learning for open-set domain adaptation. In Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho (eds.), Advances in Neural Information Processing Systems, 2022. URL https: /ttp:////p/eeevveeeeeee ew wwt. i d=IdddTwCC____ I==
Minyoung Kim, Da Li, Shell X Hu, and Timothy Hospedales. Fisher sam: Information geometry and sharpness aware minimisation. In International Conference on Machine Learning, pp. 11 1114 11161. PMLR, 2022.
Yoon- Yeong Kim, Youngjae Cho, Joonho Jang. Byeonghu Na, Yeongmin Kim, Kyungwoo Song, Wanmo Kang, and Il-Chul Moon. SAAL: Sharpness-aware active learning. In Andreas Krause, Emma Brunskill, Kyunghyun Cho, Barbara Engelhardt, Sivan Sabato, and Jonathan Scarlett (eds.), Proceedings of the 40th International Conference on Machine Learning, volume 202 of Proceedings of Machine Learning Research, pp. 16424-16440. PMLR, 23-29 Jul 2023. URL https://procestt www.ingg.ssr....mmmmm mlr www.ppresssvvee http:/kmmmmcc...mm
Diederik Kingma and Jimmy Ba. Adam: method for stochastic optimization. arXiv preprint arXiv:1412.6980 2014.
Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009.
David Krueger, Ethan Caballero, Joer--Herrik Jacobsen, Amy Zhang. Jonathan Binas, Dinghuai Zhang, Remi Le Priol, and Aaron Courville. Out-of-distribution generalization via risk extrapo- lation (rex). In International Conference on Machine Learning, pp. 5815-5826. PMLR, 2021.
Jungmin Kwon, Jeongseop Kim, Hyunseo Park, and In Kwon Choi. Asam: Adaptive sharpness- aware minimization for scale-invariant learning of deep neural networks. In International Con- ference on Machine Learning, pp. 5905-5914. PMI.R, 2021.
Beatrice Laurent and Pascal Massart. Adaptive estimation of a quadratic functional by model selec- tion. Annals of statistics, pp. 1302-1336 1338, 2000.
Da Li, Yongxin Yang, Yi-Zhe Song, and Timothy M Hospedales. Deeper, broader and artier domain generalization. 5542-5550, 2017.
Proceedings of the IEEE international conference on computer vision, pp-
Da Li, Yongxin Yang. Yi-Zhe Song, and Timothy Hospedales. Learning to generalize: Meta-learning for domain generalization. Proceedings of the AAAI conference on artificial intelligence, vol- ume 32, 2018a.
Haoliang Li, Sinno Jialin Pan, Shiqi Wang, and Alex C Kot. Domain generalization with adver- sarial feature learning. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 5400-5409, 2018b.
Lei Li, Ke Gao, Juan Cao. Ziyao Huang, Yepeng Weng, Xiaoyue Mi, Zhengze Yu, Xiaoya Li, and Boyang Xia. Progressive domain expansion network for single domain generalization. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 224- 233, 2021a.
Pan Li, Da Li, Wei Li, Shaogang Gong, Yanwei Fu, and Timothy M Hospedales. A simple feature augmentation for domain generalization. In Proceedings of the IEEE/CVF International Confer- 2021b.
ence on Computer Vision, pp.
Ya Li, Xinmei Tian, Mingming Gong, Yajing Liu, Tongliang Liu, Kun Zhang, and Dacheng Tao. Deep domain generalization via conditional invariant adversarial networks. In Proceedings the European conference on computer vision (ECCV), pp. 639, 2018c
Alcksander Madry, Aleksandar Makelov, Ludwig Schmidt, Dimitris Tsipras, and Adrian Vladu. Towards deep learning models resistant to adversarial attacks. arXiv preprint 17X.vv/706000000 2017.
David A McAllester. Pac-bayesian model averaging. In Proceedings of the twelfth annual confer- ence on Computational learning theory, pp. 164-170. 1999.
11

---

Published as a conference paper at ICLR 2024
Hyeonseob Nam, HyumJae Lee, Jongchan Park, Wonjun Yoon, and Donggeun Yoo. Reducing do- main gap by reducing style bias. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 8690-8699, 2021.
Giambattisa Parascandolo, Alexander Neitz, ANTONIO ORVIETO, Luigi Gresele, and Bernhard Schölkopf. Learning explanations that are hard to vary. In International Conference Learning Representations, 2020a.
Giambattista Parascandolo, Alexander Neitz, Antonio Orvieto, Luigi Gresele, and Bernhard Schölkopf. Learning explanations that are hard to vary. arXiv preprint urXiv:2009.00322 2020b.
Xingchao Peng, Qinxun Bai, Xide Xia, Zijun Huang, Kate Saenko, and Bo Wang. Moment matching for multi-source domain adaptation. In Proceedings of the IEEE International Conference on Computer Vision, pp. 1406-1415, 2019.
Fengchun Qiao, Long Zhao, and Xi Peng. Learning to learn single domain generalization. In Pro- ceedings of the IEEE/CVE Conference Computer Vision and Pattern Recognition, pp. 12556 12565, 2020a.
Fengchun Qiao, Long Zhao, and Xi Peng. Learning to learn single domain generalization. In Pro- ceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 12556- 12565, 2020b.
Alexandre Rame, Corentin Dancette, and Matthieu Cord. Fishr: Invariant gradient variances for out-of-distribution generalization. International Conference on Machine Learning, pp. 18347- 18377. PMLR, 2022.
Harsh Rangwani, Sumukh K Aithal, Mayank Mishra, Arihant Jain, and Venkatesh Babu Radhakr- ishnan. A closer look at smoothness in domain adversarial training. In International Conference on Machine Learning, pp. 18378-18399 PMLR, 2022.
Shiori Sagawa, Pang Wei Koh, Tatsunori B Hashimoto, and Percy Liang, Distributionally robust neural networks. International Conference on Learning Representations, 2019.
Nicol N Schraudolph. Fast curvature matrix-vector products for second-order gradient descent. Neural computation, 14(7):1723 1738, 2002.
Seungjae Shin, Heesun Bae, Donghyeok Shin, Weonyoung Joo, and I1-Chul Moon. Loss-curvature matching for dataset selection and condensation. In International Conference on Artificial Intel- ligence and Statistics, pp. 8606-8628. PMLR, 2023.
Changjian Shui, Boyu Wang, and Christian Gagné. On the benefits of representation regularization in invariance based domain generalization. Machine Learning, 11 111(3):95-9155 2022.
Baochen Sun and Kate Saenko. Deep coral: Correlation alignment for deep domain adaptation. In Computer Vision-ECCV 2016 Workshops: Amsterdam, The Netherlands, October 8-10 and 15-16, 2016, Proceedings, Purt III 14, pp. 443-450. Springer, 2016.
Hemanth Venkateswara, Jose Eusebio, Shayok Chakraborty, and Sethuraman Panchanathan. Deep hashing network for unsupervised domaín adaptation. Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 5018-5027, 2017.
Yoav Wald, Amir Feder, Daniel Greenfeld, and Uri Shalit. On calibration and out-of-domain gener- alization. Advances in neural information processing systems, 34:2215-22 22227, 2021.
Jindong Wang. Cuiling Lan, Chang Liu, Yidong Ouyang, Tao Qin, Wang Lu, Yigiang Chen, Wenjun Zeng, and Philip Yu. Generalizing to unseen domains: A survey domain generalization. IEEE Transactions on Knowledge and Data Engineering, 2022.
Pengfei Wang, Zhaoxiang Zhang, Zhen Lei, and Lei Zhang. Sharpness-aware gradient matching for domain generalization. In Proceedings of the IEEE/CVF Conference on Computer Vision and Patlern Recognition, pp. 3769-3778, 2023.
12

---

Published as a conference paper at ICLR 2024
Zijian Wang, Yadan Luo, Ruihong Qiu, Zi Huang, and Mahsa Baktashmotlagh. Learning to diversify for single domain generalization. In Proceedings of the IEEE/CVE International Conference on Computer Vision, pp. 834 843, 2021a.
Zijian Wang, Yadan Luo, Ruihong Qiu, Zi Huang, and Mahsa Baktashmotlagh. Learning diversify for single domain generalization. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 834 843, 2021b.
Shen Yan, Huan Song, Nanxiang Li, Lincan Zou, and Liu Ren. Improve unsupervised domain adaptation with mixup training. arXiv preprint arXiv:2001.00677, 2020.
Sangdoo Yun, Dongyoon Han, Seong Joon Oh, Sanghyuk Chun, Junsuk Choe, and Youngjoon Yoo. Cutmix: Regularization strategy to train strong classifiers with localizable features. Proceed- ings of the IEEE/CVF international conference computer vision, pp- 6023 6032,603, 2019.
Hongyi Zhang, Moustapha Cisse, Yann N Dauphin, and David Lopez-Paz. mixup: Beyond empirical risk minimization. In International Conference on Learning Representations, 2018.
Marvin Zhang. Henrik Marklund, Nikita Dhawan, Abhishek Gupta, Sergey Levine, and Chelsea Finn. Adaptive risk minimization: Learning to adapt to domain shift. Advances in Neural Infor- mation Processing Systems, 244233644-23778 2021.
Xingxuan Zhang, Renzhe Xu, Han Yu, Yancheng Dong, Pengfei Tian, and Peng Cu. Flatness- minimization for domain generalization. arXiv preprint arXiv:2307. / 108, 2023a.
Xingxuan Zhang, Renzhe Xu, Han Yu, Hao Zou, and Peng Cui. Gradient norm aware minimization seeks first-order flatness and improves generalization. In Proceedings of the IEEE/CVF Confer- ence on Computer Vision and Pattern Recognition, pp. 20247-20257, 2023b.
Zhun Zhong, Yuyang Zhao, Gim Hee Lee, and Nicu Sebe. Adversarial style augmentation for do- main generalized urban-scene segmentation. Advances in Neural Information Processing Systems, 35:338-355 350, 2022.
Kaiyang Zhou, Yongxin Yang, Yu Qiao, and Tao Xiang. Domain generalization with mixstyle. arXiv preprint arXiv:2104.02000 2021.
Kaiyang Zhou, Ziwei Liu, Yu Qiao, Tao Xiang, and Chen Change Loy. Domain generalization: A survey. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2022.
13

---

Published as a conference paper at ICLR 2024
A EXPLANATION OF SHARPNESS VARIANTS FOR DOMAIN GENERALIZATION
Gradient norm-Aware Minimization (GAM) Zhang et al. (2023b) introduces first-order flatness, which minimizes a maximal gradient norm within a perturbation radius, to regularize a stronger flatness than SAM. Accordingly, GAM seeks minima with uniformly flat curvature across all direc- tions.
Sharpness-Aware Gradient Matching (SAGM) Wang al. (2023) minimizes original loss,
the corresponding perturbed loss, and the gap between them. This optimization aims to identify minima that is both flat and possesses sufficiently low loss value. Interpreting the given formula, this optimization inherently regularizes the gradient alignment between the original loss and the perturbed loss.
Minimization (FAM) Zhang et al. (2023a) concurrently optimizes both zeroth- order and first-order flatness to identify flatter minima. compute various sharpness metric on the different order, it incurs higher computational cost.
B PROOFS AND DISCUSSIONS
B.1 PROOF FOR THEOREM
First, we provide some theorem, definition, and assumptions needed to prove the Theorem 3.2
Theorem B.1. (Foret et al. 2020) For any p 0 which satisfies Lg, (0) S Be-p(e) Cg. (0 + e), with probability at least ó over realized dataset Defrom 2e with Del = n, the following holds under some technical conditions L. (0):
1011
Cz, (0) S max CD. (0,(0+) +e) + hel HellaSp
p²
where be RR+ + R4 is strictly increasing function.
Definition B.2. Let Og,p {0] I1le' 0|12 S p} and NT.6 (0' 0. LD. (0') LDD (0) S 7)- Assumption B.3. (0) S Eemple Cz (0 + E) where p(c) ~ NN(0, o² for some a >
Assumption B.4. maXegg LD. (6') 2 LD (8') for all 0 E N2
In practice, Assumption B.4 acceptable. Contrary to the dataset D. from an unobserved domain D. a source domain dataset provided to us and hence, available for training. Moreover, the region of N2 would be flat with respect to local minima, 0", from the perspective of the source domain dataset. Therefore, the perturbed loss on the source domain dataset, LD, (9)) would be likely to have a sufficiently low value.
Theorem B.5. For 0 ④ and arbitrary domain e E E, with probability at least 5 over realized datuset from De with [D] = n, the following holds under some technicul conditions on Le.
1
max max oEE B'EN (6p (D, )l hi 1 13
(0) < max (p. + + ||e||
(13)
[3|
where h. IR R2 is a strictly increasing function.
we assume the case of single source domain general- ization, where represents single domain in E. It should be noted that the number of avail- able source domain does not affect the validity of this proof because we can consider multi- ple source domains a single source domain by D. U, Based on the definition,
Proof. For the derivation of Theorem 3.2
(0) E Eees Lg, (0) |e|
(C,,()) (0) + EcES.efs (o)( From Theorem B.1 we can
derive the generalization bound of a source domain s as follows:
12 0
Lg, (0) < max CD. (0 + + ha
(14)
14

---

Published as a conference paper at ICLR 2024
where hg R R4 is strictly increasing function. To define the upper bound of (t)))), we need to find the upper bound of the remaining term, EeeE.efs Lg, (0). Here, we introduce parameter set pr == arg m é. which is the largest p'-ball region around 0 in 8,0" Then,
can construct inequality as follows:
max LD. (0 + + max 1
LD. (0') S
max LD. (0 +
(15)
d'ENTS
||e Is
This is because 6r C No. C D. Similar to Foret et al. (2020):
Kim et al. (2022), we make use
of the following result from
Laurent & Massart 2000):
2
log k
~ .=,N(0,00)) | ko²
1+
with probability at least 1
(16)
Vn
We set oVT + Vlogni Then, it enables us to connect expected perturbed loss and maximum
loss follows:
E CD. (0 + c)] s
max LD. (0 c) Vn Jelemax 1clpSS
(17)
Vn
Here, lam is the maximum loss bound when || p². Also, we introduce o' where p' o' d'(VT- + v)o Then, similar to Eq.
17 we also derive the equation for p' as follows:
EL C..vo..o/ppCCC.O00 cc)
s (1
1 max LD. (0 +e)+ 2
1 _ lemma
(18)
Vn
Vn
(1 vn
max ENNT.
LD. (0')
vn
e,maz
(19)
where le,maz is the maximum loss bound when by summing up all e j and using the fact that le,max S l'emaxi
||:113 (p')². Then, have the below derivation
1
Σ
E E
(0. +0]
(20)
(1
eEE,ey's
<1
(1E) 1)
1
cEE, 8' E'ENTO
Σ
max
LD. (0')
Vn (E) 11
Σ
(21)
EEE,
< (1
max max CD. (8') + max 1' EEE 0' N'E22
(22)
Vn
c.mor oEE
Using Assumption
B.3 and PAC-Bayesiaa generalization bound (McAllester 1999
Dziugaite &
Roy 2017 Foret et al. 2020), we find the upper bound of the sum of target domain losses.
Σ
La, (0) < (() (EE-1) eEE,eds
Σ
Ee..No + c)
(23)
(18) - 1)
eEE,egs
10110 }}}
1011
< (18)
[ (E..NO0.GrpnnCCo.. (0 t)] + hel
(24)
2²
eeE
1012 p
< (1
max max LD. (0)) + cEE o' d'EN7,0
max C' c,max
Σ
hel
(25)
Vn
Vn aEE
(18) -1)
eEC,est
1
max max LD. (0') + Ti( 11012 EEE 6' 6'E
Σ Lo (18-1)(( DEE,n0A
(26)
Vn
P
where he h are strictly increasing functions. We use the fact that the sum of strictly increasing functions is also a strictly increasing function. By integrating results of Eq. 14 and 26
(g) + (2)
<
max LD, (0 +c)
18
E
CEF,eTs
1013
(1
max max LD_ (0') h1 (27)
In
CEE 8' EN
15

---

Published as a conference paper at ICLR 2024
where h : R2+ - R4 is strictly increasing function. The first and second terms of RHS in the above inequality are upper bounded by the maximum perturbed loss for source domain and unknown domain inconsistency loss.
max LD, (0 + + + 1
max max Lp. (0') cEE o' 2
(28)
Vn
-
max Lp, (0+c) (1
max max LD. EEE o' E VNNT
(29)
[l Sp
= max LD. (0 + ()+ + +01+++111-
max max LD. (d)) EEE of D'ENT'
max LD, (0 +())
(30)
|
1kllasp
s max LD, (0 + +6)+ + (1
[
max max LD. (0))
max LD. (0))
(31)
EEE e' N'
d'ENT,
< max LD. ++)+ + (1- |kc|:SS
max max EEE o' EN
(Ln. (o)(
(32)
[
max LD. (0 + + + SeclaSS
max max EEE 6' V'N
[CD, (0') LD, (
(33)
[
The last equality comes from the Assumption B.4 To sum up, we can derive the upper bound of the population loss for whole domain using the maximum perturbed loss for source domain and unknown domain inconsistency loss with weight decay term.
max max (t') (D, ) + h 10113) p²
L I max LD. (0 +c)++11-- [3]
(34)
|c
eEE 6' d'EN7.
B.2 DETAILED EXPLANATION ON APPROXIMATION
Here, we show the full derivation of Eq. and
as follows.
max
(LD,(0') (0') LD. LD.(t')
(t')
max
(6' LL, (0)+ +
(35)
8' ENNTP
8' ENN
1 0 L
2 2nd Taylor
(0) (.)
VoL (0)l + max
H H
(36)
2
/VVVV (0)ll2 + max X D. /XP
=
(37)
B.3 DISCUSSION ON HESSIAN MATRIX AND GRADIENT VARIANCE
Hessian Matching This section first discusses how the Hessian matrix matching between two different datasets could be substituted by the gradient variance matching for the respective datasets. It should be noted that we follow the motivation and derivation of Rame et al. (2022). and this section just re-formalize the derivations based on our notations. gi a per-sample gradient for i-th sample; and G = |0 is a set of per-sample gradient for I; E D. Accordingly, the variance
of G.. which we denote Var Par(G)), calculated Var(G = |dd Σ DI
(g: ?.) We first
revisit the Eq. 37 which is our intermediate objective as follows:
(FD. (0) (0) + o'llVoL (0) + 7 max /
(38)
The last term of Eq.
38 max X5, /xpr, refers to the maximum eigenvalue ratio of the Hessian
matrices between two different datasets, D. and D.. This ratio minimized when Hessian matrices of D. and D, becomes equivalent. Then, max XP- />P is approximated to the hessian matrix
matching between D. and D, as || - HD, 1l2. Computing the full Hessian matrix in over- parameterized networks is computationally challenging. Therefore, express the formula using the diagonalized Hessian matrix, denoted as Ap,, which results in [ -Hp,l2
16

---

Published as a conference paper at ICLR 2024
Let the Fisher Information Σi= (- Ve logglllplelel pe(ū]z,) Ve log (p) J. where pa (),) is the density of fo on imput instance Fisher Information Matrix (FIM) approximates the Hessian H with theoretically probably bounded errors under mild assumptions SSchraudolpi (002). Then, diagonalized Hessian matrix matching between D, and Da. |H - Hp, lla could be replaced by |F Fp, lla, where F also denotes the diagonalized version of F. Empirically, F is equivalent to the gradient variance of the trained model, fe. This finally confirms the validation of our objective, gradient variance difference between D, and D2 ||Var(,,, ) Var p. Table on the main paper of Rame et al. (2022) empirically supports that the similarity between Hessian diagonals and gradient variances over 99.99%.
Matrix
Rame
et al. 2022
F
Loss Matching Matching the gradient variances for all parameters of our model, fe. incurs sig- nificant computational overhead. In this study, restrict the gradient variance matching subset of entire parameters, specificall y selecting the classifier parameters. In this section, we demonstrate that by matching gradient variances of two different datasets based on the classifier parameters, inherently achieve the loss matching across those datasets.
For simplicity notation, we refer an arbitrary domain index as e. Let æ represent the i-th sample and v be its corresponding target class label. We denote 2 E R as the features for this i-th sample from domain e. The associated classifier layer is characterized by weights {wx] and bias b. First, we assume the mean squared error our loss function for our analysis. For the i-th sample, the gradient of the loss with respect to b is given by VbL((o(x))) vt) = (So(x)) yt). Hence, the gradient variance based on the parameter b for domain is given by: Var(G', D.
Te Ei (fo(x() v²)², which directly aligns with the mean squared error (MSE) between the predictions and the target labels in domain e. Considering our objective, || [arrGG ) Dar((,,,)ll
the gradient variance matching on b could be recognized as mean squared error loss matching, which
|| 12.1 (o (fe(x') v²)² D.. Exs WED,, (fa(z)) 2
Analysis on the remaining term We also investigate the gradient variance matching based on {wx E W, which are remaining part of the classifier parameter W. The gradients with respect to the wa is derived as V., (vé it) = (ft =( Thus, the uncentered gradient variance in wa 1 Σ ((i ) Different from the case of b, Var(G))) adds a weight zi,h on the mean squared error. As 24kk act as a weight, gradient variance matching on wa still learns toward matching the MSE loss between different datasets.
for domain is: Var(G
17

---

Published as a conference paper at ICLR 2024
C ALGORITHM OF UDIM
Here, we present the algorithm of UDIM as follows.
Algorithm 1: Training algorithm of UDIM w/ SAM
Input: Source dataset D.: perturbation threshold for model parameter and data, o' and pai learning rate warmup ratio for source domain flatness, p; number of total training iterations, N; Other hyperpameters.
Output: Trained model fa(
for i 1,... N/p do
| Warmup fo(x) with SAM optimization as Eq. end
for : = ./,,..., N do
Define Dg = {(r v.) i.i.d. sampled from D2
Make DB = {(0,, y1) b= with 2; for all as I. = 2, Pr v., ((x16,,)-p1(V_., 0(.00))
V.
Update fe(x)) by 0++1 + O2 Vo. max LDz (0, + + // Ve, Lo (0+)112 |Ellas5
\ar Var(GGn))la A2110ell2)
end
We represent the algorithm of UDIM with SAM as a default setting. It should be noted that our method can be orthogonally utilized with other sharpness-based optimization methods.
D EXPERIMENT
D.1 IMPLEMENTATION DETAILS
Dataset Explanation
PACS (Li et al. 2017) comprises four domains, which are photos, arts, cartoons and sketches. This dataset contains 9,991 images. It consists of classes.
OfficeHome Venkateswaaa al. 2017) includes four domains, which are art, clipart, prod- uet and real. This dataset contains 15,588 images. It consists of 65 classes
DomainNet (Peng et al. 2019) consists six domains, which are clipart, infograph, paint- ing, quickdraw, real and sketch. This dataset contains 586,575 images. It consists of 345
classes.
CIFAR-IO-C (Hendrycks & Dietterich/ 2019) has been utilized to evaluate the robustness of trained classifier. Images corrupted from CIFARIO (Krizhevsky al. 2009) test dataset under 5 levels of severity. Corruption types include, brightness, contrast, defocus- blur, elastic-transfomm fog, frost, gaussian-blur, gaussian-noise glass-blur, impulse-noise. jpeg corg-ssorrssinn, motion-blur blur, pixelate, saturate, shot-noise, snow, spatter, speckle-noise zoom-blur total of 19 types.
Network Architecture and Optimization We ResNet-18 18 (He al. 2016) for CIFAR- 10-C
and ResNet-50 (He et al. 2016) for other datasets pretrained on ImageNet (Deng et al. 2009) and use
Adam (Kingma & Ba (014) optimizer basically. learning rate is set 3 x 10-3 following Wang eet al. (2023). For calculating the gradient related materials, e.g. Hessian, we utilize BackPACK
(Dangel et al. 2020
package.
Experimental settings For PACS and OfficeHome, trained for total of 5,000 iterations. For DomainNet. we trained for 15,000 iterations. For CIFAR-10 10, since usually trains for 100 epochs, translate to iterations, which becomes total of 781 100 78, 100 iterations. Unless specified,
18

---

Published as a conference paper at ICLR 2024
use batch size as 32 for PACS, OfficeHome, and DomainNet and 64 for CIFAR- 10-C. For other hyperparameters, we follow the experiment settings of Wang et al. (2023) unless specified. Although our method mainly focuses the domain generalization, concept could be also effectively utilized for domain adaptation Csurka (017)) and open-set domain adaptation Jang et al. (022).
Hyperparameter setting of UDIM The main hyperparameters of UDIM is p.p's A and Pa- Throughout all experiments, we f=0.05 without any hyperparameter tuning. For p used values [0.01, 0.025, 0.05]. and in most experiments, the value 0.05 consistently showed good per- formance. should be noted that warm-up using the SAM loss is required before the full UDIM optimization ensure that p =0.05 can be utilized validly. For both λ and Pz. we used values the range [1,10] and reported the best performances observed among these results. Our methodology applies perturbations to each instance of the original source domain dataset, effectively doubling the number of unique instances in single batch compared to experiments for the baselinse. As a result, we utilized half the batch size of other baselines.
Evaluation Detail For reporting the model performance, model selection criterion is important. We get the test performance whose accuracy for source validation dataset is best. For PACS and OfficeHome, we evaluated every 100 iterations and for DomainNet, we evaluated every 1,000 it- erations for Leave- Seave-One-Cee One-Out Source Domain Generalization and every 5,000 iterations for Single Source Domain Generalization.
D.2 BASELINE DESCRIPTION
In this paragraph, we explain baselines that we used for comparison. Specifically, we compare our method with (1) methods whose objectives are mainly related Leave-One Out Source Domain Generalization, (2) methods which are mainly modeled for Single Source Domain Generalization, and (3) sharpness-are aware minimization related methods, as we reported in tables repeatedly.
IRM Arrovsky et al. 2019) tries to learn a data representation such that the optimal classifier matches for all training distributions. Specifically, it minimizes the empirical risk and the regular- ization term, the multiplication of samples' gradients, motivate the invariance of the predictor.
GroupDRO (Sagawa et al. 2019) minimizes the loss by giving different weight to each domain.
Weight term for each domain is proportional to the domain's 's current loss.
OrgMixup Zhang et al., 2018) represents the naive mixup technique which is generally utilized in machine learning community to boost generalization.
Mixup Yan al.
2020) a mixup among domains.
2019) is another skill which is widely used in machine learning community to Cutmix Yun et al.
boost generalization. Specifically, it mixes up parts of inputs randomly by pixel-wise.
Mixstyle (Zhou et al., 2021) mix up the statistics (specifically, mean and standard deviation) of the feature. The mixed feature statistics are applied to the style-normalized input. We did not consider the domain label.
MTL (Blanchard et al. 2021 considers the exponential moving average (EMA) of features.
MLDG (Li et al. 8al is meta learning based method for domain generalization. Specifically, it simulates the domain shift between train and test during training procedure by synthesizing virtual testing domains within each mini-batch. Then it optimizes meta loss using the synthesized dataset.
MMD Li et al. 2018b) minimizes the discrepancy of fcature distributions in a every domain pair- wise manner, while minimizing the empirical risk for source domains.
CORAL (Sun & Saenko 2016) is similar to MMD. However, while MMD employs the gaussian kernel to measure the feature discrepancy, CORAL aligns the second-order statistics between dif- ferent distributions with a nonlinear transformation. This alignment achieved by matching the correlations of layer activations in deep neural networks.
SagNet (Nam et al, 202T) disentangles style features from class categories to prevent bias. Specif- ically, it makes two networks, content network and style network, and trains both networks to be
19

---

Published as a conference paper at ICLR 2024
invariant to other counterpart by giving randomized features (updating the content network with randomized styled features and vice versa).
ARM Zhang et al. 2021 represents adaptive risk minimization. Specifically, it makes adaptive risk representing context.
DANN represents Domain Adversarial Neural Networks, and it iteratively trains discriminator which discriminates domain and a featurizer to learn a feature which becomes invariant to domain information.
CDANN is class conditional version of DANN.
VREx Krueger et al.
2021 controls the discrepancy between domains by minimizing the variance
of loss between domains.
RSC (Huang et al.
challenges the dominant features of training domain (by masking some specific percentage of dominant gradient), so it can focus on label-related domain invariant features.
2020)
2022) approximates the hessian the variance of gradient matrix, and they align the gradient variance each domain.
Fishr (Rame et al.
M-ADA (Qiao et al. 2020a) perturbs input data to simulate the unseen domain data, yet with ad- equate regularization not to make the data be too far from the original one. The adversarial per- turbation direction is affected by the wasserstein autoencoder. Note that this method is specifically designed for Single source domain generalization.
LTD (Wang al. 2021a) perturbs source domain data with augmentation network, maximize the mutual information between the original feature and the perturbed feature so that the perturbed feature not too far from the original feature (with contrastive loss), and maximize likelihood of the original feature. Note that this method is also specifically designed for Single source domain generalization.
SAM (Foret et al. 2020 is optimization technique to consider the sharpness of loss surface. It first perturbs parameter its worst direction, gets gradient and update the calculated gradient the original parameter point.
SAGM (Wang et al., 2023) minimizes an original loss, the corresponding perturbed loss, and the gap between them. This optimization aims to identify minima that is both flat and possesses sufficienlly low loss value. Interpreting the given formula, this optimization inherently regularizes the gradient alignment between the original loss and the perturbed loss.
GAM (Zhang et al.) 2023b) introduces first-order flatness, which minimizes maximal gradient norm within a perturbation radius, regularize stronger flatness than SAM. Accordingly, GAM seeks minima with uniformly flat curvature across all directions.
RIDG (Chen al. 2023b) presents new approach in deep neural networks focusing on decision- making in the classifier layer, diverging from the traditional emphasis on features. introduces a 'rationale matrix', derived from the relationship between features and weights, to guide decisions for each input. A novel regularization term is proposed to align each sample's rationale with the class's mean, enhancing stability across samples and domains.
ITTA (Chen et al. 2023a) proposes an Improved Test-Time Adaptation (ITTA) method for domain generalization. ITTA uses learnable consistency loss for the TTT task to better align with the main prediction task and introduces adaptive parameters in the model, recommending updates solely during the test phase. This approach aims address the issues of auxiliary task selection and parameter updating in test-time training.
D.3 ABLATION
Figure of the main paper presents the ablation results of UDIM, which were carried out by replac- ing a subpart of the UDIM's objective with alternative candidates and subsequently assessing the performances. This section enumerates enumerates each ablation candidate and its implementation. We conduct various ablations: 'SourceOpt' represents the optimization method for D.. 'Perturb' indicates the perturbation method utilized to emulate unknown domains, and 'PerturbOpt' indicates
20

---

Published as a conference paper at ICLR 2024
the optimization for the perturbed dataset D. It should be noted that each ablation means that only the specific part is substituted, while keeping the other parts of UDIM unchanged.
SourceOpt The optimization for the source domain dataset in UDIM is originally based on the SAM loss, which max LD, (0 + c). To verify the significance of flatness modeling, we replaced the original optimization with simple ERM, which we refer to as the ERM ablation.
Perturb The perturbation process of UDIM is conducted based on Eq. 11 We substituted the perturbation method in Eq. 17 with traditional adversarial attack techniques, which the cases of FGSM (Goodfellow et al. 2014) and PGD (Madry et al. 2017), to compare their performance outcomes.
PerturbOpt Lastly, to observe the ablation for inconsistency minimization between the perturbed domain dataset D, and D., we replaced the optimization in Eq. 10 with both ERM and SAM-based optimizations. Each case the PerturbOpt ablation is denoted as ERM or SAM.
D.4 ADDITIONAL RESULTS
In this section, we report more results that we did not report in the main paper due to the space issue.
Table Ashows shows the model performance of total baselines (At the table 3 of the main paper, there are only the model performance of some baselines). As we can see in the table, our method, UDIM, con- sistently improves SAM-based optimization variants and show best performances for each column. We mark- for training failure case (when the model performance is near 1%).
Table 4: Test accuracy for PACS. For Leave-One-Our Source Domain Generalization, each col- umn represents test domain, and train domain for Single Source Domain Generalization. *denotes performances are from its original paper considering LOODG. For SDG scenario, we generated the experiment results for all baselines. Bold indicates the best case of each column or improved per- formances when combined with the respective charpness-based optimizers.
Leave-One-Out Source Domain Generalization
Single Source Domain Generalization
Method
Art
Cartoon
Photo
Sketch
Avg
Art
Cartoon
Photo
Sketch
Avg
BRM
86.9
79.5
96.62n5
78.2
85.3
79.9209
79.9
48. 1258
59.6211
66.9
IRM
85.0" 11
77.6' =0.9
96.7' 103
78.5 #26
84.4"
73.3
77.8+23
46.9:as
49.7a30
61.9
GroupDRO
84.8#22
79.4s12
97.3+03
75.8
84.3
79.0#a5
79.0sas
42.0a29
60.8s39
65.2
OrgMixup
87.7+03
77.4a13
97.6+03
76.3=00
84.8
74.5
79.8+a4
46.8+21
55.5#19
64.1
Mixup
86.9413
78.2.00
97.8
73.7=29
84.2
77.4
80.0+12
47.3+16
58.2+12
65.7
CutMix
80.5
75.7
97.0kgs
74.8
82.0
71.1
76.41.1
37.7203
5014444
58.9
Mixstyle
84.4=23
80.4=06
95.6+a1
80.5
85.2
18.
78.8
56.1+33
$4.7=29
66.9
MTL
452
78.8
96.5+02
74.4=22
83.8
76.7+1.2
78.7+117
44.7#20
59.5+14
64.9
MLDG
87.7
77.5
96.6106
75.3219
84.3
MMD
84.5" 106
79.7" =0.7
97.5"=04
78. 13
85.0'
75.4+111
80.1
45.2
58.2
64.7
CORAL
87.7 106
79.2' =1.1
97.6'=00
79.4" +007
86.0'
76,3+05
79.2+20
45.9
57.0+14
64.6
SugNet
87.1
78.0
96.8102
78.4=11
85.1
71.0
78.9118
47.6224
56.4140
65.1
ARM
86.4
78.8200
16.1
75.1
84.1
76.2+a0
75.5144
45.2+57
61.9:20
64.7
DANN
85.9 +05
79.9' 114
97.6'=02
75.2" +2.
84.6'
79.0
76.5+20
48.7+21
$7.9
65.5
CDANN"
84.0"
78.5"
97.0"
71.8"
82.8"
78.5
78.7420
48.3
56,9
65.6
VREs
87.2
77.8
96.82ns
75.2
84.3
75.3
80.2194
44.9
56.8226
64.3
RSC
81.0
77.6
95,3108
75.0=14
82.2
68.9223
70.6136
41.1a3.
45.9251
56.6
Fishr"
88.4"
78.7"
97.0"
77.8"
85.5*
75.9
81.1
46.9207
57.2144
65.3
RIDG
86.3
81.0=
97.4+00
77.5=25
85.5
76.2+14
80.0
48.5#28
54.8+24
64.9
ITTA
87.9+14
78.6#27
96.2402
80.7=22
85.8
78,4
79.8+1.
56.5437
60.7
68.8
M-ADA
85.5
80.7a13
97.2405
78.4
85.4
78.0a1.1
79.5
47.1
55.7
65.1
LTD
85.7
79.9209
96.92as
83.3
86.4
76.8107
82.5104
56.2
53.6214
67.3
SAM
86.8
79.6214
96.8201
80.2.07
85.9
77.741a
80,54u
46.7a1.1
54.2
64.8
UDIM w/ SAM
88.5
86.1
97.3
82.7:20.
88.7
81.5
85.3104
67.4:08
64.6211
74.7
SAGM
85,3
80.9
97.14n4
77.8
85.3
78.94112
79.8
44.7a18
55.621.1
64.8
UDIM w/ SAGM
88.9
86.2gn3
97.4204
79.520x
88.0
81.6103
84.8112
68.1208
63.3209
74.5
GAM
85 s5.s
81.1
96.4402
81.0
86.0
19.1
79.7
46.3
56.6211
65.4
UDIM w/ GAM
87. ao.
26.3
97.2
81.8
88.1
82.4
84.2104
68.8
64.0
74.9
Table 5 and 7 shows the model performances for OfficeHome /Venkateswara et al. 2017) and Do-
mainNet (Peng al. 2019) dataset, respectively. Similar to the above tables, UDIM shows per-
21

---

Published as a conference paper at ICLR 2024
Table 5: Results on OfficeHome dataset. represents we got the results from Wang al. (2023) and from Rame et al. (2022) considering Leave-One-Ohr Source Domain Generalization. For
Single Source Domain Generalization case, we report the model performances generated under our experiment setting.
Leave-One-Our Source Domain Generalization
Single Source Domain Generalization
Method
Art
Clipart
Product
Real World
Avg
Art
Clipart
Product
Real World
Avg
ERM
61.4s12
53.5-02
75.9s02
77.1
67.0
55.6s06
52.8=16
50.3+1.1
59,4
54.5
IRM"
61.8
52.3" sin
75.2" 108
77.2" 24.1
66.6"
54,9
53.2
48.6
59.2
54.0
GroupDRO
53.3gn4
75.4.00
76.0111
66.5
52.0.os
50.3
59.3109
OrgMixup
63.7+11
55,42u1
19.1
78.9
68.8
56.0a11
54,4
50,4204
61 0.:07
55.5
Mixup
64.1#a6
54.9=07
76.6.0A
78.7=00
68.6
55.5e00
54.1
49.4=117
59.4
54.6
CuiMix
63.2
52.1
77.2:08
78.1
67.7
53.5108
52.2212
47.7
60.2:00
53.4
Mixstyle*
51.1"
53.2" 2044
68.2" =0.7
69.2" #06
60.4
44,3+00
29.8+112
33.6+00
48. 33.5
MIL
60.1
52,0
75.7.0.3
77.2
66.3
55.3d63
53.3804
49.0
60.4
54.5
MLDG'
13.
54.5" .066
75.9" 104
78.6" 1.0.1
68.2"
MMD
63.0'
53.7"
76.1"
19.5
67.7
52.0a05
50.3a1.1
59.3
54.2
CORAL
64.1=a5
54,5=17
76.2+0.4
77,8+005
68.
55.6+00
$2,8+16
50,3+1.1
59 420.
54.5
SagNet
62.3at1
51,7203
75.4
78. 1807
66.9
56.9
53.4
50.8.003
61.2499
55.6
ARM
59.9#06
51,8
73.3+005
75.7=00
65.2
55.0=a.
51.6=1.1
47.3+00
59. 3003
53.3
DANN
$9.9*
53.0"
76,9"
65.9"
55.2+08
49.3
CDANN
61.5"
50.4 124
74.4" 109
76.6" 10.3
65.7"
$5.2.607
49.9
47.61113
$8.6199
52.8
VREx
61.4110
52.2g03
76.1
77.6208
55.5206
52.6202
49.1
59.3
54.1
RSC
Fishr
62.4
54.4
76.2,00
18.
67.8
$5.1
51.2.661
49.2
59.9114
53.9
RIDG
63.6103
55.0:n9
76.0
77.5
68.0
56.8105
55.4107
50.5
60.9
ITTA
61.81n
57.01
74.3103
77.3
67.6
56.0104
51.5108
50,5106
61.61004
54.9
SAM
62.21n0
55,9.0.1
77.0
78.8106
68.5
56.9104
53.81111
50.9100
61.5
55.8
UDIM (w/ SAM)
63.5g11
58.6204
76.9
79.1
69.5
58.1
55.0
$3.8
64.3
57.8
SAGM
63.1
56.2,n4
77.3102
78.4104
68.8
$7.7
$4.8
$1.5111
61.4101
56.3
(w/ SAGM)
64.42m3
57.3:n5
77.1104
79.1
69.5
58.5
55.7.06
54.5
64.5
GAM
64.0ans
58.6212
77.5
79.3a02
69.8
59.4406
56.1
53.3
63 63,4a02
58.1
UDIM (w/ GAM)
64.2
57.4.n9
77.5
79.3
69.6
58.7.00
55.7
53.6100
64.4:00
58.1
Table 6: Results on DomainNet datasets Leave-One-Out (Multi) Source Domain Generalization
Method
clipart
infograph
painting
quickdraw
real
sketch
Avg
ERM
62. 62.840.4
20,2
50.3403
13.7
63.7
52.
43.8
IRM
48.5+28
15.0
38.34443
10,9
48.2.152
42.343.1
33.9
GroupDRO
47.2
17.5
33.840.
9.3
51.6204
40.1
33.3
MTL.
57.940.
18.5
46.04..
12.5
59.5-0o3
49.24a.1
40.6
MLDG
59.1+022
19.1=003
45.8++0.
13,4+:03
59.6=202
50.2+0.4
41.2
MMD
33.1
11.0=1.6
26.8+113
8.7=2.1
32.7=1388
28.9+11.9
23.4
CORAL
59.2+0.1
19.7=002
46,6103
13,4+0.4
59.8=002
50.1
41.5
SagNet
57.7403
19.0=02
45.3
12.7
58.1
48,8+0.2
40.3
ARM
49.7.1.3
16.3
40.9+11
9.4+0.1
53.4=04
43.5+004
35.5
DANN
53.8*
17.8" +0.3
43.5
11.9"
56.4"
46.7
38.4*
CDANN
53.4"
18.7
44.8*
12.9*
57.5*
46.7"
38.9*
VREx
47.3438
16.0215
35.8144
10.9400
49.6-14
42.0+3.
33.6
RSC
55.0+1.2
18.3200
44.4100
12,2+02
55.7=07
47.8+a99
38.9
Fishr
58.2" 205
20.2"
47.7"
12.7"
60.3*402
50.8*
41.7*
SAM
64.5
20.7
50.2+..1
15.1+03
62.6=02
52.740.
44.3
UDIM (w/ SAM)
63.5
21.01+0.1
50.63
14.7640.1
62.5
53.39
44.3
formance improvement over sharpness-aware baselines, and good performances for each column consistently.
D.5 ADDITIONAL ANALYSES
In this section, we additionally provide analysis on 1) Comparison with Shui et al. (2022), which utilzes both 1) data-based perturbation and 2) parameter-based regularization their own frame- work. Afterwards, we provide visual inspections on the perturbed domain instances, which are constructed by [T
22

---

Published as a conference paper at ICLR 2024
Table 7: Results on DomainNet datasets Single Source Domain Generalization
Method
clipart
infograph
painting
quickdraw
real
sketch
Avg
ERM
27.5+0.
26.6+0.1
28.5=033
7.1+0.1
28.9=04
29.4+0.
24.7
IRM
GroupDRO
27.6 07.6
26.6=008
28,9=03
7.3+03
28.7=002
29.4
24.8
OrgMixup
28.3+001
27.3
29,4=003
7.7
30.2=022
30.0=0.7
25.5
Mixup
27.5
26.9108
28.6100
7.0.0.1
28.7
29.610.6
24.7
CutMix
28.0+02
26.5+022
28, 20.6
6,4+0.1
29.0=033
29.7=0.8
24.6
Mixstyle
18.3
14.5
19.6102
5.010.1
20.5
21.3.008
16.5
MTL
27.3.002
26.610.33
28.7
7.810.1
28.2
28.7.00
24.5
MLDG
MMD
27.61.0.4
26.6.10.7
28.5
7.4103
28.910.
29.5
24.8
CORAL
18.1 112.8
17.4+12.3
18,9+134
4,8+34
19.5+138
20.2
16.5
SagNet
27.6100.
25.610.5
28.5
7.3102
28.8
28.8100
24.4
ARM
17.01120
16.8.119
17.7
4.1129
17.5
18.8
15.3
DANN
26.8+0.8
25.2+09
28.0=04
6,8+066
27.6=022
28.0+002
23.8
CDANN
26.810.7
25.81022
27.6102
6.810.6
27.6102
28.0.0.2
23.8
VREx
18.6+13.1
17.3+123
19.3+13.6
4.7+33
19.3
14.9
16.5
RSC
Fishr
30.0
26.610.77
28.9
7.5
28.4
28.940.3
24.7
SAM
28.4+02
26.9+0.1
29.1
6.9405
30.0=0.2
29.8 29.8+07
25.2
UDIM (w/ SAM)
30.0+0.1
23.8=00
31.0=0.1
12.6=0.1
30.7=002
34.0
27.0
D.5.1
ANALYTICAL COMPARISON WITH SHUI ET AL.
2022
UDIM and Shui et al. (022) share similar direction, both methodologies involve 1) generating virtual samples through distinct data perturbation methods, and 2) implementing their own types of regularization these samples.
Shui et al. (2022) introduced novel regularization techniques for the embedding function based on theoretical analysis. Their approach involves establishing upper bound the balanced error rate the test environment, which is obtained from the combination of the balanced error rates in the source environments, the feature-conditions invariance, and the smoothness of the embedding function. In particular, they focused on minimizing the smoothness term by reducing the Frobenius norm of the Jacobian matrix with respect the embedding function. To implement this regulariza- tion term over unobserved regions, they use virtual samples generated by linear combination of samples from each source.
On the other hand, UDIM also introduces an upper bound on the generalization loss in an unknown domain. This bound includes the SAM loss on the source domain and region-based loss disparity between the worst-case domain and the source domain. This disparity is implemented by gradient variance-based objective outlined Eq. T0 The objective involves a perturbed dataset constructed by perturbing samples, where the direction of perturbation is determined by the inconsistencies described in Eq. 7
D.5.2 ANALYTIC COMPARISON WITH DOMAIN AUGMENTATION AND ADVERSARIAL ATTACK
The proposed method, UDIM, involves applying arbitrary perturbation given instance a new instance, which is similar to domain augmentation and adversarial attacks. However, UDIM has the following differences and advantages compared them.
Comparison with domain augmentation First, taxonomize domain augmentation based on its dependency on learning signals such as parameters or loss, dividing it into not-learnable domain augmentation Zhang al. (0018) Zhou et al. (201): Li al. (2021b) and learnable domain aug- mentation Zhang et al. (2118); Zhou et al. (2021) et al. (2021b). Please note that we briefly cite the representative methodologies among wide range of augmentation techniques.
While non-leamable domain augmentations are effective generating new styles and may gener- alize well specific types of domains, it does not guarantee generalization across wide range of
23

---

Published as a conference paper at ICLR 2024
unseen domains, discussed in the Theorem 3.2 In contrast, UDIM's data perturbation method is designed to generate perturbations towards the most vulnerable or worst domain from a param- eter space perspective, enabling reduction of the generalization bound in Eq 5 even in scenarios involving numerous unobserved domains. Additionally, it's important to note that these domain augmentation techniques could be applied orthogonally to the UDIM framework, for instance, by implementing domain augmentation prior to UDIM's data perturbation.
Learnable augmentations, similar UDIM, determine its augmentation direction based on the cur- rent parameter response. However, these methodologies do not link their augmentation with a the- oretical analysis to assure minimization of the target objective, which is left-hand side of Eq [ of our manuscript. UDIM's ' data perturbation impacts the generalization bound from a parameter per- spective as it takes into account parameter loss curvature information, rather than just a single parameter point, when determining perturbations.
Comparison with adversarial attack Adversarial attacks also introduce perturbations in the di- rection most vulnerable to the current parameters, but methodologies like FGSM Goodfellow et al. (2014) and PGD Madry et al. (2017) do not consider the local parameter curvature in their pertur- bation process. By integrating perturbations on instances with attention to parameter loss curvature; and parameter perturbation, we facilitate the modeling of inconsistency in unknown domains, as described in 3 Having said that, Kim et al. (2023) also utilize worst-case instance selection on the active learning framework by utilizing the parameter perturbation. In the literature of coreset selection Feldman (2020). [Shin et al.) (2023) also utilizes the perturbed parameter region to attain samples which effectively represent whole dataset.
From a mathematical perspective, UDIM's data perturbation involves receiving not only gradients related to the simple cross-entropy loss but also additional gradients concerning the norm of gradient, as elaborated in Eq
Combination of domain augmentation with SAM optimizer We accordingly report the exper- imental results of models, which combines various domain augmentation techniques with SAM optimization in Table We reported the average test accuracy for each domain each setting- Applying SAM optimization to data instances of augmented domains led to mixed results: some methodologies improved, others didn't, but all still under-performed compared to UDIM. We hy- pothesize that the observed performance decline in certain augmentations combined with the SAM optimizer might stem from an unstable learning process. This instability may arise from attempting to minimize sharpness in the perturbed domain prematurely, before ensuring flatness in the source domain.
Method
Leave-One-Out Source Domain Generalization
Single Source Domain Generalization
SAM
85.9%
64.8%
SAM Mixup
62.28%
SAM Mixstyle
86.56%
68.599
SAM w/ Simple Augment Ii et al.
2021b
86.26%
64.97%
SAM advatyle Zhong al.
85. 85.28%
02.%
UDIM
88.7%
74.7%
Table 8: Performance comparison of models using combinations of domain augmentation variants and SAM optimizer, alongside UDIM model, on the PACS dataset.
D.5.3 VISUAL INSPECTION ON THE PERTURBED DOMAIN
In this section, we aim to illustrate how perturbed instances are created depending on the size of P. which represents the magnitude of data perturbation. Each plot utilizes the PACS dataset and model trained under the Single Source Domain Generalization setting.
Figure 6 displays instances perturbed through the original UDIM methodology. As the perturba- tion size increases, we can observe a gradual distortion of the image's semantics, highlighting the importance of selecting an appropriate perturbation size.
24

---

Published as a conference paper at ICLR 2024
Figure shows the scenario where the data perturbation of the original UDIM method is applied to the input channel, instead of input pixels. This approach of perturbing the channel over pixels has the advantage of maintaining the basic shape and line information of the image, ensuring the preservation essential visual features.
=
0.1
Pr = 0.5
Px = 1.0
Pr = 5.0
10.0
50.0
Px 100.0
Art
Cartoon
Photo
Sketch
Figure 6: Pixel-perturbed domain instances by UDIM model trained on PACS dataset by varying the perturbation size
Pr 0
Pr 0.1
Pg = 0.5
Pr = 1.0
Px =
Px = 10.0
P1 50.0
Px = 100.0
Art
Cartoon
Photo
Sketch
Figure 7: Channel-perturbed domain instances by UDIM model trained on PACS dataset by varying the perturbation size
25


Reviews:
[{"review_id": "ljQGeLnGoA", "reviewer": "Reviewer_KWQw", "summary": "This paper focuses on domain generalization through extending the flattened loss landscape in the perturbed parameter space to the perturbed data space. Specifically, they first simulate the unknown domains via perturbing the source data, and then reduce the loss landscape inconsistency between source domains and the perturbed domains, thereby achieving robust generalization ability for the unobserved domain. Theoretical analysis and extensive experiments demonstrate the effectiveness and superiority of this method.", "questions": "See weakness.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1.This work extends the parameter perturbation in existing SAM optimization to data perturbation, achieving loss landscape alignment between source domains and unknown domains. Experiments show the validity of the proposed objective.  \n\n2.This work establishes an upper bound for DG by merging SAM optimization with the proposed objective.\n\n3.The proposed objective can be combined with multiple SAM optimizers and further enhance their performance, demonstrating the necessity of the loss landscape consistency between source and unknown domains.", "weaknesses": "1.I believe that the proposed data perturbation method is consistent in both ideology and essence with traditional domain augmentation and adversarial attack techniques. So, what is the main difference and advantage of the proposed objective? And what if combining some domain augmentation techniques with the SAM optimizers?\n\n2.How to guarantee that the perturbed data is still meaningful, rather than generating some noisy samples? If so, will enforced the loss landscape alignment across domains bring negative impact? Besides, the unknown distribution may not necessarily be within the scope of perturbed data region.\n\n3.What is the sampling strategy for sampling data from $D_s$ to generate perturbed data?\n\n4.Is the optimization only conducted on the perturbed samples during the formal training phase? Would it be more stable to train on both the source and perturbed samples simultaneously?\n\n5.Since the training procedure involves gradient calculation, what is the time complexity after applying the BackPACK technique?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "kASeONNapb", "reviewer": "Reviewer_x9yc", "summary": "This paper considered both parameter- and data-  perturbation in domain generalization. The method is inspired by sharpness aware minimization (SAM). A theoretical analysis is further conducted to show the importance of different perturbations. Finally, the model is deployed in standard benchmarks with improved performance.\n\n------Post-rebuttal \nI would appreciate the rebuttal, which addressed my concerns. I would maintain a positive rating.", "questions": "See weakness part.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "1. This paper considered a reasonable solution in domain generalization. Both parameter and data perturbations are conducted for a robust OOD generalization. \n2. The idea seems novel for me in some settings. \n3. Extensive empirical results.\n\nBased on these points, I would recommend a borderline positive.", "weaknesses": "1. Sometimes I find it a bit hard to understand the rationale of the proposed approach. Why do we need to consider both parameter and data perturbation? For example, in paper [1], a theoretical analysis is proposed, which is analogous to equation (11) as the parameter robust. \n2. Does the choice of data perturbation matter? We know we may face many different possible data-augmentation approaches. Which method(s) do you think should work in this scenario?\n3. Is it possible to consider the subgroup distribution shift in the context of fairness such as papers [2-3]? A short discussion could be great. \n\nReferences:\n\n[1] On the Benefits of Representation Regularization in Invariance based Domain Generalization. Machine Learning Journal (MLJ) 2022.\n\n[2] On Learning Fairness and Accuracy on Multiple Subgroups. Neural Information Processing Systems (NeurIPS) 2022.\n\n[3] Fair Representation Learning through Implicit Path Alignment. International Conference on Machine Learning (ICML) 2022.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ROJE9lV3d2", "reviewer": "Reviewer_DtQi", "summary": "This paper, titled \"Unknown Domain Inconsistency Minimization for Domain Generalization,\" introduces an approach to improve domain generalization using Unknown Domain Inconsistency Minimization (UDIM) in combination with Sharpness-Aware Minimization (SAM) variants.  The paper's novelty lies in its approach to improving domain generalization by focusing on both parameter and data perturbed regions. UDIM is introduced as a novel concept, addressing the need for robust generalization to unknown domains, which is a critical issue in domain generalization. The idea of perturbing the instances in the source domain dataset to emulate unknown domains and aligning flat minima across domains is innovative. While SAM-based approaches have shown promise in DG, UDIM extends the concept to address specific shortcomings, which is a novel contribution.", "questions": "see weakness", "limitations": "", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The paper appears to be technically sound. It provides a well-defined problem statement for domain generalization and formulates UDIM as an optimization objective. The authors validate the theoretical foundation of UDIM and provide empirical results across various benchmark datasets, demonstrating its effectiveness. The methodology is explained clearly, and the experiments are well-documented.\n\n2. The paper is well-structured and clearly written. It provides a thorough introduction, problem definition, and a detailed explanation of the proposed method. The methodology is presented step by step, and mathematical notations are used effectively. The experimental setup and results are also presented in a clear and organized manner. However, the paper is quite technical, and readers with less familiarity with domain generalization and machine learning might find some sections challenging to follow.\n\n\n3. The paper addresses an important challenge in domain generalization, namely, the ability of models to generalize to unknown domains. The proposed UDIM method appears to be effective in improving the performance of existing SAM-based approaches, as demonstrated through experimental results. The potential impact of this paper on the AI research community is significant, particularly in the field of domain generalization.", "weaknesses": "1. Baselines. The baselines are not enough because the latest DG method is Fisher, which is published at 2022.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "AHNBRmBned", "reviewer": "Reviewer_ExGX", "summary": "This paper highlights the importance of exploring the perturbation in data space to enhance the performance of domain generalization based on the sharpness-aware minimization approach. The authors proposed an unknown domain inconsistency minimization approach, which combines data perturbation to generate the worst-case unseen domains and weight perturbation which is sharpness-aware minimization. They further showed some theoretical analysis for their algorithm. Experiments have shown the effectiveness of the algorithm.\n\n-----Post rebuttal\n\nI appreciate the efforts in writing the response. While the efficiency problem is not solved, I acknowledge the efforts in trying to address it. Therefore, I increase my score to 6.", "questions": "1. I cannot see the advantage of UDIM in Figure 5: it seems that the loss landscape (bottom right) of UDIM is no better than SAGM and GAM? Can authors offer further explanations?\n2. On PACS, the results are not consistent with the original results from DomainBed. Authors should double check their results.\n3. What is the efficiency of the method?\n4. What do the generated domains look like?", "limitations": "", "rating": 6, "confidence": 5, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1. The idea of combining data perturbation with weight perturbation is interesting. The weight perturbation is a pretty “standard” approach to learn robust models and the combination of data perturbation can further enhance such robustness.\n2. The paper is easy to understand and follow. The algorithm design is clear.\n3. The experiments have shown the effectiveness of the approach.", "weaknesses": "1. The novelty of the approach is limited since the perturbation of input data is somewhat similar to existing approach called CrossGrad [Shankar et al., 2018] which also perturbs existing training data by reversing their gradient direction to generate unseen training domains. That being said, the paper can be seen as a combination of CrossGrad (by adding original data perturbation) and sharpness-aware minimization.\n2. There lacks guaranteed proof showing that by perturbing the input to the size of $\\rho$ (which is pretty similar to adversarial perturbation), the generated unseen domains can cover the entire space of unseen domains. It seems that with the size of $\\rho$ becomes larger, the generalization risk (Eq. 3) is smaller. However, in that way, the sampling and generating efficiency will be heavily impacted. There are no clear analysis on the parameter $\\rho$: what kind of perturbation can help the method already generalize the same as existing ones; better than existing ones, or worse than existing ones?\n3. Insufficient experiments. The common benchmark in domain generalization is DomainBed, but the paper did not use it; instead, it used CIFAR-10-C and PACS, which are clearly not sufficient. DomainBed has other challenging datasets inclusing CMNIST and DomainNet, which should be tested.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)