Domain-Inspired Sharpness-Aware Minimization Under Domain Shifts
Authors: Ruipeng Zhang
Ziqing Fan
Jiangchao Yao
Ya Zhang
Yanfeng Wang
Venue: ICLR 2024 poster (2024)
Keywords: generalization
sharpness-aware minimization
domain shift
Primary area: transfer learning, meta learning, and lifelong learning
TL;DR: The paper presents DISAM, an algorithm enhancing optimization under domain shifts by ensuring domain-level convergence consistency, resulting in quicker convergence, improved generalization, and outperforming existing methods.

Abstract: This paper presents a Domain-Inspired Sharpness-Aware Minimization (DISAM) algorithm for optimization under domain shifts. It is motivated by the inconsistent convergence degree of SAM across different domains, which induces optimization bias towards certain domains and thus impairs the overall convergence. To address this issue, we consider the domain-level convergence consistency in the sharpness estimation to prevent the overwhelming (deficient) perturbations for less (well) optimized domains. Specifically, DISAM introduces the constraint of minimizing variance in the domain loss, which allows the elastic gradient calibration in perturbation generation: when one domain is optimized above the averaging level w.r.t. loss, the gradient perturbation towards that domain will be weakened automatically, and vice versa. Under this mechanism, we theoretically show that DISAM can achieve faster overall convergence and improved generalization in principle when inconsistent convergence emerges. Extensive experiments on various domain generalization benchmarks show the superiority of DISAM over a range of state-of-the-art methods. Furthermore, we show the superior efficiency of DISAM in parameter-efficient fine-tuning combined with the pretraining models. The source code is released at https://github.com/MediaBrain-SJTU/DISAM.

Published as a conference paper at ICLR 2024
DOMAIN-INSPIRED SHARPNESS-AWARE MINIMIZA- TION UNDER DOMAIN SHIFTS
Ruipeng Zhangl* Ziqing Fan Jiangehao Yao Ya Zhang " Yanfeng Wang 9.8m
f Cooperative Medianct Innovation Center, Shanghai Jiao Tong University
$ Shanghai Artificial Intelligence Laboratory Izhangrp, zqfan_ gnight, Sunarker, ya zhang, wangyanfeng @ @sjtu.edu.cn
ABSTRACT
This paper presents Domain-Inspiret Sharpness-Aware Minimization (DISAM) algorithm for optimization under domain shifts. It is motivated by the inconsistent convergence degree of SAM across different domains, which induces optimization bias towards certain domains and thus impairs the overall convergence. To address this issue, we consider the domain- convergence consistency the sharpness estimation to prevent the overwhelming (deficient) perturbations for less (well) optimized domains. Specifically, DISAM introduces the constraint of minimizing variance in the domain loss, which allows the elastic gradient calibration in pertur- bation generation: when one domain is optimized above the averaging level w.zt. loss, the gradient perturbation towards that domain will be weakened automati- cally, and vice versa. Under this mechanism, we theoretically show that DISAM achieve faster overall convergence and improved generalization in principle when inconsistent convergence emerges. Extensive experiments on various domain generalization benchmarks show the superiority of DISAM over range of state- of-the-art methods. Furthermore, we show the superior efficiency of DISAM in murameter-efficien fine-tuning combined with the pretraining models. The source code is released at https: https://githule com/ wwt.//o/////i//eeeeeeeeeeeeeeeee MeeeMeediiaaannnnn SataaiinnSSSUUDIAAANNAN
INTRODUCTION
Although deep learning has achieved remarkable advances in various areas (He et al., 2016; Doso- vitskiy et al., 2020), it remains a challenge for optimization in pursuit of strong generalization. Especially, a lower training loss does not necessarily guarantee a better generalization, as there exist numerous local minima in the complex and non-convex hypothesis space. Recent empirical and theoretical investigations (Dziugaite & Roy, 2017: Chaudhari et al., 2019; Jiang et al., 2020; 2023; Dinh et al., 2017b; Keskar et al., 2017b) have identified significant correlation between generalization and the sharpness of the loss landscape. This correlation suggests that generalizability be interpreted as flatness in the loss surface, leading a wide range of explorations that have contributed to the rapid development of Sharpness- Aware Minimization (SAM) (Foret et al., 2021).
Existing SAM-based methods predominantly focus on the narrowly defined generalizability between training and test data under the Independent and Identically Distributed (i.id) assumption, which can be summarized two categories. The first strives to improve the performance by creating a more effective estimation of sharpness like the enhanced minimization in GSAM (Zhuang et al., 2022), PGN (Zhao et al., 2022), SAGM (Wang et al., 2023b) and VaSSO (Li & Giannakis, 2023), as vanilla perturbation in SAM fails to accurately capture the geometric flatness of the loss landscape. The other category targets to improve computational efficiency by reducing perturbation directions (Liu et al., 2022) or using more efficient perturbation surrogate (Du et al., 2022a;b), as the original SAM incurs double the computational overhead compared to Empirical Risk Minimization (ERM), Nonetheless, these methods cannot solve generalizability scenarios that involve training data of multiple domains with domain shifts like Domain Generalization (DG) (Ben-Dace Devid et al., 2010; Li et al., 2017).
In this study, we observed that sometimes SAM even has detrimental impact in situations where there exist domain shifts across multiple domains as shown in Figure I(a). While a few studies have incorporated SAM-based methods in domain generalization tasks (Wang al., 2023b; Foret et al.,

---

Published as a conference paper at ICLR 2024
ERM vs SAM on Terralicc
Training on domain 100, 46 of Terralnc
Elr SM
Convergence Inconsistency under domain shift
179
SAMOGES
3496301
30
50
16
with dffoeert
(a) Performance under domain shifts.
(b) Convergence curves under domain shifts.
Figure 1: Illustration of SAM's degradation of the training process under domain shifts. (a) Perfor- mance comparison between ERM and SAM, where SAM consistently performs worse than ERM across all hyperparameters p. (b) Convergence curves of SAM and ERM for each domain during training. with the convergence degree normalized to [0,1]. J. SAM exacerbates the disparity in conver- gence degree among different domains in domain shift scenarios, resulting in inferior generalization performance. The dataset used here is Terra Inc from the DomainBed benchmark, and the back-
bone is ResNet50. Further experimental details are provided in Section 4.1 and Appendix C.5.
2021), they cannot ensure consistent improvements in generalizability during domain shifts due to their reliance on the i.i.d assumption. Upon a thorough analysis of the behavior of SAM under domain shifts, we discovered that the degradation of the training process caused by SAM from the disparity in convergence degree among different domains as shown in Figure (a)). Given the inconsistency in the degree and direction of convergence among different domains during training (Arjovsky et al., 2019; Krueger et al., 2021), the straightforward application of SAM for perturbations may not only disrupt convergence but also generate perturbation directions that are not adequately coherent to the geometric characteristics of the entire loss landscape.
To solve the aforementioned problem, we propose a Domain- Inspired Sharpness-Aware Minimization (DISAM) algorithm. As the degradation origins from the inconsistency in convergences of these domains, DISAM incorporates domain-level convergence information to intervene in the perturbation of the vanilla SAM: the perturbation direction should focus more on domains with higher convergence degree while being mild to domains with lower convergence degree. Under such balancing in perturbation, the gradient update actually implements the domain- level personalization, thus mitigates the impact of domain shifts and enhances the generalization performance. Technically, we ingeniously accomplish the adaptive adjustment the perturbation direction in accordance with the degree of convergence through the domain loss variance minimization constraint. The perturbation of DISAM directs towards location with more consistent convergence degree, enabling a better global view of the loss landscape for gradient update. We summarize our contributions follows:
We identify that the use of SAM has a detrimental impact on training under domain shifts, thereby compromising generalizability, and further analyze that the reason the inconsistent convergence of training domains that deviates from the underlying i.i.d assumption of SAM.
We introduce novel approach called Domain Inspired the problem above. DISAM incorporates domain-level convergence consistency by imposing a variance minimization constraint on domain loss during the sharpness estimation process, thereby enabling a more representative perturbation location and enhancing generalization.
Serppeess-Aaar Minimization to mitigate
Extensive experiments show the superiority of DISAM in improving the current state-of-the-an aft methods on several benchmarks. We also provide a comprehensive analysis of its merit of faster convergence compared to SAM, and show its persistent generalization capabilities under parameter-cfficient fine-tuning with large models like CLIP.
PRELIMINARIES
2.1 BASIC NOTATIONS
S = (Di, ......, DM Overall training set of a M-souree domain generalization task. We denote each domain by D; and the number of samples in D, by n4 = --,]--
E. E A specific sample and the j-th sample in i-th domain D,, respectively.
2

---

Published as a conference paper at ICLR 2024
C, C(w), [(w; A loss function, expected loss under and the specific loss of ç, respectively.
(((w)= = Been,L(us E): Expected loss under w for each domain D.-
Varl.) The variance among M training domains, which holds: Var(Ci(w)) 257 Lis1 ( ((()))))
(w)(w) ((w) Xaar (w)) A loss function with domain-inspired regularizer Var(Li(w)) L. is constant value that controls the strength of the constraint.
Cp(w) maxi( f(w + e): The perturbed loss and the objective of SAM.
w + P (r The sharpness estimation of DISAM with gradient ascend at step t.
VCpr(w/
me Learning rate at step t.
w: Parameters of neural network R& where the dimension.
c R² perturbation on the parameters w with scale p R.
2.2 SHARPNESS- -WWARE MINIMIZATION
In general, simply minimizing ERM tends overfit training data and extensive studies show the correlation between generalizability and the sharpness of minima (Dinh et al., 2017b; Hochreiter Schmidhuber, 1994b; McAllester, 1999; Chaudhari al., 2019). We clarify the concepts as below.
Sharpness.
The sharpness on parameter w with a dataset D and loss function C is:
s sww, D) max EeED[C(w e; [(w; E)]
Sharpness-Aware Minimization (SAM). Foret et al. (2021) proposed SAM to improve the generalization by simultaneously minimizing the loss and the sharpness of the overall loss surface. The objective is defined as:
min max EgED[C(w + e)
(2)
|elSSS
From the above equation, we can see SAM minimizes a perturbed loss "max BGEDIC( ")" which aims to maximize the loss L within radius centered at the parameter w.
+
3
METHOD
3.1 MOTIVATION
Although existing SAM-based methods that minimize the sharpness have achieved good general- ization, in the case of multiple domains with shifts, the inherent heterogeneity in quantity and task difficulty among domains can considerably distort their sharpness estimation, yielding a degradation in the performance. Concretely, with collection S of M domains, each of which contains set of n, samples, i.e., {E) = (z), v² 15 the training objective can be then formulated as follows:
M ne
min (ges[[((:E)) w
1
Σ
((w:
Σ C,C,(w).
N
where N = EM 74, # and L((w) = Σ L(w; g)). Note that, we clarify here that we will ignore the notations data properly in some subsequent equations to avoid clutter. Then, on the basis of Eq. (3), the corresponding objective of SAM under domain shifts is defined as:
min Eges(LsAAMMurr
min max Eges [[(w + e: min max Σ CaC((w+e) (4) Ielss
w
The core that we should point out is whether the approximation from maxijell Sp Eges [C(w + e; E)] max]ielluSp EM Ciwi e) in Eq. (4) is reasonable. There is harm when samples in S

---

Published as a conference paper at ICLR 2024
Loss Surface for Training Domain
Sharpness-Awee Aware Minimization Under Domain Shift
Visualization of Loss Landscape
Convergence Degree
V Gradlent voctrr
Domain
Loss surfice
Sharp
SAME
Pertrtaaioon op" ofter
Vile (iig
DISAM
ERM
DISAM
SKM Fff
update dieccion
DISAM meccaia
Domain 2
gradient convergence degres
will difterent wC"f this have differont updane
SAM
Mig Jor.3
Flat minima (guneralizable)
Domain
The neighboroood radius
(Unganeralizable)
DISAM
Os)
Domain shifts
Overall surface all damains
Figure Toy example illustrating the problem of SAM under domain shifts. Left: Domain shifts on the loss surface of training domains, which causes the inconsistency of convergence degree. Middle: Differences between SAM and DISAM in the perturbation generation and convergence. Specifically, SAM is affected by the inconsistent degree of convergence. Right: Visualization of loss landscape for ERM, SAM, and DISAM on unseen test domain. DISAM is flatter than SAM and ERM.
are intrinsically independently and identically distributed. However, this is actually ill-posed under domain shifts. Differences in the amount of data or inconsistency in task difficulty can result in biased sharpness estimation towards specific domains, hindering the overall convergence. As shown in Figure 2, neglecting the domain shifts and the consequent convergence inconsistency issue, using SAM directly at this point, significantly misdirects the perturbation direction towards the domain with the largest gradient vectors (implying a lower degree of convergence). Consequently, it does not help find better convergence path and conversely leads to suboptimal sharp minima.
3.2 DOMAIN-INSPIRG SAM
To address the problem described in Eq. (4) and Figure 2, we need to design an adjustment mechanism that takes into account the convergence degree of cach domain during the perturbation generation. Specifically, we should make the perturbation direction V Cpp(w) to efficiently pull domains that are close to convergence out of sharp minima while minimizing the negative impact domains that have not yet converged. Here, we define the convergence degree of domain i with model parameter w as Ci(w)= - L (w)). where L; represents the optimal minimum of domain (C; > 0). The design principle is to prioritize the contribution of domains with larger (:(w) to the overall perturbation direction. To achieve this, a simple approach involves directly adding (iww) the weight as of SAM with controlling coefficient term B.
M + BC,(w) (a + (w) Σ
M B(C,(w) - az CH VL((w= =
VL,(w) (5)
i=1 ++
However, we observe that the weight adjustment in Eq. (5) that is affected by the convergence degree, constrained by the magnitude of a;, That is, domains with higher a; values can tolerate lower convergence degrees, which may not accurately satisfy our goals. To refine this, we propose to use an adaptive way to ensure fairness by calculating the average convergence the domain level, namely, (www) - a; EM C.(w) + (w) - + Σ." (,(w). With this intuition, we introduce a method called Comain--nspired Sharpness-Awar Minimization (DISAM) that incorporates variance constraint between domain losses to estimate sharpness. It enables the adaptive adjustment of the perturbation direction similar to our spirit, which will provide detailed analysis in the following Eq. (8). First of all, we give the definition of the variance between different domain losses as:
M Σ X (C,( + +c)
1
Var(C.(w + +
(6)
2M
Then, putting the above variance term into the loss, the new training objective be defined as:
M
minnEees/CCneeanuuree min max |
Σ
anLi(w + c) AV. K
(7)
4

---

Published as a conference paper at ICLR 2024
Here i is without derivative taken during backpropagation, and it only makes effect in the ma loop without affecting the optimization of the first term w.r.t w. Following the computing way of perturbation in SAM, namely, using first-order Taylor expansion (Foret al., 2021), we will VLDIAA where VLDISAM W.E. has the form:
have P PTVLOISAM
M Σ
M
VLDISAM
2A
(.(w)
1
Σ
C)(1)) V..(w)
M
M
==1
VLSAM
Σ
2A M
(i(w)
Σ (y(w) VLi(w) M
1
(8)
Adaptive adjustment: incrose weights for smaller looses, redluce for larger ones.
The first term in the RHS of Eq. (8) recovers the gradient term for perturbation generation in SAM, and the second term characterizes the working mechanism for the adaptive adjustment. As can be seen, when the loss (w) of one certain domain is above the averaging level, the second term will generate a residual gradient for this domain to cancel out the gradient contribution in VLS AMM and vice versa. It means have a mild perturbation for the domain that not well optimized, and have an aggressive perturbation for the domain that is well optimized. In total, the variance constraint ensures that the perturbation location is at a more consistent point, enabling better global view of the loss landscape for gradient update. The complete algorithm is described in Appendix B. Regarding X, default value of 0.1 is relatively stable, and we provide more discussion about in Appendix B.3.
Difference and Compatibility. Similarly, the current SAM variants will meet the same challenge, they are directly applied to this scenario. Different from existing state-of-the-ee methods like GSAM (Zhuang et al., 2022) and SAGM (Wang et al., 2023b) that modify the optimization objective based on the second derivative SAM, DISAM rectifies the domain shift issue by the domain-level adjustment in the perturbation generation, which actually alleviates the negative impacts the training objective. In Appendix A.2.5, we present a table to comprehensively characterize the differ- ence between DISAM and other domain-invarien robust optimization methods. Besides, DISAM can be easily extended into other SAM-based methods to improve the generalization performance. We have provided a comparison of the similarities and differences between DISAM and general convergence consistency methods (such as V-REx(Krueger et al., 2021) and Fishr(Rame et al., 2022)) in Appendix B.I.
3.3 UNDERSTANDING DOMAIN-INSPIRED SAM
Complexity. Compared to SAM-based methods, algorithm only additionally computes the loss variance between different domains as Eq. (6) and requires no extra storing space. Therefore it has the same space complexity and the time complexity Since it only needs to additionally count the domain loss and the corresponding variance according to the domain label when calculating the empirical loss, the overall cost on Ovar is negligible.
be represented ODISAM OSAM Ovar-
Convergence. In the following, we provide the convergence analysis of SAM and DISAM. Similar to (Zhuang et al., 2022; Jiang et al., 2023), our theorem is established on assumptions that a non- convex function L(w) is L Lipschitz-smooth the lower bound of the empirical loss is bounded by Eyans and the norm of noisy stochastic gradients is bounded ( (())(( S G) at the t-step.
Theorem Consider a non-convex function L(w) with Lipschitz- smooth constant L and lower bound Lmin. With the bounded norm assumption of noisy stochastic gradients (1) (1 s G) at the t-step, the learning rate Ny Vt and a fixed perturbation amplitude p. have:
1
Σ E (W)D(11)1
(p(wn)
Lminn
(I.G2 + 1T mo log(? (T)
T
7o
VT
VT
where in SAM, r and in our DISAM, r S L.
The complete proof is presented in Appendix B. As can be seen in Theorem 1, the critical convergence difference between SAM and DISAM is on r, and especially the in our DISAM is smaller than that in SAM due to the canonically correlated perturbations during training (see the proof for the details),
5

---

Published as a conference paper at ICLR 2024
Ganeegence domais
domain
Max p PACS
Max of OfficeHomme
0.75
SAM
DEAM
8
DISAM
Accurscy
0:30
0.30
Accuricy
00.00
3006
0.1
04
SAM
0.25
SAM
0.2
02
1::55
DISAM
DISAM
60
1000
20%
3000
0000
3000
600
1000
20.0
5000
3000
5000
0.21
0.2
0.1
0.09
0.05
0100
0.34
0:13
0.07
0.06
0.05
Step
Step
(a)
(b)
(c)
(d)
Figure 3: Convergence curves and Max p search for SAM and DISAM. (a) & (b) show the trend of L(w) during the training process on PACS dataset, while (c) & (d) search for the maximum
amplitude
perturbation amplitude of SAM and DISAM on PACS and OfficeHome datasets.
which thus leads a faster convergence rate. Note that, the overall p² LI in Theorem I indicates that a larger perturbation amplitude will result in larger upper bound of convergence. However, as analyzed in SAM (Foret al., 2021), larger perturbation amplitude p has the merit of reaching a smaller upper bound on generalization error. This means that p actually has trade-off between accelerating the convergence and improving the generalization. Fortunately, when in the same overall value of 2 as DISAM enjoys a smaller I' than SAM, DISAM can permit potential larger than that in SAM, thus yielding better generalization (Please refer to Appendix B.4 for more discussion).
To verify the theoretical analysis, we present our empirical results of DISAM and SAM in Figure 3. As shown in Figures 3(a) and 3(b), the training curves on the PACS dataset show that DISAM achieves faster and steeper convergence than SAM. In addition, as DISAM has smaller I', it is able to utilize a larger perturbation amplitude p. In Figures 3(c) and 3()), we show the experimental support that DISAM allows larger values (0.2 and 0.13) than SAM (0.09 and 0.06) on PACS and OfficeHome datasets, while achieving a better performance. In total, these experiments confirm the advantage of DISAM that allows larger p for better generalization.
EXPERIMENTS
4.1 EXPERIMENT SETUPS
Datasets. We evaluate DISAM on five datasets PACS (Li al., 2017), VLCS (Fang et al., 2013) OfficeHome (Venkateswara et al., 2017), Terralncognita (Beery et al., 2018) (abbreviated as TerraIne), and DomainNet (Peng al., 2019), following the DomainBed benchmark (Gulrajani & Lopez-Paz, 2021). For fair comparison, we adhere to the training and evaluation protocol outlined in DomainBed.
strategy is used in evaluation. Specially, the unseen domain is used to evaluate the out-of-domain generalization, and the validation sets of source domains are used to measure the in-domain generalization, while the others are used for training. Final accuracy is averaged across all settings, and the performance is the averaging over three trials with distinct random seeds. Detailed statistics for each case of all datasets are provided in Appendix C.
Evaluation. The standard serre-one-domiin
Implementation details. Our backbones are ResNet50 pretrained on ImageNet (He al., 2016) and a pretrained CLIP (Radford et al.. 2021) with ViT-B/16 structure (Dosovitskiy et al., 2020), For model nyperparameters, we adopt settings in (Wang al., 2023b) for experiments using ResNet50 and in (Shu et al., 2023) for experiments using CLIP. As the default, we set the perturbation hyperparameter p to 0.05 (Wang et al., 2023b) (Fixed value during training). and the weight of the variance constraint À to 0. For detailed description of the hyparameter settings, please see Appendix C.
4.2 PERFORMANCE UNDER RESNET5O BACKBONE
We propose incorporating our domain-inspired adaptive adjustment into three SAM-based methods: SAM (Foret et al., 2021), GSAM (Zhuang et al., 2022), and SAGM (Wang et al., 2023b) on five datasets of DomainBed with ResNet50 backbone. Table shows that our Domain-Inspired SAM can mitigate issues arising from SAM's training under domain shifts, by comparing averaged in-domain and oft-of-domain performance of leading SAM methods, with and without DISAM. In-domain results show domain- domain-inspired perturbations enhance convergence, especially on the TerraInc dataset with substantial domain gaps. In Out-of-domain results, DISAM consistently improves generalization,

---

Published as a conference paper at ICLR 2024
Table Comparison with state-of-the-art domain generalization methods based ResNet50. In-domain and Out-of-domain accuracies on five datasets from DomainBed.
Algorithm
PACS
VLCCS
Offi CeHome
Terraing
Doma nNet
Avg.
In-domain results
ERM
96.6 0.2
84.6 0.4
84.2 + 0.8
93.6 + 0.3
67.1 + 1.6
85.2
SAM
9.1
84.8 0.3
85.8 0.2
88.9 0.2
68.5 - 0.1
85.1
nomain-Inspiree
9.18
84.4 0.3
86.3 = 0.2
94.8 0.2
70.2 = 0.1
86.7
GSAM
97.8
$3.9+0.2
85.9 0.2 0.2
92.1 + 0.2
69.1 +
85.8
Domain-Inspiln
85.1 0.4 0.1
86.2 0.2
94.8 0.3
70.0
86.8
SAGM
9.1
84.6 0.3
86.1 + 0.2
92.0 0.2
69.2
85.9
Nomain-Inspiree
97.9 0.1
85.0 + 0.2
86.5 + 0.3
94.9 1 0.2
70.5 1 0.1
87.0
Out-of domain results
ERM
85.5 0.2
77.3 0.4
66.5 + 0.3
46.1 1.8
43.8 +.1
63.9
CORAL (SOTA)
86.2 0.3
78.8 0.3
68.7 0.3
47.6 1.0
41.5
64.5
SAM
85.8
79.4 0.1
69.6 0.1
43.3
4..
64.5
Nomain-Inspired
87.3
80.1 0.5
70.7 0.2
47.9 0.8
45.8 558+0.2 0.2
66.4
GSAM
85.9 0.1
79.1 0.2
0.0
47.0
44.6 1: 0.2
65.1
Domain-Inspired
0.3
80.0 0.3
0.3
50.6 1.2
45.6 0.1
66.8
SAGM
86.6 1.6:-00.2 0.2
80.0 + 0.3
70.1 0.2
48.8 + 0.9
45.0 + 0.2
66.1
Homain-Inspiree
87.5 + 0.3
80.7 + 0.2
71.0 + 0.2
50.0 1.2
46.0 + 0.1
67.0
CORAL
88.4 0.3
81.2 0.4
71.7 0.2
51.7 0.3
46.3 0.2
67.9
Table 2: Comparison with state-of-the-art domain generalization methods based CLIP with ViT-B/16. Out-of-domain accuracies on five datasets from DomainBed.
Algorithm
PACS
VLCS
OfficeHon
Terra Inc
Doma nNet
Avg.
Zero-shot
96.2
81.7
82.0
33.4
57.5
70.2
CoOp
96.8
81.2
84.2
44.9
59.9
73.4
+ SAM
97.1 + 0.1
81.3 + 0.8
84.6 + 0.2
47.7 + 1.3
60.3 + 0.2
74.2
DISAM
97.2 + 0.1
81.8 0.4
84.8 + 0.2
49.5 1.2
60.6 + 0.2
74.8
ERM
96.1 0.5
83.0
83.3 0.3
0.2
5..
76.7
CLIPOOD
97.3 0.1
85.0
87.0 0.2
60.4++07
6.1
78.6
CLIPOOD
0
0.3
86. + 0.2
59.7+00.8
6..1
77.9
+ SAM
96.9 0.2
84.3 0.6
84.4 0.4
60.0 1.4
0.2
76.9
DISAM
0.1
85.6 0.2
86.6 = 0.0
61.1 1 0.7
63.6100
78.8
with average improvements of 1.9% for SAM, 1.7% for GSAM, and 1. 1.9% for SAGM. Notably, SAM performs well when the performance gap between in mn-domain and out-of-domatt is small but worse than ERM on datasets like TerraInc with large gaps, which proves our analysis SAM's shortcomings under domain shifts. This shows SAM's inconsistent convergence for large domain shifts, which DISAM addresses by incorporating domain-ings -insiied adaptive adjustments based on domain-level convergence degree. Incorporating CORAL constraints, recognized effective traditional DG method on DomainBed improves SAGM with DISAM and sets new state-of-the oftte-----hh-rrtt- results on all settings.
4.3 PERFORMANCE UNDER CLIP-BASED PRETRAINED LARGE MODEI.
The CLIP- large pretrained models (Radford et al., 2021) show great zero-shot performance but struggle with domain shifts in downstream tasks. We assess DISAM's out-of-domain results on CLIP using the experimental setup of CLIPOOD (Shu al., 2023). We test two downstream adaptation methods: CoOp (Zhou et al., 2022a), an effective prompt learning approach, and CLIPOOD, an image encoder finetuning approach for DG problems. For CoOp, generic prompt and 5000 training steps, and For CLIPOOD settings, we follow Shu et al. (2023). As
16-length learnalee
'The original results reported in Shu et al. (2023).
Our reproduced results based their official code httpp:://ggthtteeeeeeeee cww/thum/mmccccccccmccmommm..m I

---

Published as a conference paper at ICLR 2024
Table 3: Accuracy on OfficeHome and DomainNet with both domain shifts and open classes.
officelome
Doma inNet
Split
Algorithm
Avg.
A
R
C
I
P
Q
R
S
Zero-shot
86.7 75.9
89.6
92.2
72.6
51.8
65.4
13.6
83.5
67.2
72.6
CoOp"
87.3 76.7
92.2
92.5
74.6
58.2
67.9
15.0
83.7
69.9
74.4
+SAM
89.2 79.6
93.0
93.7
73.5
58.4
67.8
14.8
83.6
69.5
75.1
Base
+DISAM
88.0 80.5
92.7
92.4
75.0
59.9
68.7
14.9
84.4
70.5
75.3
CLIPOOD
88.9 79.5
92.2
94.0
76.3
58.7
69.9
17.5
85.6
72.4
76.0
+SAM
89.1 78.9
92.3
94.1
78.7
62.1
72.0
19.9
86.5
73.5
77.0
+DISAM
89.7 79.4
92.7
94.2
77.1
61.8
71.5
20.0
86.0
73.1
77.0
Zero-shot
76.8 59.7
88.7
86.4
69.7
45.0
67.0
14.3
83.9
60.8
67.4
CoOp"
73.7 56.4
86.6
85.0
69.7
47.4
67.0
15.2
82.5
61.5
66.3
+SAM
75.2 59.1
89.6
86.0
71.1
49.2
69.3
15.4
82.2
62.9
67.9
New
+DISAM
79.3 61.5
90.9
88.4
72.1
49.0
69.6
15.5
85.5
62.9
69.6
CLIPOOD'
75.2 58.6
87.5
85.8
69.3
46.4
67.2
15.2
83.2
60.6
66.9
+SAM
77.2 60.0
89.8
87.6
66.8
45.4
64.9
14.8
82.0
57.1
66.9
+DISAM
79.7 62.0
90.5
89.0
71.8
48.7
68.7
17.5
84.7
63.0
69.7
Zero-shot
82.6 67.3
88.8
89.5
71.4
47.1
66.2
13.8
83.4
63.4
69.8
CoOp"
81.4 65.7
88.9
88.8
71.9
51.3
67.4
15.1
83.1
65.1
70.1
+SAM
83.5 69.1
91.3
90.1
72.3
52.8
68.6
15.1
82.9
65.7
71.5
Total
+DISAM
84.2 70.1
91.7
90.4
73.4
53.2
69.1
15.2
85.0
65.8
72.2
CLIPOOD'
83.3 68.8
89.9
90.1
72.5
51.2
68.5
16.4
84.4
65.6
71.4
+SAM
84.2 09.2
91.0
91.0
72.3
52.0
68.4
17.3
84.3
64.0
71.8
DISAM
84.6 69.5 91.3
91.2
73.5
53.2
69.4
18.3
84.9
66.3
72.6
shown in Table 2, DISAM effectively mitigates the impact of domain shifts on model generalization during downstream task adaptation. In addition, CoOp and CLIPOOD" primarily focus rapid adaptation with limited parameters, the overfitting risk can alleviated through early stopping, resulting in the relatively marginal improvements for DISAM in Table 2. Despite this, when handling challenging tasks like TerraInc and DomainNet, our approach still exhibits substantial enhancements.
4.4 PERFORMANCE UNDER OPEN-CLAS CENERALIZATION
In this part, we evaluate the performance of our DISAM in more realistic in-the-wild setting, where both domain shifts and open-class scenarios may arise in the test domain. This setting was first proposed by Shu al. (2023). OfficeHome and DomainNet are selected to conduct related experiments because they offer an ample number of classes suitable for evaluating open-class situations. To delineate, we segregate classes within each dataset into two categories, based on the class ID. The initial half denotes the base classes, and the latter half signifies the new classes. Based on Section 4.1, eliminate the data corresponding new classes in the training domains. Due to CLIP's open vocabulary property, we can evaluate the new classes on the unseen test domain.
Inddemain tee
Sax hov
-D- EPM
-B- SAM
DS5A
Epech
Dussaf-dommin noutss bad domain C of Offinelome
Acorscy
0
As presented in Table 3, we evaluated the classification ac- curacy of "Base" classes, "New" classes, and "Total" classes in the test domain, where total classes represent the overall test domain. It revealed that the existing adaptation approach while performing well on base classes, lacks generalization on new classes during the fitting process. Our DISAM mit- igates open-class overfitting using domain-level convergence constraints, improving performance by 3.3% over CoOp and 3.1 over CLIPOOD. Figure 4 provides detailed analysis of open classes and domain shifts dimensions. ERM tends to overfit to both in-domain and base class. While SAM outperforms ERM, it struggles with sharp minima perturbations, failing to effectively escape from them. This difficulty hampers its generalization capabilities in larger models. Please refer Appendix C.6 for more discussion about DISAM and other methods for open-class generalization.
-
Figure 4: Comparison of CoOp based ERM, SAM and DISAM on accuracy curves for base/new classes, (Top: In- Domain; Bottom: Out-of-Domain)
8

---

Published as a conference paper at ICLR 2024
Uathment per PACS
Elent p Offxctoms
Wopert RiCS
6.2 0i snights Olforttoee
0.44
8.r1
6.20
g0m0
58.00
28.00
0.85
SPA
S4l4
0.8
DEAM
0.5AM
D.G.M
6-9715MM
00 35AM
0.80
-22
Se:31e-3
0.00
0.
Weight
0.000
We
(a) p on PACS.
(b) p on OfficeHomc.
(c) PACS.
(d) OfficeHome.
Figure 5: Ablation study investigating the sensitivity of hyperparameters, namely perturbation amplitude / and variance constraint weight in DISAM.
Sharpness on test domain A PNCS
Sharpness on test domain S of PNCS
Speed Compotioon Donsie-lnspeed
Ioc
100
SAM
SAM SSM
DISAM
30C
410
300
Fpoch
Fpoch
$5200
(a)
(b)
(c)
Figure 6: (a) & (b): Sharpness curves for SAM and DISAM trained on PACS dataset, which show the trend of the estimated sharpness of the model on the test domain. (c): Computation cost with and without comain-Inspired SAM used on ResNet50 and ViT-B/16 backbone.
4.5 ABLATION STUDIES
Hyperparameter sensitivity. We performed a sensitivity analysis of the perturbation amplitude p. and the variance constraint weight X, on the PACS and OfficeHome datasets. The default p and is set 0.05 (Zhuang al., 2022) and 0, respectively. As illustrated in Figure 5(a) and 5(b) within a wide range of p. DISAM consistently achieves stable and superior results compared to SAM. However, when p too large or small, experimental results worsen. Large p hiders convergence. while small weakens sharpness constraint, both affecting generalization. As for X, Figure 5(c) and 5(d) show stable results when E [0.1, 0.7]. However, larger X values increase the variance due to excessive over-conditioning weight, which can also influence the convergence.
Estimated sharpness on unseen test domain. Estimating sharpness has a high computational cost. Early methods (Dinh al., 2017b; Hochreiter & Schmidhuber, 1994b) relied on Monte Carlo sampling, but recent advancements (Jiang et al., 2023; 2020) use gradient-bases approxinations for efficiency. We assess model sharpness on unseen test domains at each epoch's end, based on the work of Jiang et al. (2023). As depicted in Figure 6(a) and 6(b), DISAM achieves much smaller gradient variance Var V (ww; B,)) than SAM during the whole training, indicating the incorporation of domain-inspired information can further reduce the sharpness of the loss surface.
Computation cost of DISAM. In Figure 6(c), show the extra computational cost from adding domain-inspired perturbation direction generation versus the original algorithm (time cost/step, batch size 32, RTX 3090 GPU). Empirical findings show DISAM integration incurs minimal overhead ( (000ls) across heaprtthm//backbnee.. linked solely to domain number and batch size, not model size, via strategic domain loss variance constraints for domain- level convergence consistency.
CONCLUSION
This paper presents Nomain-Inspiree Charpness-Aware Minimization (DISAM), an algorithm that incorporates domain-level convergence consistency into the generation of SAM's perturbations, to address the dilemma under multiple domains. DISAM mitigates SAM's bias in domain shifts that can detrimentally impact the convergence during training, yielding perturbations towards highly converged domains and limiting those in less optimized ones. This is achieved by minimizing the variance of domain loss during perturbation generation, enabling an adaptive weight adjustment for each domain based on its convergence degree, thereby enhancing the convergence across training unseen domains. Extensive experiments on the domain generalization
domains and generalization benchmarks prove DISAM's superiority over existing methods. In addition, DISAM persistents generalization capabilities under parameter-efficient fine-tuning with large models like CLIP.

---

Published as a conference paper at ICLR 2024
ETHICS STATEMENT
This paper does not raise any ethics concerns. This study does not involve any human subjects, practices to data set releases, potentially harmful insights, methodologies and applications, potential conflicts of interest and sponsorship, bnberimination/biasfaaana concerns, privacy and security issues, legal compliance, and research integrity issues.
REPRODUCIBILITY STATEMENT
All experiments were conducted using NVIDIA GeForce RTX 3090 GPU, Python 3.9.15, Pytorch 1.12.1, 1, and clip 1.0. Further details regarding experimental setups and implementation can be found in Section 4.1 and Appendix C, while theoretical proofs are provided in Appendix B. The principal code for implementing Domain- Inspiredd SAM is presented in Appendix D.
ACKNOWLEDGMENTS
This work is supported by the National Key R&D Program of China (No. 20222D0160702), STCSM (No. 22511 1 106101, No. 18D22270700, No. 21DZ1 100000000,.0000000000000000)) 111 plan (No. BPO719010), and State Key Laboratory of UHD Video and Audio Production and Presentation. Ruipeng Zhang and Ziqing Fan are partially supported by Wu Wen Jun Honorary Doctoral Scholarship, Al Institute, Shanghai Jiao Tong University.
REFERENCES
Martin Arjovsky, Léon Bottou, Ishaan Gulrajani, and David Lopez-Paz. Invariant risk minimization. arXiv preprint arXiv:190.028893, 2019. 17
Yogesh Balaji et al. Metareg: Towards domain generalization using meta-regularization. In NeurIPS, pp.
2018. 17
Sara Beery, Grant Van Horn, and Pietro Perona. Recognition in terra incognita. In Proceedings of the European conference on computer vision (ECCV), pp. 456-473, 2018. 6, 22
Shai Ben-David, John Blitzer, Koby Crammer, Alex Kulesza, Fernando Pereira, and Jennifer Wortman Vaughan. A theory of learning from different domains. Machine learning, 79:151-175, 2010.
Fabio Maria Carlucei, Paolo Russo, Tatiana Tommasi, and Barbara Caputo. Hallucinating agnostic images to generalize across domains. In 2019 IEEE/CVE International Conference on Computer Vision Workshop (ICCVW), pp. 3227-3234. IEEE, 2019. 17
Junbum Cha, Sanghyuk Chun, Kyungjae Lee, Han-Cheol Cho, Seunghyun Park, Yunsung Lee, and Sungrae Park. Swad: Domain generalization by seeking flat minima. Advances in Neural Information Processing Systems, 34:22405-2244 2021. 17, 18
Woong-GG Gi Chang, Tackgeun You, Seonguk Seo, Suha Kwak, and Bohyung Han. Domain-specific batch normalization for unsupervised domain adaptation. In Proceedings of the IEEF/CVF
conference on Computer Vision and Pattern Recognition, pp. 7354-7362, 2019. 17
Pratik Chaudhari. Anna Choromanska, Stefano Soatto, Yann LeCun, Carlo Baldassi, Christian Borgs, Jennifer Chayes, Levent Sagun, and Riccardo Zecchina. Entropy-sgd: Biasing gradient descent into wide valleys. Journal of Statistical Mechanics: Theory and Experiment, 2019112):124018, 2019. 3,3
Laurent Dinh, Razvan Pascanu, Samy Bengio, and Yoshua Bengio. Sharp minima can generalize for deep nets. In International Conference on Machine Learning, pp. 1019-1028. PMLR, 2017a. 24
Laurent Dinh, Razvan Pascanu, Samy Bengio, and Yoshua Bengio. Sharp minima can generalize for deep nets. In International Conference on Machine Learning, pp. 1019-1028, PMLR, 2017b. 1, 3, 9, 16, 24
10

---

Published as a conference paper at ICLR 2024
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghami. Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010. / 1929, 2020. 1,6
Qi Dou, Daniel Coelho de Castro, Konstantinos Kamnitsas, and Ben Glocker. Domain generalization via model-agnostic learning of semantic features. In NewrIPS, pp. 6450-6461, 2019. 17
Jiawei Du, Hanshu Yan, Jiashi Feng, Joey Tianyi Zhou, Liangli Zhen, Rick Siow Mong Goh, and Vincent Tan. Efficient sharpness-aware minimization for improved training of neural networks, In International Conference on Learning Representations, 2022a. 1, 16
Jiawei Du, Daquan Zhou, Jiashi Feng, Vincent Tan, and Joey Tianyi Zhou. Sharpness-aware training for free. Advances in Neural Information Processing Systems, 01223339-23455 2022b. 1, 16
Gintare Karolina Dziugaite and Daniel M. Roy. Computing nonvacuous generalization bounds for deep (stochastic) neural networks with many more parameters than training data. In Proceedings of the 33rd Annual Conference Uncertainty in Artificial Intelligence (UAI), 2017. 1
Chen Fang, Ye Xu, and Daniel N Rockmore. Unbiased metric learning: On the utilization of multiple datasets and web images for softening bias. In Proceedings of the IEEE International Conference Computer Vision, pp. 1657-1664, 2013. 6, 22
Chelsea Finn, Pieter Abbeel, and Sergey Levine. Model-agmostic meta-learning for fast adaptation of deep networks. In International conference on machine learning, pp. 1126-11355 135. PMLR, 2017. 17
Pierre Foret, Ariel Kleiner, Hossein Mobahi, and Behnam Neyshabur. Sharpness-aware minimization for efficiently improving generalization. In International Conference on Learning Representations, 2021. 1, 3, 6, 16, 17, 18, 21
Ishaan Gulrajani and David Lopez-Paz. In search of lost domain generalization. In International Conference on Learning Representations, 2021. URL https: opwnreveeeeeeeeeeeeeeeeeee net/ forum? id=1@dxeXDoWtL 6, 22
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 770-778, 2016. 1,6
Sepp Hochreiter and Jürgen Schmidhuber. Simplifying neural nets by discovering flat minima. Advances in neural information processing systems, 7, 1994a. 16
Sepp Hochreiter and Jürgen Schmidhuber. Simplifying neural nets by discovering flat minima. Advances in neural information processing systems, 7, 1994b. 9, 24
Pavel Izmailov, Dmitrii Podoprikhin, Timur Garipov, Dmitry Vetrov, and Andrew Gordon Wilson. Av- eraging weights leads to wider optima and better generalization. arXiv preprint 2018. 17, 18
Weisen Jiang, Hansi Yang, Yu Zhang, and James Kwok. An adaptive policy to employ sharpness- aware minimization. In The Eleventh International Conference on Learning Representations, 2023. URL
net forum? id= 6W17 M2BC-. 1, 24
Yiding Jiang, Behnam Neyshabur, Hossein Mobahi, Dilip Krishnan, and Samy Bengio. Fantas- tic generalization measures and where to find them. In International Conference on Learning Representations, 2020. URL https: 9, 24
https::/ hpt///oeeevveeeeeeeeeee/eee.e net./ /t rorum? www.mmid==sggiioiiie d=STg
1,
Xin Jin, Cuiling Lan, Wenjun Zeng. and Zhibo Chen. Style normalization and restitution for domain generalization and adaptation. IEEE Transactions on Multimedia, 24:3636-363 2022. doi: 10. 10.1109//MM.22221333330
17
Nitish Shirish Keskar, Dheevatsa Mudigere, Jorge Nocedal, Mikhail Smelyanskiy, and Ping Tak Peter Tang. On large-batch training for deep learning: Generalization gap and sharp minima. In International Conference Learning Representations, 2017a. 16
11

---

Published as a conference paper at ICLR 2024
Nitish Shirish Keskar, Dheevatsa Mudigere, Jorge Nocedal, Mikhail Smelyanskiy, and Ping Tak Peter Tang. On large-batch training for deep learning: Generalization gap and sharp minima. In International Conference Learning Representations, 2017b. URL httpt:t/ /RLLttps///ppeeeeieer net/ met//foree forum? oyR Ygg. 1, 24
Dachee Kim, Youngjun Yoo, Seunghyun Park, Jinkyu Kim, and Jackoo Lee. Selfreg: Self-supervised contrastive regularization for domain generalization. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 9619-9628, 2021. 17
Minyoung Kim, Da Li, Shell X Hu, and Timothy Hospedales. Fisher sam: Information geometry and sharpness aware minimisation. In International Conference on Machine Learning, pp. 11148 11161. PMLR, 2022. 16
David Krueger, Ethan Caballero, Joern-Henrik Jacobsen, Amy Zhang, Jonathan Binas, Dinghuai Zhang, Remi Le Priol, and Aaron Courville. -ut-of-distribution generalization via risk extrapola- tion (rex). In International Conference on Machine Learning, pp. 5815-5826. PMLR, 2021. 2, 5, 17, 18
Jungmin Kwon, Jeongseop Kim, Hyunseo Park, and In Kwon Choi. Asam: Adaptive sharpness aware minimization for scale-invariant learning of deep neural networks. In International Conference on Machine Learning, pp. 5905-5914, PMLR, 2021. 16
Bingcong Li and Georgios B Giannakis. Enhancing sharpness-aware optimization through variance suppression. arXiv preprint arXiv:2309. /5639, 2023. 1, 16, 17
Da Li, Yongxin Yang. Yi-Zhe Song, and Timothy M Hospedales. Deeper, broader and artier domain generalization. In Proceedings of the IEEE international conference on computer vision, pp. 5542-55500 2017. 6, 22
Da Li, Yongxin Yang, Yi-Zhe Song, and Timothy M. Hospedales. Learning to generalize: Meta- learning for domain generalization. AAAI, 2018a. 17
Hao Li, Zheng Xu, Gavin Taylor, Christoph Studer, and Tom Goldstein. Visualizing the loss landscape of neural nets. Advances neural information processing systems, 31, 2018b.
Haoliang Li, Sinno Jialin Pan, Shiqi Wang, and Alex C Kot. Domain generalization with adversarial feature learning. In CVPR, pp. 5400-5409, 2018c. 16
Ya Li, Xinmei Tian, Mingming Gong, Yajing Liu, Tongliang Liu, Kun Zhang, and Dacheng Tao. Deep domain generalization via conditional invariant adversarial networks. In Proceedings of the European conference on computer vision (ECCV), pp. 024-639, 2d.
Yiying Li, Yongxin Yang, Wei Zhou, and Timothy Hospedales. Feature-crice networks for heteroge- neous domain generalization. In International Conference on Machine Learning, pp. 3915-3924944 PMLR, 2019. 17
Yajing Liu, Zhiwei Xiong, Ya Li, Xinmei Tian, and Zheng-Jun Zha. Domain generalization via encoding and resampling unified latent space. IEEE Transactions on Multimedia, 25:126-139, 2023. doi: 10.1 10.1199/MM..2211.31 31211664 17
Yong Liu, Siqi Mai, Xiangning Chen, Cho-Jui Hsieh, and Yang You. Towards efficient and scalable sharpness-awan minimization. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 12360-12370, 2022. 1, 16
David A McAllester. Pac-bayesian model averaging. In Proceedings of the twelfth annual conference
on Computational learning theory, pp. 164-170 1999. 3
Peng Mi, Li Shen, Tianhe Ren, Yiyi Zhou, Xiaoshuai Sun, Rongrong Ji, and Dacheng Tao. Make sharpness aware minimization stronger: A sparsified perturbation approach. In Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho (eds.), Advances in Neural Information Processing Systems, 2022. 16
12

---

Published as a conference paper at ICLR 2024
Saeid Motiian, Marco Piccirilli, Donald A Adjeroh, and Gianfranco Doretto. Unified deep supervised domain adaptation and generalization. Proceedings of the IEEE international conference on computer vision, pp. 11-557155577 5725, 2017. 17
Ziwei Niu, Junkun Yuan, Xu Ma, Yingying Xu, Jing Liu, Yen Wei Chen, Ruofeng Tong, and representation learning for domain
Lanfen Lin. Knowledge distillation-bat based generalization. IEEE Transactions on Multimedia, pp. 1-11, 2023. doi: 10.1109/TMM.200 3263549. 17
Matthew D Norton and Johannes O Royset. Diametrical risk minimization: Theory and computations. Machine Learning, pp. 1-19, 2021. 17
Xingchao Peng, Qinxun Bai, Xide Xia, Zijun Huang, Kate Saenko, and Bo Wang. Moment matching for multi-source domain adaptation. In Proceedings of the IEEE/CVF international conference on computer vision, pp. 1406-1415, 2019. 6, 22
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pp. 8748-8763. PMLR, 2021. 7,
Alexandre Rame, Corentin Dancette, and Matthieu Cord. Fishr: Invariant gradient variances for out- of-distribution generalization. In International Conference on Machine Learning, pp. 18347-18377.
PMLR, 2022. 5, 17, 18
Sconguk Seo, Yumin Suh, Dongwan Kim, Geeho Kim, Jongwoo Han, and Bohyung Han. Learning to optimize domain specific normalization for domain generalization. In Computer Vision-ECCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part XXII 16,
pp. 68 83. Springer, 2020. 17
Rui Shao, Xiangyuan Lan, Jiawei Li, and Pong C Yuen. Multi-adversarial discriminative deep domain generalization for face presentation attack detection. In CVPR, pp. 1003111031,, 2019. 16
Yuge Shi, Jeffrey Seely, Philip HS Torr, N Siddharth, Awni Hannun, Nicolas Usunier, and Gabriel Synnaeve. Gradient matching for domain generalization. arXiv preprint arXiv:2104.09937 2021. 17, 18
Yang Shu, Xingzhuo Guo, Jialong Wu, Ximei Wang, Jianmin Wang, and Mingsheng Long, Clipood: Generalizing clip to out-of-distributions. In International Conference on Machine Learning, 2023. 6, 7, 22
Baochen Sun and Kate Saenko. Deep coral: Correlation alignment for deep domain adaptation. In ECCV, pp. 443-450. Springer, 2016. 16
Hemanth Venkateswara, Jose Eusebio, Shayok Chakraborty, and Sethuraman Panchanathan. Deep hashing network for unsupervised domain adaptation. Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 5018-5027, 2017. 6, 22
Jindong Wang, Cuiling Lan, Chang Liu, Yidong Ouyang, Tao Qin, Wang Lu, Yiqiang Chen, Wenjun Zeng, and Philip S. Yu. Generalizing to unseen domains: A survey on domain generalization. IEEE Transactions on Knowledge and Data Engineering. 35(8):8052 2023a. doi: 10. 109/ TK2E..202.3117112. 16
Pengfei Wang, Zhaoxiang Zhang, Zhen Lei, and Lei Zhang. Sharpness-aware gradient matching for domain generalization. In Proceedings of the JEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 3769-3778. 2023b. 1, 6, 16, 17,22 22
Shujun Wang, Lequan Yu, Caizi Li, Chi-Wing Fu, and Pheng-Amn Heng. Learning from extrinsic and intrinsic supervisions for domain generalization. In ECCV, 2020. 17
Qinwei Xu, Ruipeng Zhang. Ya Zhang, Yanfeng Wang, and Qi Tian. A fourier-based framework for domain generalization. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 1383-14392. 2021. 17
13

---

Published as a conference paper at ICLR 2024
Qinwei Xu, Ruipeng Zhang, Ziqing Fan, Yanfeng Wang, Ya- Wu, and Ya Zhang. Fourier-basee augmentation with applications to domain generalization. Pattern Recognition, 139:109474, 2023a.
17
Qinwei Xu, Ruipeng Zhang, Yi-Yan Wu, Ya Zhang, Ning Liu, and Yanfeng Wang. Simde: A simple domain expansion approach for single-source domain generalization. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 4797-4807, 2023b. 17
Qinwei Xu, Ruipeng Zhang. Ya Zhang, Yi-Yan Wu, and Yanfeng Wang. Federated adversarial domain hallucination for privacy-preserving domain generalization. IEEE Transactions Multimedia, pp. 1-13, 2023c. doi: 10. (0009//MM.2023.33255
17
Shen Yan, Huan Song, Nanxiang Li, Lincan Zou, and Liu Ren. Improve unsupervised domain adaptation with mixup training. arXiv preprint arXiv:2001.00677, 2020. 17
Lei Zhang, Yingjun Du, Jiayi Shen, and Xiantong Zhen. Learning learn with variational inference for cross-domain image classification. IEEE Transactions on Multimedia, 25:3319-3322 2023a. doi: 10. 1.1100/TMM.222231115
17
Ruipeng Zhang, Qinwei Xu, Chaoqin Huang, Ya Zhang, and Yanfeng Wang. Semi-supervised domain generalization for medical image analysis. In 2022 IEEE 19th International Symposium on Biomedical Imaging (ISBI), pp. 1-5, IEEE, 2022. 17
Ruipeng Zhang, Qinwei Xu, Jiangchao Yao, Ya Zhang, Qi Tian, and Yanfeng Wang. Federated domain generalization with generalization adjustment. In Proceedings the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 3954-3963, 2023b. 17
Xingxuan Zhang, Renzhe Xu, Han Yu, Hao Zou, and Peng Cui. Gradient norm aware minimization seeks first-order flatness and improves generalization. Proceedings of the IEEE/CVE Conference on Computer Vision and Pattern Recognition, pp. 20247-20257, 2023c. 16
Yang Zhao, Hao Zhang, and Xiuyuan Hu. Penalizing gradient norm for efficiently improving generalization in deep learning. In International Conference on Machine Learning, pp. 26982 26992. PMLR, 2022. 1, 16
Kaiyang Zhou, Yongxin Yang, Timothy Hospedales, and Tao Xiang. Deep domain-adversarial image generation for domain generalisation. In Proceedings of the AAAI conference on artificial intelligence, volume 34, pp. 13025- 13032, 2020a. 17
Kaiyang Zhou, Yongxin Yang, Timothy Hospedales, and Tao Xiang. Learning to generate novel domains for domain generalization. In Computer Vision-ECCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings. Part XVI 16, pp. 561-578. Springer, 2020b. 17
Kaiyang Zhou, Yongxin Yang. Yu Qiao, and Tao Xiang. Domain generalization with mixstyle. arXiv preprint arXiv:2104 10.000000....... 2021. 17
Kaiyang Zhou, Jingkang Yang, Chen Change Loy, and Ziwei Liu. Learning prompt for vision- language models. International Journal of Computer Vision, 130(9):2337-2343 2022a. 7
Zhihan Zhou, Jiangchao Yao, Yan-Feng Wang, Bo Han, and Ya Zhang. Contrastive learning with boosted memorization. In International Conference on Machine Learning, pp. 27367-27377.
PMLR, 2022b. 17
Zhihan Zhou, Jiangchao Yao, Feng Hong, Ya Zhang, Bo Han, and Yanfeng Wang. Combating representation learning disparity with geometric harmonization. In Thirty-seventh Conference on Neural Information Processing Systems, 2023. 17
Juntang Zhuang, Boqing Gong, Liangzhe Yuan, Yin Cui, Hartwig Adam, Nicha C Dvorekk, sekhar tatikonda, James Duncan, and Ting Liu. Surrogate gap minimization improves sharpness-awan training. In International Conference on Learning Representations, 2022. 1, 5, 6, 9, 16
14

---

Published as a conference paper at ICLR 2024
Appendix
Table of Contents
A Related Work
16
A.1 Sharpness- Aware Minimization (SAM)
16
Domain Generalization
B Details of DISAM
18
B.1
Comparative Analysis of DISAM Versus General Convergence Consistency
18
B.2 Algorithm of DISAM
19
B.3 Proof of DISAM's Convergence
19
B.4 Discussion of the role of p in DISAM
21
Detailed Experiments
22
C.I Detailed Experiment Setups
22
C.2 Detailed Experimental Results
22
C,3 Details about Estimated Sharpness Unseen Test Domain
24
C,4 Details about Comparison of Computation Cost
24
C.5 Details about Convergence Curves of SAM and ERM
BE..........
24
Detailed Analysis about Open-Class Generalization
................................
25
Detailed Analysis of Convergence Speed Comparison
................................
25
D Pseudo code of DISAM
26
15

---

Published as a conference paper at ICLR 2024
A
RELATED WORK
A.1
SHARPNESS-AWAR MINIMIZATION (SAM)
Numerous studies (Hochreiter Schmidhuber, 1994a; Li et al., 2018b; Dinh et al., 2017b) have been conducted to enhance our understanding of the generalization capabilities of deep learning models through an exploration of the geometric properties of the loss landscape. These investigations have consistently demonstrated that deep neural networks exhibiting flat minimum tend to exhibit superior generalization performance. In order to obtain a flat minimum, the Sharpness-Awar (SAM) approach (Foret et al., 2021) minimize both the vanilla training loss and the sharpness metric. The sharpness metric, as defined by (Keskar et al., 2017a), quantifies the flatness of a minimum through the eigenvalues of the Hessian matrix. In practice, SAM involves obtaining fixed-length perturbation through gradient ascent on the initial parameter, followed by updating the gradient based on this perturbed parameter with respect to the initial parameter. Although SAM can result in a flat minimum and substantially enhance the generalization capability, it incurs a twofold increase computational overhead. The variants of SAM have been extensively investigated from two perspectives: the first pertains to the enhancement of SAM's generalizability (Kwon et al., 2021; Zhuang et al., 2022; Zhang et al., 2023c; Zhao et al., 2022; Wang et al., 2023b), while the second focuses on improving its efficiency (Liu et al., 2022; Du ct al., 2022a;b; Mi et al., 2022).
Minimization proposed, which utilizes base optimizer simultaneously
A.1.1 GENIRALIZABILITY IMPROVEMENT OF SAM
One key problem of SAM is that the perturbation obtained by gradient ascent might disagree with sharpness since gradient ascent is only first-order approximation of the sharpness calculation. Zhuang et al. (2022) introduced a surrogate gap to enhance the evaluation of sharpness, while (Wang al., 2023b) integrated the perturbed loss and the surrogate gap from (Zhuang et al.. 2022) into unified objective. Additionally, (Zhao et al., 2022) revealed that SAM inherently optimizes both the empirical risk loss and the corresponding gradient norm. Besides, FisherSAM (Kim et al., 2022) and ASAM (Kwon et al., 2021) achieved improved perturbations by adjusting the scales of the perturbation magnitudes. (Zhang et al., 2023c) further proposed Gradient norm Aware Minimization (GAM), which regularized the Hessian of the gradient norm. VaSSO (Li & Giannakis, 2023) focuses on addressing the issue of SAM's subpar performance in perturbation direction generation due to the noise introduced by bini-batch sampling.
A.1.2 EFFICIENCY IMPROVEMENT OF SAM
Due the doubled overhead of SAM in comparison Gradient Descent), considerable efforts have been devoted to mitigating this overhead. (Liu et al., 2022) introduced LookSAM means to reduce the number of perturbations. Meanwhile, (Mi et al., 2022) achieved sparse perturbations through the use of a binary mask. Furthermore, Du et al al.explored various proxy methods (ESAM (Du al., 2022a), SAF (Du et al., 2022b)) for computing perturbations, thereby replacing the gradient ascent derivation process employed in SAM.
base optimizer like SGD (Stochastic
A.2 DOMAIN GENERALIZATION
Domain generalization is a vital research direction that focuses on training models capable of generalizing well unseen domains by leveraging knowledge from multiple source domains (Wang et al., 2023a). Over the past decade, several methods have been proposed to address the challenges of domain generalization. These methods can be broadly categorized into five main approaches: domain alignment, meta-learning, domain hallucination, domain disentanglement, and robustness training. In this section, we provide a brief overview of each of these categories.
A.2.1 DOMAIN SAIGGNENTT-BSSEE METHOD
The goal of domain alignment is to mitigate discrepancies among distinct source domains by aligning the marginal feature distributions extract -nvariint representations. This objective can be accomplished using various strategies, including adversarial training (Li et al., 2018d: Shao et al., 2019), maximum mean discrepancy (Li et al., 2018c), moment matching (Sun & Saenko, 2016),
16

---

Published as a conference paper at ICLR 2024
self supervvie learning (Wang al., 2020), contrastive learning (Kim et al., 2021; Motiian et al., 2017; Zhou et al., 2022b; 2023). All these methods improve generalization across unseen domains by either directly or indirectly reducing the discrepancy between different feature distributions and imposing domain-invariant constraints on these discriminative features.
A.2.2 META LEARNING-BASED METHODS
These approaches aim to address unforeseen domain shifts and enhance the generalizability of models to such shifts through oetta-optiizztion,, achieved by partitioning the training domains into distinct meta-train and meta-test domains. (Li et al., 201 8a) first introduced meta learning into DG, following the concept of Modal-Agnostic Meta-Learnning (MAML) (Finn et al., 2017). Subsequently, (Balaji al., 2018 8) designed weight regularizer based the meta-learning framework, while (Li et al., 2019) chose to meta-learn feature critic loss. (Dou et al., 2019) constrained the invariance of learned semantic relations between the meta-train and meta-test domains. Additionally, (Zhang et al., 2023a) integrated meta learning into Bayesian framework and enforced the model to learn meta-variational distribution to enhance knowledge transfer.
A.2.3 DOMAIN HALLUCINATION -AASAA-SEN-----D METHODS
Domain hallucination, also known as data augmentation in the presence of domain shifts, aims to encompass a wider range of domain variations by generating additional training samples from fictional domains while preserving their semantic integrity. Early approaches such as (Xu et al., 2021; 2023a; Zhang et al., 2022; Xu et al., 2023b; Zhou et al., 2020a; Yan al., 2020; Zhou et al., 2020b; Carlucci et al., 2019; Xu et al., 2023c) involve cross- doosa- data augmentation in the input space and can be categorized into non-parametrie and adversarial sample-based approaches. Non-parametric methods (Xu et al., 2021; 2023a; Yan et al., 2020; Zhang et al., 2022) employ traditional image transformations to achieve enhancement, while adversarial sample-based methods (Xu et al., 2023b; Zhou al., 2020a;b; Carlucci et al., 2019; Xu et al., 2023c) generate samples from new domain through adversarial training. Adversarial training ensures the quality of generation by enforcing consistency in terms of category among the samples from the generated fictional domain. Some recent work focuses on augmentation in the latent space (Liu et al., 2023; Zhou et al., 2021), which achieves more efficient augmentation perturbations by applying perturbations the latent features to improve the generalization of the model.
A.2.4 DOMAIN DISENTANGLEMENT-BASSE METHODS
In contrast to the majority of domain generalization approaches that aim for domain invariance, disentanglement-bann based approaches focus on separating the domain-inven invariant and domain-specifie components. To achieve this, Seo et al. (2020) introduced domain-specific batch normalization (Chang et al., 2019) for each training domain, effectively balancing feature discrimination and invariance. In a similar vein,Jin al. (2022) designed a style restitution module that encourages the separation of task- relevant and task- iireeeaeeevant features. Furthermore, Niu et al. (2023) proposed a two-stage distillation approach, aimed learning domain-invante representation while preserving domain-spectie -specffcc features.
A.2.5 ROBUSTNESS TRAINING- TRAINING-BASED METHODS
The objective of robustness training-based methods is incorporate constraints that enhance the model's robustness or flatness during the training process. Robustness-related methods aim to learn domain-invariant representations by employing technique known as Invariant Risk Minimization (IRM) (Arjovsky et al., 2019). By minimizing the risk across different domains, these methods (Ar- jovsky et al., 2019; Krueger et al., 2021; Norton & Royset, 2021; Shi et al., 2021; Rame al., 2022; Li & Giannakis, 2023) seek to learn features that are insensitive to domain variations, thereby improving the model's ability to generalize. On the other hand, a separate class of flatness-related methods (Izmailov et al., 2018; Cha et al., 2021; Zhang al., 2023b; Foret et al., 2021; Wang et al., 2023b) aims to address the effects of domain shifts by identifying flat minima. These methods strive to find regions in the loss landscape where small perturbations in the input have minimal impact on the model's 's predictions. By leveraging flat minima, these methods enhance the model's ''s robustness to domain variations.
17

---

Published as a conference paper at ICLR 2024
Table 4: Comparison of SAM-based methods and other robustness training-based methods on the optimization objective.
Method
Total Optiniization Function
Optimization on
Optimization
(.))
Σ (or) (W
to
Fish
min
MCM (JeRk.M)
VeL(m)- VCy(m)
Same to left
min
(cL)(w) 1x1 [] ve(uo))
Same
min DUKS
outif+++))
THICE
CaCifor
DISAM
IIT IBAX
aaLift + Cati +c)ya
IAX
C.CC(w +) (Nar(S(((
In Table 4, we provide the comparison of optimization objectives between representative algorithms in the two categories. Domain-invariant methods solely concentrate on optimizing the parameter w. For instance, V-REx (Krueger et al., 2021) directly minimizes the variance of the domain loss, which can have a detrimental effect on convergence. Similarly, Fish (Shi al., 2021) and Fishr (Rame et al., 2022) impose constraints on gradient updates. SAM-based methods require the estimation of sharpness, so in addition to optimizing the parameters w, they also need to optimize the perturbation directions 6.
This paper primarily concentrates on the batness-based method, which encompasses two main approaches for enhancing the flatness of the model. The first approach involves leveraging the self-ensemble of multiple minima attained during the training process to passively acquire result that favors flatness minima. Notable examples of this approach include Stochastic Weight Averaging (SWA) (Izmailov et al., 2018) and Stochastic Weight Averaging Densely (SWAD) (Cha et al., 2021). The second approach involves directly optimizing for flatness and is referred Sharpness- Aware Minimization (SAM) (Foret et al., 2021). In the subsequent section, we will delve into a comprehensive review of the relevant literature pertaining to these approaches.
B DETAILS OF DISAM
B.1 COMPARATIVE ANALYSIS OF DISAM VERSUS GENERAL CONVERGENCE CONSISTENCY
Here, we present a thorough examination of the distinctions between our proposed DISAM framework and the broader, conventional convergence consistency issue like V-REx et al., 2021) and Fishr al., 2022). Specifically, we address the following aspects:
Distinct Focus: DISAM focuses on the issue where SAM- based methods are unable to accurately estimate sharpness in domain shift scenarios, leading to the ineffective sharpness minimization and reduction in generalization performance.
Enhancing on Top of General Methods: While traditional Solutions(Krueger et al., 2021; Rame et al., 2022; Shi et al., 2021) aim convergence consistency in parameter opti- mization, DISAM's methodology is distinct and orthogonal. It builds upon methods like V-REx(Krueger et al., 2021) and Fishr(Rame al., 2022), but goes further in enhancing out-of-domain generalization through better sharpness minimization. This is evident in our experiments, where combining DISAM with Fishr results in significant performance gains (shown in Table 5).
We also provide extensive experimental results validate DISAM's effectiveness and its practical implications various domain-shie shift scenarios.
It is imperative reiterate the contributions of our DISAM. We provide detailed exposition of how simplistic applications of SAM compromise training robustness, especially when dealing with domain shifts. DISAM strategically mitigates these issues by finely tuning the perturbation vectors and their
18

---

Published as a conference paper at ICLR 2024
Table 5: Comparison with other general convergence consistency methods.
Algorithm
PACS
VLCS
OfficeHome
Terra Inc
Doma nNet
Avg.
V-REx
84.9
78.3
66.4
46.4
33.6
61.9
V-REx + DISAM
85.8
78.4
70.5
45.9
42.3
64.6
Fishr
86.9
78.2
68.2
53.6
41.8
65.7
Fishr + DISAM
87.5
79.2
70.7
54.8
43.9
67.2
Algorithm Domain Domain-Inspired Sharpness- Minimization (DISAM).
Input: Source Domains S = {Di, DMr. initial model wi, perturbation ratio variance constraint weight learning rate ne training iterations T'
Output: Generalization model wy.
for in 1 do
2: Sample mini-batch B = (BI. BBr S S, where B; S D; and [, 2 0.
3: Compute the domain-inspired loss gradient:
VLDr B) = VL(w;; B) X V Var ( B; Bijiz
4: Get the perturbation weight: w
wi [[Dr("M
5: Update weights: W1+1 = Wi 17. V ((u B)}.
6:
end for
location points, thus significantly enhancing model generalization. Furthermore, we underscore the notable enhancements achieved with DISAM, as corroborated by comprehensive experimental analyses and the ensuing performance metrics.
B.2 ALGGRRIHHM OF DISAM
We give specific algorithmic details for DISAM in Algorithm 1, and the python code implementation is in Appendix D.
B.3 PROOF OF DISAM'S CONVERGENCE
Theorem (Convergence During Training). Consider non-convex function L(w) with Lipschitz- smooth consiant I. and lower bound fomin- With the bounded norm assumption of noisy stochastic gradients (l[V (w)ll2 I G) at the t-step. the learning rate т)t = 170/ /t and a fixed perturbation amplitude p, we have:
Σ EEVE, Cp(wo)
Lmin
(LG2 + LL mo log(T)
170
VT
VT
where in SAM, r L and when use DISAM r L.
Proof. For simplicity of notation, denote the update step as de - where π) is the decayed learning rate and sh is the expected gradient of perturbation loss Cp. By T-smoothness of the loss function L and the definition of (p(w)) L(wf*0), where wase represents the parameters after the perturbation of gradient ascent, we have:
1)(w++1 ((wfft) < L(w(*c)= + (VL(wf>)), wass w.*C) L asc w 1
(10)
where L is the Lipschitz constant of loss and with the definition of W - wi and wase wi + we have:
Cp(we-1)+ S C(wf(e) + (VL(w(fC), W.41 Ex+1 w; =e) + L || + Ex+1 Wt 1
I L(wfse) + (VL(w(5)) ). de) + L| + (VL(w (eall=++V[((ufff])
e,41 4e) + L||e41 - eell
(11)
19

---

Published as a conference paper at ICLR 2024
Let us take the expectation conditioned on observations up to step t. For the sake of simplicity, the symbol to denote the expectation over all possible data points on the training data distribution.
Moreover, given the observations up step f, we can use the definition of dg to obtain:
B(Lp(u++1) & ((we No ( EB 11 LE [1 19 11
+ Cr+ Et) + LE [x+1 - Eell
(12)
$ (w 7i V ( _ )13 11 + np LG
+ LVVL( c1+1 - Ex) + LE cx+1 - Eell
By the definition of 6,, we have:
g(i)
g(tt1)
c = || C+1 = |1g(++111
(13)
where (¹) is the gradient of L at w, with the domain-inspired gradient in Eq. 8). We denote 2A M (E' M ΣØ= (2 Since both Et and Ex+1 are unit length vectors, Ec+1 E, can be bounded by the are length Φe between them. Here the difference vector between 6++1 and e, can be regarded as a random noise in the gradient direction and in SAM, p > Tt. so the expectation of the inner product with the gradient direction ((wf=))) can be approximated and its gradient direction can be considered approximately independent of the gradient direction in the neighborhood of wy. its difference with the inner product between 6 and e, is negligible). Therefore, we have:
VLa(w)) = E V where Bj
(wfe is updated from we with a larger step size p.
B[cp(we+1] S [(w)*0) El V ( Wt
a80 112 + n LG
g g
(())
|
+ + 600) (+11 (V (00) ),,,)] 1 + LppEE
(14)
[|g(+)1|||
|
S L(ww) E VL + ni LG + Lp²o?
Because of the continuity of the optimization, the angle between the gradient perturbations before and after is small. Therefore,
get:
|e141 sell
= [V - V (w)ill + +
Φe. Rs tan th
+ +003)
ccll
[VLa(W)))
DM (st V (L((w(+1) VL (wx)) +0(03)
(15)
ME M ((344 VL) + (V VLC + 0
[La(w,))
Here we consider the effect of the weight coefficients generated by DISAM in the perturbation of LL, for the part of ('(ww) that is large, B is smaller, we assume that the larger L' ('(w,) is, the larger the corresponding gradient LL (w:) is also, and after one optimization process, the variability between the domains will be reduced, so B4 is a little bit smaller than the weight of 8t. in the place where the gradient is large, and by the rearranging incquality, we can obtained:
M
M
Σ
Bis VL (w++1) S
3
(wz)
(16)
So bring Eq.( 16) to Eq.( 15), and with vL(w++) = VL(w: + d+) = vL(wx) + Hd, O(Ild.19)
we can get:
M (Bt
V.) + Hd. + 13)11 + + S m&
Φ: <
(17)
20

---

Published as a conference paper at ICLR 2024
Here since - -B)0--8))--00 (wwc))) O, we use I' to denote upper bound that is smaller than L.
Plug Eq.( 17) into Eq.( 14), have:
B[Cp(we+1]] s L(ugsc) T/ (L n T. Lpprn
(18)
Perform telescope and note that y/ V have:
T
[Cp(WT) - Ep(un) -
[ + (LG2 + + LI
Σ
1
i
(19)
m B. V[ (LG + Ln
Hence,
V VV((w
112 -
Th VL( 2 Cp(wo) - im + (LG² + LT log(T)
(20)
Note that 7/r
No VT
we have:
Lp(wo) L. in 1
(LG + p² TT2 (,Tr))n log(T)
E V (L
(21)
T t=1
710
VT
The influence of x: In the proof of Theorem 1, specifically in Eq. (15), is integrated into B, serving hyperparameter that regulates the weight adjustment in DISAM. It functions by modulating the degree of correction for domain shifts:
M
Bi
2A
('(w))=
Σ M
1
cd (1xr)
(wi)
M
The choice of influences how aggressively DISAM responds to variance domain shifts, with a higher À leading to more pronounced adjustments in 3. Our experimental analysis in Figure 5(c) and 5(d), reveals that DISAM's performance remains relatively stable across wide range of À values. However, choosing too large À can result in overly aggressive early training adjustments, yielding the increased variance in repeated experiments. Consequently, we adopted default À value of 0.1 in all experiments.
B.4 DISCUSSION OF THE ROLE OF IN DISAM
Here, we provide detailed discussion on how p affects both generalization and convergence, First, we introduce the generalization theorem of the upper bound on generalization error, which is only related to the magnitude of p. and DISAM follows the same upper bound on generalization error as SAM. In the SAM framework, the parameter plays crucial role in determining generalizability. As established SAM (Foret et al., 2021), there exists an upper bound on the generalization error for SAM, suggesting that larger P could potentially enhance generalization, provided that convergence not impeded. Here is the relevant generalization bound from SAM (Foret al., 2021);
Theorem 2. (Generalization Bound of SAM). For any p > 0 and any distribution D, with probability DD
1 over the choice of the training set S
klog
[+
|| nr (1
log(n)
+ 4 2 # +(
CD(w)= < max Cs(w +
(22)
||l||55p
72
21

---

Published as a conference paper at ICLR 2024
where n |S), k is the number of parameters and we assumed CD(w) S (, [Lp(w (w + e)]. This theorem's 's proof focuses solety on the magnitude of p. thus affirming the applicability of this theoretical framework to DISAM.
When considering domain shift, the upper bound on generalization error for the test domain is:
Theorem 3. (PAC-Bayesian Generalization Bound). For any p and the unseen domain T, suppose we have multi-source domains = {S1, -2. with a total of N samples. Let be the hypothesis space and Q be the corresponding parameter space, where the VC dimension of H is d. We denote the domain divergence between two domains D, and Dj on the hypothesis space H as dHAH ( Di, D2). Then, for any 8 E (0, with probability at least 1 for all E Q2 have:
log d + log
Sell>SSp Cr(w) & max Ls(w + + duan(S.T) +
+
2N
2
(23)
k log
1
1
(og(N)
4og NN O(1)
where X is the optimal combined risk on T and S that can be achieved by the parameters in Ω.
Combining this with the convergence theorem (Theorem 1), there is trade-off with respect p. larger might theoretically enhance generalization but poses greater challenges for convergence. This reflects the intuitive notion that searching for flatter minima across broader range inherently more complex, which can potentially affect training efficiency. However, if Cs(w + E) can be converged with a sufficiently small value, a larger P corresponds to better generalization. DISAM, compared to SAM, converges faster, which means that under the same convergence speed, larger & can be used to achieve better generalization. This advantage is empirically showcased in Figure 3(c) and (d), where we demonstrate that DISAM effectively employs larger p compared traditional SAM. This ensures both convergence and enhanced generalization. Such a capability to balance between convergence efficiency and generalization is distinguishing feature of DISAM over conventional SAM methods.
C DETAILED EXPERIMENTS
C.I DETAILED EXPERIMENT SETUPS
We present the detailed results obtained from five datasets, namely PACS (Li al., 2017) (9,991 images, 7 classes, 4 domains), VLCS (Fang et al., 2013) (10,729 images, 5 classes, 4 domains), Of- ficeHome (Venkateswara et al., 2017) (15,588 images, 65 classes, 4 domains), Terralncognita (Beery et al., 2018) (abbreviated as TerraInc, 24,788 images, 10 classes, domains), and DomainNet (Peng al., 2019) (586,575 images, 345 classes, 6 domains), following the DomainBed benchmark (Gulra- jani & Lopez-Paz, 2021) with the ResNet50 backbone architecture. We set the hyperparameters for the Domain Comain-IIsppred SAM method as follows: p 0.5 and 0.1 for PACS, VLCS. OfficeHome. and NomainNet; for TerraInc, we use 0.01 and X = 0.2. Both Domain-Inspirce + GSAM and Domain-Inspired + SAGM employ the strategy described in the supplementary material of SAGM (Wang et al., 2023b). As for the CoOp with CLIP, we set the batch size as 32 and the default learning rate as 2e-3. Given the detailed experimental hyperparameter settings provided in the SAGM supplement (Wang et al., 2023b) and the official open-source CLIPOOD code (Shu et al., 2023), we directly applied these official settings. The results, replicated using the official open-source CLIPOOD code, are presented in Table 2 of the main text.
As for the experiments on open class, we found that CLIPOOD requires a lower learning rate and correspondingly lower p, and therefore used learning rate lc-07 and le-05 default settings.
C.2 DETAILED EXPERIMENTAL RESULTS
We present the specific -o-aann experimental results for each dataset in Table corresponding each mane-one-domain-eee setting.
22

---

Published as a conference paper at ICLR 2024
Table 6: Comparison with state-of-the-art domain generalization methods. Out-of-domain accuracies on the PACS dataset with ResNet50 backbone.
Algorithm
Art
Cartoon
Photo
Sketch
Avg.
ERM
84.7
80.0
97.2
79.311.0
85.5
SAM
85.6+2.1
80.9
97.0+0.4
79.641.6
85.8
Domain-Inspired
87.1
81.940.5
96.2
83.1
87.1
GSAM
86.9
80.440.2
97.5.0.0
78. 78.7
85.9
Domain-Inspired
88.4+0.2
81.1+0.3
97.020.0
82.3
87.2
SAGM
87.4
80.2..03
98.0 10.2
80.8
86.6
Domain- Inspire +CORAL
89.7
81.5
97.0+0.1
81 81.8+0.6
87.5
89.8+0.5
82.9+0.2
97.4+0.2
83.4+0.2
88.4
Table 7: Comparison with state-of-the-art domain generalization methods. Out-of-domain accuracies on the VLCS dataset with ResNer50 backbone.
Algorithm
Caltech
LabelMe
Pascal
Sun
Avg.
ERM
98.0+u.3
64.7+11
71.4+1.2
75.2
77.3
SAM
99.1
65.041.0
73.7
79.8
79.4
Domain-Inspired
99.3
66.3
81.040.1
73.2
79.9
GSAM
98.7 10.3
64.9
74.3
78.5
79.1
Domain-
99.8
66.6
74.210.9
79.3
80.0
SAGM
99.0
65.2
75.1 10.3
80.7
80.0
Domain-Inspired
99.9 10.1
G0.1
75.1.0.3
81.8 $1.810.0
80.7
+CORAL
99.710.1
67.840.7
75.5
81.610.2
81.2
Table 8: Comparison with state-of-the-art domain generalization methods. accuracies on the OfficeHome dataset with ResNet50 backbone.
Out-of-domain
Algorithm
Art
Clipart
Product
Real World
Avg.
ERM
61.3
52.410.3
75.8
76.640.3
66.5
SAM
64.540.3
56.540.
77 77.4+0.1
79.8+0.4
69.6
Domain-Inspired
65.8
55.610.2
79.2
80.610.1
70.3
GSAM
64.940.1
55. 25.2+0.2
77 77.8+0.0
79 09.2+0.2
69.3
Domain-Inspired
65.740.3
57.4
79.4 79.4
80.7
70.8
SAGM
65.440.4
57.0+0.3
78.0+0.3
80.040.2
70.1
Domain-Inspired
67.2
56.340.3
79.640.2
81.0+0.3
71.0
+CORAL
68.540.1
57 57.640.1
79.3
81.3
71.7
23

---

Published as a conference paper at ICLR 2024
Table 9: Comparison with state-of-the-art domain generalization methods. Out-of-domain accuracies on the TerraInc dataset with ResNet50 backbone.
Algorithm
L100
L38
L43
L46
Avg.
ERM
49.8
42.1
16.9
35.7
46.1
SAM
46.3+1.0
38.4+2.4
54.0+1.0
34.5
43.3
Domain-Inspired
46.2+2.9
41.640.1
58.0
40.5+2.2
46.6
GSAM
50.8
39.311.2
59.640.0
38. 25..8
47.0
Domain-Inspieee
56.7
46.7+1.0
59.2
39.941.5
50.6
SAGM
54.841.3
41.440.8
57.7
21.3
48.8
Domain- hnspire
57.6 1..6
44.8
58.6
38.940.6
50.0
+ CORAL
57.9
46.640.6
59.9+0.3
42.5
51.7
Table 10: Comparison with state-of-the-art domain generalization methods. Out-of-domain accuracies on the DomainNet dataset with ResNet50 backbone.
Algorithm
Clipart
Infograph
Painting
Quickdraw
Real
Sketch
Avg.
ERM
62. 62.840.4
20.2+0.3
50.3+0.3
13.7
63.7+0.2
52.1
43.8
SAM
64.540.3
20. 20.7+0.2
50.2+0.1
15.1
62.640.2
52.7+0.3
44.3
Inspire
65.9
20.7+0.2
51.7+0.3
16.640.3
62.8+0.5
54.8.0.4
45.4
GSAM
64.2+0.3
20.8+0.2
50.940.0
14.4+0.8
63.5+0.2
53.9
44.6
Domain-Inspired
65.7+0.1
21.3+0.1
52.2+0.1
15.620.0
64.520.2
54.1
45.6
SAGM
64.940.2
21.140.3
51.5+0.2
14.8
64.1
53.640.2
45.0
Domain-Inspired
65.9
21.4.00.0
52.6
15.8
65.3
54.8 go.2
46.0
+CORAL
66.440.3
21.940.2
53.110.1
16.110.0
65.3
55. 55.04..0
46.3
C.3 DETAILS ABOUT ESTIMATED SHARPNESS ON UNSEEN TEST DOMAIN
Estimating sharpness involves a significant computational overhead. the earliest methods, Monte Carlo random sampling the only viable approach (Dinh et al., 201 7b; Hochreiter & Schmidhuber, 1994b). However, recent udvancements have introduced efficient approximation techniques based on gradients estimate sharpness (Jiang et al., 2023; 2020). Based the work of Jiang al. (2023), we assess the sharpness of the training model on the unseen test domain at the end of each epoch. Sharpness commonly characterized by the eigenvalues of the Hessian matrix (Keskar et al., 2017b; Dinh et al., 201 7a), but direct computation incurs substantial overhead. To address this, computationally efficient measurement of sharpness is proposed by Jiang et al. (2020), which utilizes the gradient variance Var (MrVV((w: B,)) as an estimate (B; represent the batch data sampled at step t).
C.4 DETAILS ABOUT COMPARISON OF COMPUTATION COST
We selected the PACS dataset for experimentation, using platform with 16-core CPU, a single RTX30900 GPU, and 64GB RAM. The time overhead for one training step was calculated and averaged over 500 iterations. Due the lack of optimization for parallel acceleration in the variance calculation code, which employs simple 'for" loop approach, the actual overhead is larger than theoretically expected. Nonetheless, DISAM's advantage lies in its overhead being unrelated to gradient size, but only to batch size and domain number. This drawback can be addressed through parallel code optimization, and additional memory overhead is present.
C.5 DETAILS ABOUT CONVERGENCE CURVES OF SAM AND ERM
In this section, we provide detailed analysis the convergence curves depicted in Figure 1(b). Figure 7(a) presents the same Figure 1(b), with normalized representation of the loss curves,
24

---

Published as a conference paper at ICLR 2024
Training on domain 100, 43, 46 of Terrainc
Training on domain 100, 43, 46 of Terrainc
1.8
Conveggece
R.6
Convergence Inconsistency under domain shift
Lage
1.0
Convergence Inconsistency under domain shift
0.8
0..
0.5
000
02
Epoch
Epach
(a) Convergence curves under domain shifts
(b) Loss curves under domain shifts
Figure 7: Illustration of SAM's degradation of the training process under domain shifts. (a) Con- vergence curves of SAM and ERM for each domain during training, with the convergence degree normalized to [0.1]. (b) Loss curves of SAM and ERM for cach domain during training.
1
Convergence curve test domain A
1.0
Convergence curve on test domain S
ERM
ERM
0.8
0.8
SAM
SAM
Loss
0.6
DISAM
3 0.6
DISAM
0.4
0.4
0.2
0.2
0.0
1000
2000
3000
2000
5000
0.0
1000
2000
3000
4000
5000
Step
Step
(a)
(b)
Figure 8: Convergence curves for ERM, SAM and DISAM. (a) & (b) show the trend of [ww) during the training process on PACS dataset.
ranging from 0 to 1. achieved by subtracting the minimum loss value and dividing by the maximum
loss value. Our intention is to emphasize the inconsistency in convergence trends across different SAM domains, as illustrated by the optimization overshoot observed in Figure 7(a). Figure 7(b) showcases the actual loss change curve. It is apparent that due to the consistency issue encountered during the early phase of convergence, the in-domain convergence is compromised, resulting in poor generalization performance in the out-of-domain scenario.
C.6 DETAILED ANALYSIS ABOUT OPEN-CLASS GENERALIZATION
In the experiments of open-class generalization, as presented in Table and Figure 4 of section 4.4, specifically explore the effectiveness of DISAM for parameter-efffiiee fine-tuning (PEFT). Our quantitative analysis compares the performance of ERM, SAM, and DISAM in fine-tuning scenarios. As shown in Table 3, although CoOp and CLIPOOD perform better base classes than zero-shot, their performance on new classes is worse than zero-shot. This suggests that the fine-tuned parameters overfit to the existing training data distribution from both the domain and class perspectives. This overfitting is particularly detrimental to the generalization of large VLM models, which often have feature representations too rich for the downstream task, especially when only a small number of parameters are fine-tuned. Figure 4 visualizes the change in performance trends during the training process, and we observe a trend where ERM initially performs well on base classes but then exhibits decline on new classes, suggesting a collapse of the feature space onto the training data classes. Although SAM offers some relief from overfitting, its performance on new classes does not match zero-shot levels. In contrast, DISAM, by minimizing sharpness more effectively, shows improved performance on new classes, especially in domain shift scenarios.
C.7 DETAILED ANALYSIS OF CONVERGENCE SPEED COMPARISON
We presented comparison of the convergence speed with the inclusion of ERM in Figure 8. It can be observed that although DISAM converges much faster compared to SAM, the overall convergence speed is still slower than ERM due the introduction of p.
25

---

Published as a conference paper at ICLR 2024
D PSEUDO CODE OF DISAM
We present pseudo-code for DISAM using Python syntax. PyTTorch is utilized the deep learning experimental framework. The code for the optimizer in the SAM-based method can be referenced from the provided open source links in the relevant papers.
Listing 1: Training Code for DISAM
def train train_epoch_diise (datalos) loader, model, zer)
nun
Train the DISAM model for one epoch.
Args:
dataloader (DataLoader) The training dataloader. model (nn.Module): The training model. optimizer (Optimizer): The SAM-based optimizer, such SAM, GSAM, and SAGM.
model.traini
for 1, data_list in tqdm tqdm (enumerate (dattaladee))) )
imgs, labels = data_list
imgs, Labels imgs. labels.
preds model (imgs)
Calculate domain losses and total loss using the cross-entropy 1oss function
domain_loss__ue total_ Loss = get denain losss (preds, labels, Bommin__abeell loss loss_func)
loss_variance compute variance (domain loss list)
loss total_loss landa * loss_variance
optimiler..ero_grrae.
loss (ossbbacwward)))
Perform the first step of SAM: gradient ascent with fixed length rho optimizer. saaiiiier.firststtee (zero_ (zero_grad-True))
output model (imgs)
loss Lune_ (output, Labels) loss. loss. backward
Obtain the actual gradient from the perturbation location of DISAM
optimi zer, secood step (zero (zero_grad-True)
def got_domain_loss (preds, labels, dabeln_labels, loss_ func))
una
The function to compute the Loss each domain.
Args:
preds (Tensor): The predi cti ona of the training model in one batch.
labels (Tensor): The labels of batch data. domain_labels (Tensor) The domain labels of batch data. loss_func: (Punction): The loss function.
un.
1ist all domains domain_list 1ist (set (dbelin_ domain_loss_lkt = []
total_loss 0.
for nomein_name in domain_list:
Get the mask for the current domain domain_mask domsin_labeis == domain_name
1abe1 s_per_domain 1abel s [doma m_mask]
26

---

Published as a conference paper at ICLR 2024
preds_pre_ donain preds dommainn _ask]]
Compute the loss for the current domain s Langle_domain__eee
Loss_fund (prode_pre_domall) sabels_per_domain]
domain_loss, _isaass___ttte___eee apped (s ingge_domaileen dossainnosss)
Add the loss for the current domain to the total loss, taking into account the number samples in the domain
total_loss len (labe' s_per_domain) single_domainn
total_loss /- len (labels)
return domain loss list, total_ loss
def compute_variance (domain
The function compute the variance the list of domain losses Args:
Nomain__oss List_list (Iist): the 1ist of each domain's s loss. u..
loss variance
for domain_i_loss in domain_iosa_list: for domain_j_loss in Bomain_loss_litt:
$ Compute the square of the difference in loss between each pair of elements and add it to the loss variance doss_variance += (domain_i_loss
domair_j_loss)
loss_variance /= (2+len
xx2)
return loss_variance
27


Reviews:
[{"review_id": "BGpXY97QRW", "reviewer": "Reviewer_sVwJ", "summary": "Due to the inconsistent convergence degree of SAM across different domains, the optimization may bias towards certain domains and thus impair the overall convergence. To address this issue, this paper considers the domain-level convergence consistency in the sharpness estimation to prevent the overwhelming perturbations for less optimized domains. Specifically, DISAM introduces the constraint of minimizing variance in the domain loss. When one domain is optimized above the averaging level w.r.t. loss, the gradient perturbation towards that domain will be weakened automatically, and vice versa.", "questions": "1.\tIn the definition of the variance between different domain losses, the values of loss between different domains are restricted. Which one is more import? The value of losses in different domains, or the minimization speed of loss in different domains?\n2.\tIn the learning of multiple domains, there is Multi-Objective Optimization, so the domain-level convergence consistency is a general issue under domain shifts? Or the convergence consistency is a general issue in Multi-Objective Optimization?\n3.\tThis paper considers the domain-level convergence consistency in SAM for multiple domains, and proposes to adopts the domain loss variance in training loss. The convergence consistency is a general issue, and the solution is normal, thus the novelty is not so clear.", "limitations": "", "rating": 5, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "They identify that the use of SAM has a detrimental impact on training under domain shifts, and further analyze that the reason is the inconsistent convergence of training domains that deviates from the underlying i.i.d assumption of SAM.", "weaknesses": "This paper considers the domain-level convergence consistency in SAM for multiple domains, and proposes to adopts the domain loss variance in training loss. The convergence consistency is a general issue, and the solution is normal, thus the novelty is not so clear for publication in ICLR.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ku4NzSCV00", "reviewer": "Reviewer_mma9", "summary": "This paper introduces a novel optimization algorithm named Domain-Inspired Sharpness Aware Minimization (DISAM) tailored for challenges arising from domain shifts. It seeks to maintain consistency in sharpness estimation across domains by introducing a constraint to minimize the variance in domain loss. This approach facilitates adaptive gradient adjustments based on the optimization state of individual domains. Theoretical and empirical findings show the proposed method offers faster convergence and superior generalization under domain shifts.", "questions": "1.\tThe article presents a theoretical analysis suggesting that larger values of parameter $\\rho$ should lead to improved generalization, given that convergence is guaranteed. It is important to reflect this aspect in the experiments to provide stronger evidence and validation.\n\n2.\tRegarding the open class generalization of the clip-based model, further experimental analysis should be conducted to elucidate the reasons behind the superior performance of DISAM.\n\nFor other questions, please refer to the weaknesses.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1.\tThe proposed method targets at the model generalization under domain shifts, which is a common challenge in machine learning. To date, there has been a lack of thorough investigation into sharpness-based optimization in the context of domain shifts, and the idea of constraint the variance of losses among training domains is interesting.\n\n2.\tThe paper not only presents theoretical evidence showcasing the efficiency of DISAM, but it also provides empirical data to support this claim, demonstrating the improved performance across various domain generalization benchmarks.\n\n3.\tThe analytical experiments conducted in this paper are comprehensive and lucid, providing evidence of DISAM's efficacy in enhancing convergence speed and mitigating model sharpness. Additionally, the study investigates the application of DISAM for fine-tuning a clip-based model, aiming to achieve improved open-class generalization.", "weaknesses": "1.\tSAM-based optimization incurs twice the computational overhead and additional storage overhead in comparison to the commonly used SGD. While DISAM, the method proposed in this paper, demonstrates faster convergence under domain shift conditions when compared to SAM, it does not include a comparison with optimizers such as SGD or Adam.\n\n2.\tThis paper employs multiple benchmarks to evaluate the performance of multi-source domain generalization. The article highlights the need for advancements in the domain shift perspective of the SAM method and suggests conducting comparisons between DISAM and the state-of-the-art (SOTA) method to further validate the effectiveness of the proposed approach.\n\n3.\tThe value of $\\rho$ in DISAM significantly influences both the convergence speed and generalizability. And it needs more discussion on how to effectively determine the value to maximize the benefits of proposed method.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "0bcLv8wwPA", "reviewer": "Reviewer_d1EH", "summary": "The paper introduces the Domain-Inspired Sharpness Aware Minimization (DISAM) algorithm, a novel approach for optimizing under domain shifts. The motivation behind DISAM is to address the issue of inconsistent convergence rates across different domains when using Sharpness Aware Minimization (SAM), which can lead to optimization biases and hinder overall convergence.\n\nThe key innovation of DISAM lies in its focus on maintaining consistency in domain-level convergence. It achieves this by integrating a constraint that minimizes the variance in domain loss. This strategy allows for adaptive gradient perturbation: if a domain is already well-optimized (i.e., its loss is below the average), DISAM will automatically reduce the gradient perturbation for that domain, and increase it for less optimized domains. This approach helps balance the optimization process across various domains.\n\nTheoretical analysis provided in the paper suggests that DISAM can lead to faster overall convergence and improved generalization, especially in scenarios with inconsistent domain convergence. The paper supports these claims with extensive experimental results, demonstrating that DISAM outperforms several state-of-the-art methods in various domain generalization benchmarks. Additionally, the paper highlights the efficiency of DISAM in fine-tuning parameters, particularly when combined with pretraining models, presenting a significant advancement in the field.", "questions": "Similar to how transitioning from ERM to vREX in optimization has been shown to enhance domain generalization performance, the application of vREX to SAM in the form of this methodology could be seen as a natural extension that brings comparable performance improvements. Furthermore, it is a valid assertion that incorporating various algorithms tailored for domain generalization (such as Fish, Fishr, gradient alignment) into the SAM optimization framework could potentially yield performance enhancements. The logic here is that these methods, when applied within the context of SAM, could enhance its ability to generalize across domains.\n\nHowever, the critique that DISAM may simply be an incremental version of applying domain generalization methodologies to SAM is not without its counterarguments. It's important to consider the specific challenges and nuances of the SAM framework and how DISAM addresses these. If DISAM introduces significant modifications or adaptations that are uniquely tailored to the idiosyncrasies of SAM, then its contribution could extend beyond a mere incremental update. The key would lie in the specifics of how DISAM modifies or enhances the existing principles of SAM and domain generalization methods, making it more than just a straightforward application of known techniques.\n\nIn summary, while the perspective that DISAM is an incremental version of existing methodologies is certainly tenable, a comprehensive evaluation would require a deeper exploration of how DISAM specifically adapts or augments the SAM framework to address its unique challenges. If such adaptations are significant, they could justify the novelty and utility of DISAM beyond a simple combination of existing techniques.\n\nCan you provide the reproducible code during the rebuttal period?", "limitations": "", "rating": 6, "confidence": 5, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "As of now, there has not yet been a sharpness-aware minimization (SAM) methodology developed specifically for addressing distribution shifts. The issue of varying convergence rates across different domains, as observed in SAM, is undeniably a significant challenge.\n\nThis methodology presents an impressive degree of compatibility, as it can be integrated with a variety of sharpness-variants. An especially commendable aspect of this approach is its computational efficiency. Compared to standard SAM techniques, it does not incur additional computational costs, making it a practical option for scenarios where resource constraints are a consideration.\n\nIn summary, the development of a SAM methodology that is adept at handling distribution shifts, and particularly its implications for domain convergence, is both novel and highly relevant in the current landscape of optimization challenges.", "weaknesses": "The idea of minimizing the variance between losses, a core aspect of the presented methodology, is not entirely novel. Similar concepts have been previously explored in methods like vREX (Out-of-Distribution Generalization via Risk Extrapolation) and further extended to gradient computations in methodologies like Fishr (Invariant Gradient Variances for Out-of-Distribution Generalization). In this context, the proposed approach appears to be an incremental adaptation of vREX principles applied specifically to the challenges faced in Sharpness Aware Minimization (SAM) scenarios.\n\nThe improvement in out-of-distribution (OOD) performance using the DISAM methodology does not appear intuitive. In fact, when comparing its performance enhancements to those achieved with CLIPOOD, as reported, the difference seems marginal. This observation raises questions about the actual effectiveness of DISAM, particularly in the context of fine-tuning methodologies.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "uGpWP0oKoS", "reviewer": "Reviewer_BeTH", "summary": "Targeting at domain generalization scenario with possible shifts among domains, this paper proposes to take 'per domain optimality' into consideration for finding the perturbation of SAM. The proposed DISAM is shown to have an improved convergence rate. Numerically, DISAM outperforms other SAM alternatives.", "questions": "Q1. Relation with a recent work (https://arxiv.org/abs/2309.15639).\n\nThe paper above also proposes approaches to reduce variance for finding perturbations, although not designed for the domain generalization setting. How does this work relate with the proposed DISAM?\n\n\nQ2. Theorem 1 illustrates that the *convergence* of DISAM benefits from $\\Gamma$. Can the authors explain more on the discussion of \n> as DISAM enjoys a smaller $\\Gamma$ than SAM, DISAM can permit the potential larger $\\rho$ than that in SAM, thus yielding a better generalization\n\nIn particular, how does the convergence rate link with generalization?\n\nQ3. The last sentence in Sec 3 claims that\n>  ... allowing larger $\\rho$ for better generalization.\n\nWhy does larger $\\rho$ relate to better generalization?\n\nQ4. (minor) The notation in e.g., eq (5) can be improved, because the multiple subscripts $i$ in $\\Sigma_{i} \\frac{C_i}{\\sum_i C_i}$ are confusing.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "S1. The idea of tackling domain shift in SAM is novel. \n\nS2. A new algorithm, DISAM, is proposed with satisfying numerical results. DISAM improves over state-of-the-art by a large margin.", "weaknesses": "W1. Stronger motivation needed. The authors motivates the domain difference using Fig. 1 (b). While the convergence behaviors among domains are indeed inconsistent at the early stage,  the losses are similar after e.g., 30 epoch. The authors should also explain why the difference of convergence in **early phase** impact the generalization of SAM.\n\n\nW2. More discussions on $\\lambda$ in eq. (7) are needed. This is a critical parameter that considers the variance/domain shifts in DISAM. However, this $\\lambda$ does not appear in Theorem 1. Can the authors illustrate more on this point? And how does the choice of $\\lambda$ influence convergence and generalization?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)